What a forecasting platform's score actually measures

A leaderboard number is the output of a formula, applied to a corpus of questions somebody else chose, sometimes with a crowd subtracted from it. Change any of those three and the number moves while your forecasts stay exactly where they were. Which is why two people comparing their scores across two sites are usually comparing nothing at all.

This page is about the number itself. Which of three named products to sign up for is a different question with its own page, and everything below stays true if all three of them close tomorrow.

How it works

Four things happen between the probability you state and the figure on your profile, and each one is a separate design decision made by somebody who was not thinking about you:

1. A scoring rule turns your probability and the realised outcome into a single number.
2. That number is usually averaged over time, so a forecast earns for every day it stood rather than once.
3. A crowd term may be subtracted from it, which converts an absolute measurement into a comparison.
4. The per-question numbers are aggregated over a corpus of questions that reached resolution — which is not the same set as the questions that were asked.

Steps 3 and 4 are where most of the confusion lives, and step 1 is where all the incentives are.

What makes a rule proper, and why that is not an academic nicety

The definition is narrow and worth having exactly. A scoring rule is proper if the forecaster maximises their expected score by quoting the distribution they actually believe rather than any other one, and strictly proper if that maximum is unique — nothing else ties with the truth. That is the formulation in the standard review of the subject, which puts the consequence in one line: propriety is essential in scientific and operational forecast evaluation.

Metaculus states the same thing in the operational register on its scores page — the log score is proper, which means that to maximise your score you should predict your true beliefs.

Under an improper rule the honest answer and the optimal answer come apart, and the platform is then quietly teaching you the second one. The cleanest example is the rule a reader is most likely to meet, because it is the one that needs no explanation on a marketing page: rank people by the share of questions they got right.

share correct = (calls above 50% that happened + calls below 50% that did not)
                / questions answered

Notice what the formula does with your probability: it uses it to pick a side and then throws it away. A 51% and a 99% score identically. Two strategies follow immediately, and neither is forecasting.

• Avoid the uncertain questions. A well-judged 60% on a genuinely hard question is worth 0.6 expected share-correct, which is worse than skipping it and answering an easy one instead. The rule pays nothing for the judgement and everything for the certainty, so the optimal corpus is the one you were never confused about.
• State extremes for free. Since magnitude is discarded, there is no cost to saying 99% when you mean 70%, and a habit formed that way is expensive to unlearn on a platform that does measure magnitude.

Meteorology worked through this a century before forecasting platforms existed, and the working group that maintains the field's verification reference still leads with the case. In March 1884 Sergeant John Finley began twice-daily tornado forecasts for eighteen regions of the United States and reported overall accuracy between 95.6% and 98.6%. A critic pointed out that 98.2% was available by forecasting "no tornado" every time. The same page's standing verdict on proportion correct is that it is simple and intuitive and can be misleading, because it is heavily influenced by the most common category.

So: if a platform ranks you by hit rate, streaks, or points whose formula it will not print, the number is not a measurement of your judgement and a season spent improving it will not improve your judgement either.

Brier and the log score give the same instruction and behave differently

Both are proper, so both tell you the same thing — say what you believe. They are not interchangeable, and the difference is entirely in the tail.

Brier, one outcome     (p - y)^2                           y = 1 if it happened, else 0
Brier, both outcomes   (p - y)^2 + ((1-p) - (1-y))^2
log score              ln(p) if it happened, ln(1-p) if not

Take the decision this actually turns into: you think something is unlikely and you are choosing between saying 2% and saying 5%. Here is what that choice is worth, using the one-outcome Brier convention and natural logs.

The two scales are not comparable as printed, so put both in the same unit: how much of a coin-flip forecast the decision is worth. A 50% forecast scores 0.25 under this Brier convention and −0.693 under the log score. Choosing 2% over 5% and being wrong therefore costs you about a quarter of a coin flip under Brier and about one and a third coin flips under the log score — roughly six times more consequential under log, for the same two probabilities.

That ratio widens as you go further out, because Brier is bounded and the log score is not. A miss at 2% scores 0.9604 under Brier; a miss at 0.02% scores 0.9996. Those are the same number for any practical purpose, so Brier can barely tell the difference between being confidently wrong and being absurdly confidently wrong. The log score scores the same two misses at −3.91 and −8.52, and there is no floor under it at all — a 0% forecast on something that happens is undefined, which is why platforms that use it clamp the extremes before they will accept them.

Two practical consequences for somebody deciding whether to say 2% or 5%:

• On a log-scored platform, the tail is worth thinking about. The difference between 2% and 5% is a real decision, and it is the one that separates people at the top of the table. Sharpness you cannot defend is the most expensive thing you can buy there.
• On a Brier-scored platform, it barely registers in either direction, and your rank will be decided by the middle of the range, where the squared error has room to move. Time spent agonising over a tail probability is time that scores nothing.

Neither of those is advice to state something other than your belief — both rules are proper, so that stays the answer. It is advice about where the effort goes, and about the variance of your season. A single confident miss at 2% costs you the equivalent of roughly five and a half ordinary good calls at 90% under the log score, and about three under Brier.

The Brier convention trap, which is the factor of two

This is the one that silently corrupts every cross-platform comparison, and it is not obscure: both conventions are in live use, in documents a reader will actually be handed.

Good Judgment Open's FAQ sums the squared error over every possible outcome. Its own worked example is a yes/no question forecast at 70% where the event occurred:

(1 - 0.7)^2 + (0 - 0.3)^2 = 0.18

and it states the worst possible score as 2. The WMO working group's verification reference gives the Brier score as the mean of (forecast − observation)² over forecast/observation pairs, and states the range as 0 to 1. The same forecast under that form is:

(1 - 0.7)^2 = 0.09

Same rule, same forecast, two numbers, a factor of two apart. Neither document is wrong; they are simply not the same statistic, and almost nobody publishing a Brier score says which one they mean. Manifold prints "0.17375 Brier" on its calibration page with no convention named anywhere near it, and the forecasting platforms listing has to carry the same caveat on the category as a whole.

Three things follow, and the third is the one people get wrong:

• A raw Brier number from one site cannot be compared to one from another until both conventions are named. Not "should be compared carefully" — cannot, because one of the two candidate answers is exactly twice the other.
• The quick test is a single question. Ask what a 70% forecast on something that happened scores. If the answer is 0.18 it sums both outcomes; if it is 0.09 it does not. If nobody can answer, the published figure is not usable as a comparison.
• The rules of thumb inherit the ambiguity. "Under 0.25 is decent" is a statement about the one-outcome convention, where 0.25 is exactly what you get by saying 50% to everything. Under the two-outcome convention the coin flip is 0.5 and the same rule of thumb is twice as strict as whoever repeated it intended.

A relative score is a statement about the crowd, by construction

Subtracting the crowd is a reasonable thing to do — it strips out how hard the questions were, which is otherwise the largest term in anybody's score. What it produces is not a harder version of an absolute score. It is a measurement of a gap, and a gap has two ends.

Metaculus Peer      (mean over all other forecasts on the question of
                     your log score minus theirs) x 100      higher is better

GJ Open Relative    (your average daily Brier
 Brier Score         minus the average daily median Brier)
                     x your Participation Rate               lower is better

Four differences between those two lines, any one of which is enough on its own to make the numbers incomparable:

• The rule underneath. Log on one, Brier on the other, with all of the tail behaviour above attached to the choice.
• The crowd statistic. A mean of everyone else on one, the median of everyone with an active forecast on the other. A mean is moved by one terrible forecast; a median is not.
• The sign. Metaculus states plainly that higher is better. Good Judgment Open states that negative means you were more accurate than the crowd. The arrows point in opposite directions.
• Whether it cancels. Metaculus states that on any given question the sum of all participants' Peer scores is always 0 — zero-sum by construction. The median-and-participation form does not cancel, because a median is not an average and multiplying by each forecaster's own participation rate breaks the symmetry anyway.

The arithmetic consequence is the part worth carrying away: your relative score moves when the crowd moves and you do not. Post identical forecasts to a weaker field and your number improves. Post them against a stronger one and it falls. Two forecasters with the same relative score on two platforms, or in two tournaments on the same platform, have told each other nothing.

None of which makes a relative score a bad number. It is the right number for the question "was I better than these people, on these questions". It is the wrong number for "how good am I", and it is not a number at all for "am I better than someone over there". If what you actually want is to pick between two of these products, the head-to-head page does that comparison with the feature lists attached.

Profit is a different quantity, not a rougher version of the same one

On a platform with a price, the headline figure on a profile is usually profit. Profit is position size multiplied by price movement, summed over your fills. Your stated probability is not an input to it anywhere.

What it pays for is finding a mispriced question, getting in early while a market is thin, sizing the position, and supplying liquidity for other people to trade against. Those are real skills, they are difficult, and not one of them is the skill of stating a well-calibrated probability. A trader taking many small edges on questions whose subject matter they have no view about can out-earn a well-calibrated forecaster working on genuinely hard ones, and the leaderboard cannot tell you which of the two you are reading.

Two features of play-money platforms change the incentive further.

The reward arrives in a consumable. The familiar argument for prices — that a market stays honest because correcting somebody else's error pays — assumes the payment is worth something outside the market. On a play-money platform it is not: Manifold's own FAQ describes mana as the play-money currency used on the platform and says it cannot be converted to cash. Correcting a price therefore buys you more of the same site and nothing else. That is not an argument that the incentive fails — a permanent public record is a real motivation and the platforms publish data you can examine — but it is a different argument from the one people think they are making, and it should be made on its own evidence.

On some platforms the question is resolved by the person who wrote it. Manifold's FAQ puts it in five words: whoever created the market gets to resolve it. That is a deliberate design choice and it is what makes questions cheap to list, but it changes what a profit figure records. Part of it is a verdict on how well a series of strangers read resolution criteria they wrote themselves. The four different answers venues give to "who decides" have their own page, and it is worth reading before you treat any settled question as a scored one.

A third caveat sits underneath both: in a thin market the price is not a probability in the first place, which where the liquidity comes from takes apart in detail.

What a platform-wide score is a claim about

The last step is aggregation, and an aggregate is a claim about the platform, not about the question open in front of you.

Manifold's calibration page is the most useful example precisely because it publishes its method. The figure is 0.17375, beside a stated sample size of 98k trades, produced by sampling 2% of past trades hourly on resolved binary questions with 15 or more traders, grouping them by probability band and checking how often each band resolved yes. The page notes that this is trade-weighted rather than time-weighted calibration, and adds that market accuracy may be better than reflected, because large miscalibrated trades are usually corrected immediately.

Count the decisions in that paragraph: a trader threshold, a sampling rate, a weighting choice, and a caveat about what the weighting does to the result. Four, all disclosed, plus the Brier convention which is not. Most published aggregates disclose none of them and print one number.

Then there is selection, which no methodology note can fix. A corpus of resolved questions is not a random sample of questions asked. Anything that was voided, annulled, left open or quietly delisted is out of the average, and those are disproportionately the ambiguous ones — the questions where the criteria turned out not to survive contact with the world. A platform whose question supply is chosen by research funders has a different corpus from one where anyone lists anything in thirty seconds, and the two aggregates are measuring different populations even before they reach a formula.

What it costs

None of these platforms charges money for a score. What a score costs is attention, in units the formula picks rather than the ones you would pick, and the bill is mostly invisible until you have already paid it.

• Days, not answers. Metaculus states that predictions count towards scores proportionally to their duration, and that earlier predictions accumulate more weight the longer they stand. Good Judgment Open computes a Brier score for each day you had an active forecast and averages across those days. A perfect forecast entered the day before a question closes earns almost nothing under either. The unit of work is the day the forecast is standing, which means maintaining a position you already hold is most of the job.
• Participation, priced explicitly. Good Judgment Open multiplies your relative score by your Participation Rate — the share of possible days you had a forecast in. Half the coverage halves the number, in both directions, so breadth and early entry are paid for at the same rate as being right.
• Variance, on a log-scored site. One confident miss at 2% costs roughly five and a half ordinary good calls at 90%, measured against a coin flip. Under Brier the same miss costs about three. If you are ranked by a log-based score, your season is decided by your worst handful of forecasts more than by your median one.
• Non-transferability, which is the largest cost and the last one noticed. A record built under one formula does not convert into a record under another. Three seasons spread over three platforms produce three short histories under three incomparable rules, which adds up to no track record at all. Choosing a formula is therefore a commitment, and it is worth making deliberately at the start rather than discovering a year in that the number you have been growing does not say what you want it to say.

What you can do about it

Put four questions to a platform before you chase its number. Each one has a factual answer that a support desk or a docs page can give, and a platform that cannot answer one of them has told you something.

1. Which scoring rule? Brier, log, something derived from one of them, or none of the above. If the answer is a hit rate, a streak, or "points", it is not a proper rule and the optimal strategy under it is not forecasting. If the answer is profit, it is not a scoring rule at all — it is a trading result, which is a different quantity and a fine one, just not this one.
2. Which Brier convention? Ask what a 70% forecast on something that happened scores. 0.18 means both outcomes are summed and the worst case is 2; 0.09 means one outcome and a worst case of 1. Every published figure differs by a factor of two between the two answers, so until you have this, do not compare the platform's number to any other number anywhere.
3. Who resolves the questions the score was earned on? Platform staff against published criteria, a named external source, or the person who wrote the question. The last one is not disqualifying, but it means part of your score measures other users' drafting. The full taxonomy is in who decides the outcome.
4. Is there a calibration curve you can look at yourself? Not a claim of accuracy — a plot, with the number of forecasts behind it, the date range, and the rule by which they were sampled. A site that publishes a curve with a denominator has handed you the means to disagree with it. A site that publishes a record with no denominator has handed you a slogan.

Score yourself before you trust anyone else's arithmetic. Keeping your own log is the only way to know your own convention and your own corpus, and it makes the platform's number a second opinion rather than the only one. Fatebook writes down private questions, computes Brier and relative Brier scores and draws a calibration chart; Quantified Intuitions scores you on questions that already resolved, which compresses the feedback loop from months to minutes.

For any claim that spans platforms, use a scorer that matched the questions by hand. Brier.fyi scores hand-matched questions across four venues and publishes its pipeline as open source, which is the only form in which a cross-venue accuracy claim can be checked — and its card also records that the pipeline is currently paused, which is the sort of fact a scoreboard rarely prints about itself.

When you quote your own score, quote the formula with it. "Relative Brier of −0.03 on Good Judgment Open, 40 questions, participation rate 0.7" is a statement somebody can evaluate. "My Brier is 0.09" is two different claims with the same words, and the reader cannot tell which one you made.

And put the volume in one place. The formula you choose is the thing you will get good at over a season, so choose it first and let the feature lists follow. If the decision in front of you is between specific products rather than between formulas, the three-way comparison makes it with the access, the resolution policy and the question supply attached; the forecasting platforms listing has the rest of the field.