# What a forecasting platform's score actually measures

Proper rules, the two Brier conventions that differ by a factor of two, crowd-relative scores that do not travel between sites, and why profit is not accuracy.

*https://predictionmarkets.tools/guides/what-a-forecasting-score-measures · background to Forecasting Platforms*

**Answer:** A score measures whatever its rule rewards. A proper rule, such as the Brier or the logarithmic score, is maximised by stating the probability you actually believe; anything else pays for a different skill. The logarithmic score punishes a confident miss far harder than Brier does, Brier is published under two conventions that differ by a factor of two, and a crowd-relative score describes the crowd you stood beside rather than you.

A leaderboard number is the output of a formula, applied to a corpus of questions somebody else
chose, sometimes with a crowd subtracted from it. Change any of those three and the number moves
while your forecasts stay exactly where they were. Which is why two people comparing their scores
across two sites are usually comparing nothing at all.

This page is about the number itself. Which of three named products to sign up for is a different
question with [its own page](https://predictionmarkets.tools/compare/forecasting-platform-scoring), and everything below stays
true if all three of them close tomorrow.

## How it works

Four things happen between the probability you state and the figure on your profile, and each one
is a separate design decision made by somebody who was not thinking about you:

1. A **scoring rule** turns your probability and the realised outcome into a single number.
2. That number is usually **averaged over time**, so a forecast earns for every day it stood
   rather than once.
3. A **crowd term** may be subtracted from it, which converts an absolute measurement into a
   comparison.
4. The per-question numbers are **aggregated over a corpus** of questions that reached
   resolution — which is not the same set as the questions that were asked.

Steps 3 and 4 are where most of the confusion lives, and step 1 is where all the incentives are.

### What makes a rule proper, and why that is not an academic nicety

The definition is narrow and worth having exactly. A scoring rule is **proper** if the forecaster
maximises their expected score by quoting the distribution they actually believe rather than any
other one, and **strictly proper** if that maximum is unique — nothing else ties with the truth.
That is the formulation in [the standard review of the
subject](https://sites.stat.washington.edu/raftery/Research/PDF/Gneiting2007jasa.pdf), which puts
the consequence in one line: propriety is essential in scientific and operational forecast
evaluation.

Metaculus states the same thing in the operational register on its scores page — the log score is
proper, which means that to maximise your score you should predict your true beliefs.

Under an **improper** rule the honest answer and the optimal answer come apart, and the platform
is then quietly teaching you the second one. The cleanest example is the rule a reader is most
likely to meet, because it is the one that needs no explanation on a marketing page: rank people
by the share of questions they got right.

```
share correct = (calls above 50% that happened + calls below 50% that did not)
                / questions answered
```

Notice what the formula does with your probability: it uses it to pick a side and then throws it
away. A 51% and a 99% score identically. Two strategies follow immediately, and neither is
forecasting.

- **Avoid the uncertain questions.** A well-judged 60% on a genuinely hard question is worth
  0.6 expected share-correct, which is worse than skipping it and answering an easy one instead.
  The rule pays nothing for the judgement and everything for the certainty, so the optimal corpus
  is the one you were never confused about.
- **State extremes for free.** Since magnitude is discarded, there is no cost to saying 99% when
  you mean 70%, and a habit formed that way is expensive to unlearn on a platform that does
  measure magnitude.

Meteorology worked through this a century before forecasting platforms existed, and the working
group that maintains the field's verification reference still leads with the case. In March 1884
Sergeant John Finley began twice-daily tornado forecasts for eighteen regions of the United
States and reported overall accuracy between 95.6% and 98.6%. A critic pointed out that
[98.2% was available by forecasting "no tornado" every
time](https://jwgfvr.github.io/forecastverification). The same page's standing verdict on
proportion correct is that it is simple and intuitive and can be misleading, because it is
heavily influenced by the most common category.

So: if a platform ranks you by hit rate, streaks, or points whose formula it will not print, the
number is not a measurement of your judgement and a season spent improving it will not improve
your judgement either.

### Brier and the log score give the same instruction and behave differently

Both are proper, so both tell you the same thing — say what you believe. They are not
interchangeable, and the difference is entirely in the tail.

```
Brier, one outcome     (p - y)^2                           y = 1 if it happened, else 0
Brier, both outcomes   (p - y)^2 + ((1-p) - (1-y))^2
log score              ln(p) if it happened, ln(1-p) if not
```

Take the decision this actually turns into: you think something is unlikely and you are choosing
between saying **2%** and saying **5%**. Here is what that choice is worth, using the
one-outcome Brier convention and natural logs.

| | you said 2% | you said 5% | the choice was worth |
|---|---|---|---|
| **Brier**, it happened | 0.9604 | 0.9025 | 0.0579 |
| **Brier**, it did not | 0.0004 | 0.0025 | 0.0021 |
| **log**, it happened | −3.912 | −2.996 | 0.916 |
| **log**, it did not | −0.020 | −0.051 | 0.031 |

The two scales are not comparable as printed, so put both in the same unit: how much of a
coin-flip forecast the decision is worth. A 50% forecast scores 0.25 under this Brier convention
and −0.693 under the log score. Choosing 2% over 5% and being wrong therefore costs you about
**a quarter of a coin flip under Brier and about one and a third coin flips under the log
score** — roughly six times more consequential under log, for the same two probabilities.

That ratio widens as you go further out, because Brier is bounded and the log score is not. A
miss at 2% scores 0.9604 under Brier; a miss at 0.02% scores 0.9996. Those are the same number
for any practical purpose, so Brier can barely tell the difference between being confidently
wrong and being absurdly confidently wrong. The log score scores the same two misses at −3.91
and −8.52, and there is no floor under it at all — a 0% forecast on something that happens is
undefined, which is why platforms that use it clamp the extremes before they will accept them.

Two practical consequences for somebody deciding whether to say 2% or 5%:

- **On a log-scored platform, the tail is worth thinking about.** The difference between 2% and
  5% is a real decision, and it is the one that separates people at the top of the table.
  Sharpness you cannot defend is the most expensive thing you can buy there.
- **On a Brier-scored platform, it barely registers** in either direction, and your rank will be
  decided by the middle of the range, where the squared error has room to move. Time spent
  agonising over a tail probability is time that scores nothing.

Neither of those is advice to state something other than your belief — both rules are proper, so
that stays the answer. It is advice about where the effort goes, and about the variance of your
season. A single confident miss at 2% costs you the equivalent of roughly five and a half
ordinary good calls at 90% under the log score, and about three under Brier.

### The Brier convention trap, which is the factor of two

This is the one that silently corrupts every cross-platform comparison, and it is not obscure:
both conventions are in live use, in documents a reader will actually be handed.

[Good Judgment Open's FAQ](https://predictionmarkets.tools/tools/good-judgment-open) sums the squared error over **every
possible outcome**. Its own worked example is a yes/no question forecast at 70% where the event
occurred:

```
(1 - 0.7)^2 + (0 - 0.3)^2 = 0.18
```

and it states the worst possible score as 2. The WMO working group's verification reference gives
the Brier score as the **mean of (forecast − observation)² over forecast/observation pairs**, and
states the range as 0 to 1. The same forecast under that form is:

```
(1 - 0.7)^2 = 0.09
```

Same rule, same forecast, two numbers, a factor of two apart. Neither document is wrong; they
are simply not the same statistic, and almost nobody publishing a Brier score says which one they
mean. [Manifold](https://predictionmarkets.tools/tools/manifold) prints "0.17375 Brier" on its calibration page with no
convention named anywhere near it, and the [forecasting platforms
listing](https://predictionmarkets.tools/categories/forecasting-platforms) has to carry the same caveat on the category as a
whole.

Three things follow, and the third is the one people get wrong:

- **A raw Brier number from one site cannot be compared to one from another** until both
  conventions are named. Not "should be compared carefully" — cannot, because one of the two
  candidate answers is exactly twice the other.
- **The quick test is a single question.** Ask what a 70% forecast on something that happened
  scores. If the answer is 0.18 it sums both outcomes; if it is 0.09 it does not. If nobody can
  answer, the published figure is not usable as a comparison.
- **The rules of thumb inherit the ambiguity.** "Under 0.25 is decent" is a statement about the
  one-outcome convention, where 0.25 is exactly what you get by saying 50% to everything. Under
  the two-outcome convention the coin flip is 0.5 and the same rule of thumb is twice as strict
  as whoever repeated it intended.

### A relative score is a statement about the crowd, by construction

Subtracting the crowd is a reasonable thing to do — it strips out how hard the questions were,
which is otherwise the largest term in anybody's score. What it produces is not a harder version
of an absolute score. It is a measurement of a gap, and a gap has two ends.

```
Metaculus Peer      (mean over all other forecasts on the question of
                     your log score minus theirs) x 100      higher is better

GJ Open Relative    (your average daily Brier
 Brier Score         minus the average daily median Brier)
                     x your Participation Rate               lower is better
```

Four differences between those two lines, any one of which is enough on its own to make the
numbers incomparable:

- **The rule underneath.** Log on one, Brier on the other, with all of the tail behaviour above
  attached to the choice.
- **The crowd statistic.** A mean of everyone else on one, the median of everyone with an active
  forecast on the other. A mean is moved by one terrible forecast; a median is not.
- **The sign.** Metaculus states plainly that higher is better. Good Judgment Open states that
  negative means you were more accurate than the crowd. The arrows point in opposite directions.
- **Whether it cancels.** Metaculus states that on any given question the sum of all
  participants' Peer scores is always 0 — zero-sum by construction. The median-and-participation
  form does not cancel, because a median is not an average and multiplying by each forecaster's
  own participation rate breaks the symmetry anyway.

The arithmetic consequence is the part worth carrying away: **your relative score moves when the
crowd moves and you do not.** Post identical forecasts to a weaker field and your number
improves. Post them against a stronger one and it falls. Two forecasters with the same relative
score on two platforms, or in two tournaments on the same platform, have told each other
nothing.

None of which makes a relative score a bad number. It is the right number for the question "was
I better than these people, on these questions". It is the wrong number for "how good am I", and
it is not a number at all for "am I better than someone over there". If what you actually want is
to pick between two of these products, [the head-to-head
page](https://predictionmarkets.tools/compare/forecasting-platform-scoring) does that comparison with the feature lists
attached.

### Profit is a different quantity, not a rougher version of the same one

On a platform with a price, the headline figure on a profile is usually profit. Profit is
position size multiplied by price movement, summed over your fills. Your stated probability is
not an input to it anywhere.

What it pays for is finding a mispriced question, getting in early while a market is thin, sizing
the position, and supplying liquidity for other people to trade against. Those are real skills,
they are difficult, and not one of them is the skill of stating a well-calibrated probability. A
trader taking many small edges on questions whose subject matter they have no view about can
out-earn a well-calibrated forecaster working on genuinely hard ones, and the leaderboard cannot
tell you which of the two you are reading.

Two features of play-money platforms change the incentive further.

**The reward arrives in a consumable.** The familiar argument for prices — that a market stays
honest because correcting somebody else's error pays — assumes the payment is worth something
outside the market. On a play-money platform it is not: Manifold's own FAQ describes mana as the
play-money currency used on the platform and says it cannot be converted to cash. Correcting a
price therefore buys you more of the same site and nothing else. That is not an argument that the
incentive fails — a permanent public record is a real motivation and the platforms publish data
you can examine — but it is a different argument from the one people think they are making, and
it should be made on its own evidence.

**On some platforms the question is resolved by the person who wrote it.** Manifold's FAQ puts it
in five words: whoever created the market gets to resolve it. That is a deliberate design choice
and it is what makes questions cheap to list, but it changes what a profit figure records. Part
of it is a verdict on how well a series of strangers read resolution criteria they wrote
themselves. The four different answers venues give to "who decides" have [their own
page](https://predictionmarkets.tools/guides/who-decides-the-outcome), and it is worth reading before you treat any settled
question as a scored one.

A third caveat sits underneath both: in a thin market the price is not a probability in the first
place, which [where the liquidity comes from](https://predictionmarkets.tools/guides/where-liquidity-comes-from) takes apart in
detail.

### What a platform-wide score is a claim about

The last step is aggregation, and an aggregate is a claim about the platform, not about the
question open in front of you.

Manifold's calibration page is the most useful example precisely because it publishes its method.
The figure is 0.17375, beside a stated sample size of 98k trades, produced by sampling 2% of past
trades hourly on resolved binary questions with 15 or more traders, grouping them by probability
band and checking how often each band resolved yes. The page notes that this is trade-weighted
rather than time-weighted calibration, and adds that market accuracy may be better than reflected,
because large miscalibrated trades are usually corrected immediately.

Count the decisions in that paragraph: a trader threshold, a sampling rate, a weighting choice,
and a caveat about what the weighting does to the result. Four, all disclosed, plus the Brier
convention which is not. Most published aggregates disclose none of them and print one number.

Then there is selection, which no methodology note can fix. **A corpus of resolved questions is
not a random sample of questions asked.** Anything that was voided, annulled, left open or
quietly delisted is out of the average, and those are disproportionately the ambiguous ones — the
questions where the criteria turned out not to survive contact with the world. A platform whose
question supply is chosen by research funders has a different corpus from one where anyone lists
anything in thirty seconds, and the two aggregates are measuring different populations even
before they reach a formula.

## What it costs

None of these platforms charges money for a score. What a score costs is attention, in units the
formula picks rather than the ones you would pick, and the bill is mostly invisible until you
have already paid it.

- **Days, not answers.** Metaculus states that predictions count towards scores proportionally to
  their duration, and that earlier predictions accumulate more weight the longer they stand.
  Good Judgment Open computes a Brier score for each day you had an active forecast and averages
  across those days. A perfect forecast entered the day before a question closes earns almost
  nothing under either. The unit of work is the day the forecast is standing, which means
  maintaining a position you already hold is most of the job.
- **Participation, priced explicitly.** Good Judgment Open multiplies your relative score by your
  Participation Rate — the share of possible days you had a forecast in. Half the coverage halves
  the number, in both directions, so breadth and early entry are paid for at the same rate as
  being right.
- **Variance, on a log-scored site.** One confident miss at 2% costs roughly five and a half
  ordinary good calls at 90%, measured against a coin flip. Under Brier the same miss costs about
  three. If you are ranked by a log-based score, your season is decided by your worst handful of
  forecasts more than by your median one.
- **Non-transferability, which is the largest cost and the last one noticed.** A record built
  under one formula does not convert into a record under another. Three seasons spread over three
  platforms produce three short histories under three incomparable rules, which adds up to no
  track record at all. Choosing a formula is therefore a commitment, and it is worth making
  deliberately at the start rather than discovering a year in that the number you have been
  growing does not say what you want it to say.

## What you can do about it

**Put four questions to a platform before you chase its number.** Each one has a factual answer
that a support desk or a docs page can give, and a platform that cannot answer one of them has
told you something.

1. **Which scoring rule?** Brier, log, something derived from one of them, or none of the above.
   If the answer is a hit rate, a streak, or "points", it is not a proper rule and the optimal
   strategy under it is not forecasting. If the answer is profit, it is not a scoring rule at all
   — it is a trading result, which is a different quantity and a fine one, just not this one.
2. **Which Brier convention?** Ask what a 70% forecast on something that happened scores. 0.18
   means both outcomes are summed and the worst case is 2; 0.09 means one outcome and a worst
   case of 1. Every published figure differs by a factor of two between the two answers, so until
   you have this, do not compare the platform's number to any other number anywhere.
3. **Who resolves the questions the score was earned on?** Platform staff against published
   criteria, a named external source, or the person who wrote the question. The last one is not
   disqualifying, but it means part of your score measures other users' drafting. The full
   taxonomy is in [who decides the outcome](https://predictionmarkets.tools/guides/who-decides-the-outcome).
4. **Is there a calibration curve you can look at yourself?** Not a claim of accuracy — a plot,
   with the number of forecasts behind it, the date range, and the rule by which they were
   sampled. A site that publishes a curve with a denominator has handed you the means to disagree
   with it. A site that publishes a record with no denominator has handed you a slogan.

**Score yourself before you trust anyone else's arithmetic.** Keeping your own log is the only
way to know your own convention and your own corpus, and it makes the platform's number a second
opinion rather than the only one. [Fatebook](https://predictionmarkets.tools/tools/fatebook) writes down private questions,
computes Brier and relative Brier scores and draws a calibration chart; [Quantified
Intuitions](https://predictionmarkets.tools/tools/quantified-intuitions) scores you on questions that already resolved, which
compresses the feedback loop from months to minutes.

**For any claim that spans platforms, use a scorer that matched the questions by hand.**
[Brier.fyi](https://predictionmarkets.tools/tools/brier-fyi) scores hand-matched questions across four venues and publishes its
pipeline as open source, which is the only form in which a cross-venue accuracy claim can be
checked — and its card also records that the pipeline is currently paused, which is the sort of
fact a scoreboard rarely prints about itself.

**When you quote your own score, quote the formula with it.** "Relative Brier of −0.03 on Good
Judgment Open, 40 questions, participation rate 0.7" is a statement somebody can evaluate. "My
Brier is 0.09" is two different claims with the same words, and the reader cannot tell which one
you made.

**And put the volume in one place.** The formula you choose is the thing you will get good at
over a season, so choose it first and let the feature lists follow. If the decision in front of
you is between specific products rather than between formulas, [the three-way
comparison](https://predictionmarkets.tools/compare/forecasting-platform-scoring) makes it with the access, the resolution
policy and the question supply attached; the [forecasting platforms
listing](https://predictionmarkets.tools/categories/forecasting-platforms) has the rest of the field.

## Tools this bears on

- [Metaculus](https://predictionmarkets.tools/tools/metaculus.md) — Proper scoring and public track records on questions nobody can take a position in.
- [Good Judgment Open](https://predictionmarkets.tools/tools/good-judgment-open.md) — Brier scores against the crowd, and the recruiting ground for Superforecasters.
- [Manifold](https://predictionmarkets.tools/tools/manifold.md) — Anyone can open a question, anyone can take a side, and the currency buys nothing.
- [Brier.fyi](https://predictionmarkets.tools/tools/brier-fyi.md) — Brier scores and letter grades for matched questions across four platforms.
- [Fatebook](https://predictionmarkets.tools/tools/fatebook.md) — Write down what you think will happen, in Slack or a browser, and get scored on it.

## FAQ

### What does it mean that a scoring rule is proper?

That you maximise your expected score by quoting the probability you actually believe, and that no other answer does better on average. Gneiting and Raftery give the formal version, and call propriety essential in scientific and operational forecast evaluation. Under an improper rule the best strategy and the honest one come apart, and the platform is training you towards the first.

### Why do two platforms report different Brier scores for the same forecast?

Because there are two conventions in circulation and almost nobody names theirs. One sums the squared error over every possible outcome, so a yes/no question has a worst case of 2 and a 70 percent forecast on something that happened scores 0.18. The other scores the one outcome, tops out at 1, and gives 0.09 for the same forecast. Every figure differs by a factor of two between them.

### Can I compare a crowd-relative score with one from another platform?

No, and not because the platforms are being unhelpful. A relative score subtracts a crowd, so it is a measurement of the gap between you and the specific people on those specific questions. Change the crowd and your number moves while your judgement does not. The rule underneath, the crowd statistic, the sign convention and whether the scores cancel to zero all differ between sites as well.

### Is a profit number on a play-money platform a measure of accuracy?

It is a measure of trading. Profit is position size multiplied by price movement, summed, and your stated probability appears nowhere in it. It rewards finding mispriced questions, being early in thin ones, sizing and supplying liquidity. Those are real skills and none of them is stating a well-calibrated probability.

### What does a platform-wide Brier score tell me about the question in front of me?

Very little on its own. It is an average over a corpus of questions that reached resolution, under methodological choices that are usually not printed beside the number. Ask for the denominator, the date range and the sampling rule before reading it as a claim about anything, and remember that questions which were voided or left open are not in the average at all.

## Sources

1. [Strictly Proper Scoring Rules, Prediction, and Estimation](https://sites.stat.washington.edu/raftery/Research/PDF/Gneiting2007jasa.pdf) — Journal of the American Statistical Association, volume 102, number 477, 2007-03-01. A mathematical definition rather than a rulebook, and still the standard statement of what makes a scoring rule proper.
2. [Forecast verification - methods, issues and FAQ](https://jwgfvr.github.io/forecastverification) — WMO WWRP Joint Working Group on Forecast Verification Research, read 2026-09-21
3. [Scores FAQ](https://www.metaculus.com/help/scores-faq/) — Metaculus, read 2026-09-21
4. [FAQ](https://www.gjopen.com/faq) — Good Judgment Open, read 2026-09-21
5. [Calibration](https://manifold.markets/calibration) — Manifold, read 2026-09-21
6. [Manifold FAQ](https://docs.manifold.markets/faq) — Manifold, read 2026-09-21

*Last updated 2026-09-21. A reference page, corrected in place — not a dated post.*