# Proper scoring rule

Also written proper score, strictly proper.

*https://predictionmarkets.tools/glossary/proper-scoring-rule · next to Forecasting Platforms*

**Definition:** A rule for scoring a probability under which stating the probability you actually hold earns the best expected score, and no other report does better on average. It is strictly proper when nothing else ties. Brier and log scores are both proper. The property belongs to the per-question rule. A platform can rescale it or subtract a crowd without losing it, but a benchmark computed from the same sample, or a nonlinear prize on top, can break it.

The category page gives the one-sentence version, and [the guide on forecasting
scores](https://predictionmarkets.tools/guides/what-a-forecasting-score-measures) gives the definition, the classic counterexample
of ranking by hit rate, and how Brier and log differ at the extremes. None of that is repeated
here. This page is about something else: no platform shows you a raw rule. Each one rescales it,
subtracts a crowd, averages it over days, multiplies it by participation, or pays prizes on top of
it. The useful question when you choose one is which of those steps keep the rule proper and which
do not.

## How it works

The idea is older than the name. Glenn Brier's 1950 paper opens with the problem of a forecaster
"hedging" or "playing the system", issuing something other than what he thinks will occur because
the score is easier to analyse than the weather. It sets out to describe a scheme that cannot
influence the forecaster in any undesirable way. Gneiting and Raftery date the general idea to
Brier (1950) and Good (1952). They credit the word *proper* to Winkler and Murphy (1968), and give
the definition every platform now paraphrases: a rule is proper if the expected score is maximised
by issuing the forecast you actually believe, and strictly proper if that maximum is unique.

### What a platform can do without breaking it

The same review gives the exact licence, and it is short. If S is proper, then **c × S + h** is
proper too, for any positive constant c and any term h that depends only on the outcome. Strict
propriety survives the same way. Most of what the forecasting cards here do to a rule fits inside
that one line:

- **Rescaling.** Metaculus's Baseline score is the log score rescaled so that giving every outcome
  the same probability scores 0 and a perfect binary forecast scores +100. [Confido](https://predictionmarkets.tools/tools/confido)
  computes `(0.25 − (outcome − p)²) × 4`, which is a Brier score multiplied by a negative constant
  and shifted by a fixed amount. Flipping the sign only flips which direction counts as better.
- **Subtracting other people.** Metaculus's Peer score is the average difference between your log
  score and the log scores of all *other* predictions on the question, times 100. The subtracted
  term is fixed by the outcome and by forecasts that are not yours, so it plays the role of h.
- **Multiplying by participation.** Good Judgment Open multiplies your score minus the crowd's
  median by your Participation Rate. On a question you are already in, that is a positive c.

Rescaling to a *fixed* reference is safe. Rescaling to a reference computed from the same data is
a different operation. Gneiting and Raftery note that skill scores of that form are generally
improper even when the rule underneath is proper. For the Brier skill score, Murphy (1973) showed
it is only asymptotically proper: the gain from hedging shrinks towards nothing as the number of
independent forecasts grows. Confido's constant 0.25 is on the safe side of that line. A "skill"
figure normalised against the sample's own base rate is on the other.

### What a prize does to it

Metaculus's tournament rules are unusually candid on this point. Rank is the question-weighted sum
of Peer scores. The share of the prize pool is proportional to that sum **squared**, and a negative
sum pays nothing. The page then says that for a tournament with a sufficiently large number of
independent questions, the method is *essentially* proper.

Both qualifiers are there for a reason. Squaring a sum and cutting it off at zero makes the prize
a convex function of the score. That rewards a spread-out result more than a steady one with the
same average, until enough independent questions narrow the spread and the effect fades. Propriety
is a promise about the expected **score**. Money paid as a nonlinear function of the score is a
separate object, and it keeps that promise only approximately and only at scale.

### Point estimates have a version of this too

A rule for a single number rather than a probability cannot be proper in this sense. Its analogue
is being *consistent* for some summary of your belief. Gneiting's paper on point forecasts shows
that squared error rewards reporting your mean, while absolute error is consistent for the median.
[The Social Science Prediction Platform](https://predictionmarkets.tools/tools/social-science-prediction-platform) ranks by mean
absolute error on effect sizes, so the report its leaderboard pays for is your median.

## Why it matters here

**"Uses a proper scoring rule" on a card is a claim about one layer.** The rule scores each
question. The leaderboard aggregates those scores, and a prize or a recruitment decision sits on
top of the leaderboard. When you choose between [Metaculus](https://predictionmarkets.tools/tools/metaculus) and [Good Judgment
Open](https://predictionmarkets.tools/tools/good-judgment-open), the rule underneath (log or Brier) and the thing actually paid
out (a squared share of a pool, or a rank against a crowd median) are two separate things to
check.

**Transformations are fine; home-made benchmarks are not.** A rescaled number, a sign-flipped one
or a crowd-relative one is still an honest measurement, even when it looks nothing like the
textbook formula. A figure normalised against its own sample, or a payout that grows faster than
the score, is where to ask the platform what it computes.

**Not everything in the listing is a rule of this kind.** Of the eleven cards in [the
calibration-scoring collection](https://predictionmarkets.tools/collections/calibration-scoring), several score something other
than a stated probability. [Hypermind](https://predictionmarkets.tools/tools/hypermind) ranks by square roots of profits and
losses. Prediki publishes no formula. The Social Science Prediction Platform scores point estimates.
Manifold's profile figure is profit. [Quantified Intuitions](https://predictionmarkets.tools/tools/quantified-intuitions), by
contrast, states that its relative log score is strictly proper. A rank under the first group is
not wrong. It is simply a different measurement from a rank under the second.

The scale a Brier-based number is printed on is its own problem, covered under [Brier
score](https://predictionmarkets.tools/glossary/brier-score).

## Where you will meet this

- [Good Judgment Open](https://predictionmarkets.tools/tools/good-judgment-open.md)
- [Manifold](https://predictionmarkets.tools/tools/manifold.md)
- [Metaculus](https://predictionmarkets.tools/tools/metaculus.md)
- [Quantified Intuitions](https://predictionmarkets.tools/tools/quantified-intuitions.md)
- [Social Science Prediction Platform](https://predictionmarkets.tools/tools/social-science-prediction-platform.md)

## FAQ

### Is a crowd-relative score still a proper scoring rule?

Yes, when the crowd term is one your own forecast does not enter. Subtracting anything that depends only on the outcome leaves a proper rule proper. Metaculus's Peer score subtracts the log scores of all other predictions on the question. Good Judgment Open subtracts a median taken over everyone with an active forecast, so in a small crowd your own forecast can move it and the argument holds only approximately.

### Is a Metaculus tournament prize proper?

Metaculus says it is essentially proper for a tournament with a sufficiently large number of independent questions. The qualifiers matter. The prize share is proportional to the square of your summed Peer scores, and is zero if that sum is negative. That payout is not a linear function of the score, so on a short or correlated question set, what maximises the expected prize and what maximises the expected score are not the same report.

### Is mean absolute error a proper scoring rule?

Not in this sense, because it scores a single number rather than a probability. It has an analogue of its own. Gneiting shows that absolute error is consistent for the median, meaning the report it rewards is the median of what you believe, while squared error rewards the mean. A platform scoring point estimates by absolute error is therefore asking for your median.

## Sources

1. [Strictly Proper Scoring Rules, Prediction, and Estimation](https://sites.stat.washington.edu/raftery/Research/PDF/Gneiting2007jasa.pdf) — Journal of the American Statistical Association, volume 102, number 477, 2007-03-01. A mathematical review rather than a rulebook, and still the standard statement of what makes a scoring rule proper.
2. [Verification of forecasts expressed in terms of probability](https://journals.ametsoc.org/view/journals/mwre/78/1/1520-0493_1950_078_0001_vofeit_2_0_co_2.xml) — Monthly Weather Review, volume 78, number 1, 1950-01-01. Cited for its stated purpose, a score that cannot reward hedging, which is a historical fact about the paper and cannot go out of date.
3. [Scores FAQ (Wayback Machine capture of 17 July 2026)](https://web.archive.org/web/20260717094118/https://www.metaculus.com/help/scores-faq/) — Metaculus, read 2026-09-26
4. [FAQ](https://www.gjopen.com/faq) — Good Judgment Open, read 2026-09-26
5. [Making and Evaluating Point Forecasts](https://arxiv.org/abs/0912.0902) — Tilmann Gneiting, arXiv, read 2026-09-26

*Last updated 2026-09-26. A reference page, corrected in place — not a dated post.*