Brier score

This catalogue uses the Brier score as a unit more than as a subject. Eleven cards carry the calibration-scoring flag, and most of them either print a Brier figure, compute one, or explain why they do not. The trap everyone runs into first is the one where two conventions differ by a factor of two, and it has its own treatment, including the one-question test that tells you which convention a platform uses. This page covers the parts that guide does not: where the two conventions come from, why only one of them handles a question with more than two answers, and why the same forecaster gets a different average on a different set of questions.

How it works

Glenn Brier defined it in the Monthly Weather Review in January 1950 for rain forecasts. On each of n occasions the event falls into one of r mutually exclusive classes. The forecaster states a probability for every class, and the score is the squared difference between each stated probability and 1 or 0 (whether that class happened), summed over the classes and averaged over the occasions. Brier states the range himself: zero for perfect forecasting and a maximum value of 2 for the worst possible forecasting.

So the 0-to-2 scale is not a variant. It is the original. The 0-to-1 form, which the WMO verification reference gives as the mean of (forecast − observation)², is what you get when a yes/no question is written as one probability instead of two. Both are the same rule. Gneiting and Raftery list it as the quadratic score, proposed by Brier (1950), and it is proper under either scaling.

Only the original handles more than two answers

Good Judgment Open publishes a three-option worked example: 60% on A, 10% on B and 30% on C, and A happens.

(1 - 0.6)^2 + (0 - 0.1)^2 + (0 - 0.3)^2 = 0.16 + 0.01 + 0.09 = 0.26

The single-probability form has nothing equivalent to offer here. If you score only the option that happened, you get 0.16, which is not half of 0.26. The two conventions stop being related by a factor of two as soon as a question has a third answer. This is why a tool that keeps the single-probability form has to do something else with multiple-choice questions. Brier.fyi, for example, splits them into yes/no pieces or tracks only the winning option, depending on the platform.

The score has a floor, and the questions set it

Brier's paper makes a second point that a leaderboard never repeats. If a forecaster issues the same forecast on every occasion, the best score available to them comes from forecasting the observed frequencies, and it equals 1 minus the sum of the squared frequencies. His own example has ten forecasts, three of which saw rain. A forecaster who said 30% every time would score 0.42, a forecaster who used a climatological 20% would score 0.44, and the actual forecaster scored 0.19.

On a set of yes/no questions where a fraction p resolves yes, that no-skill figure is:

The last column is p(1 − p), the "uncertainty" term in the decomposition that the WMO reference credits to Murphy (1973). It depends on the questions and not on the forecaster. Gneiting and Raftery draw the consequence: scores are directly comparable only when they refer to exactly the same set of forecast situations. Otherwise, how predictable the questions were gets mixed up with how well they were predicted.

What the catalogue records

For the cards that touch a Brier score, this is the convention each one uses, and how the corpus knows:

Brier.fyi also cuts its letter grades on the 0-to-1 scale, with an F running from 0.36 to 1. A 0-to-2 figure has to be halved before it is read against that table.

Confido's formula, (0.25 − (outcome − p)²) × 4, is 1 minus the one-option Brier score divided by 0.25, the score of a 50% forecast. In other words, it is a skill score against a fixed coin flip. On yes/no questions all three scales convert exactly. The 0-to-2 figure is twice the 0-to-1 figure, and Confido's score is 1 minus four times the 0-to-1 figure. Nothing converts across a multiple-choice question. Metaculus is absent from the table on purpose, because its scores are built on the log score and its scores page does not publish a Brier figure.

Why it matters here

Two numbers from two platforms are two different measurements until you have named both conventions. Fatebook and Good Judgment Open agree with each other and with Brier's paper. Brier.fyi uses the other scale, and Confido uses a third one pointing the opposite way. If you are choosing where to keep a track record, you are choosing which of these your history will be written in. Moving it later means a conversion that works for yes/no questions only.

A platform-wide Brier score is partly a statement about the question mix. Manifold's 0.17375 is exactly what a forecaster with no skill would score if about 22% of the sample resolved yes and the scale is 0 to 1, or if about 10% did and the scale is 0 to 2. It is well clear of the no-skill figure on either scale if the sample was split evenly. Neither the yes-share nor the convention is printed beside it, so there is no way to tell which of those it is. For any aggregate, the missing figure is the share of questions that resolved yes. Without it, a low average may be showing you easy questions rather than a good forecaster.

Your own number moves when your questions change. A private log in Fatebook full of near-certain personal questions will average far lower than a season of hard geopolitics on Good Judgment Open, even with identical judgement. The crowd-relative scores on both sites exist to strip that out, and their limits are covered in the guide.

Brier is bounded, and that decides what it can tell apart. It cannot tell a confident miss from an absurdly confident one, which is the log score's territory. Whether any of these numbers rewards an honest forecast at all is a property of the rule, and that is the subject of proper scoring rule.