How to track and score your own forecasts

The short way

The record is made at the moment you write the forecast down, not at the moment somebody scores it. Three things have to be fixed before the answer is known: the probability, the resolution criterion written out in enough words that a stranger could apply it, and the date by which it is settled. Everything below is machinery for storing those three and doing arithmetic on them later.

Fatebook is the shortest path to having them stored. A question is a title, a probability and a resolve-by date; it is private until you share it; the platform reminds you when the date arrives, and resolution is YES, NO or AMBIGUOUS. A Slack command opens a question to a channel, and a browser extension opens one from any page. The API key is a query parameter rather than a header, which is why the integrations people have built on it are Alfred workflows and iOS shortcuts rather than services — writing a forecast down fits in a shell one-liner.

The scoring is where the detail is. The source walks the question day by day from creation to resolution, computes a time-weighted average of the forecasts you made within each day, scores that, and averages the days — so a forecast left standing counts for every day it stood, and updating early is worth more than updating correctly at the end. Beside your absolute Brier score it computes a relative one, subtracting the median of everyone else's score on that question that day.

Record which platform produced the number, in the same place you record the number. There are two Brier conventions in circulation and they differ by a factor of two on every figure:

# The same forecast: 70% on something that then happened.
p, outcome = 0.7, 1

single = (p - outcome) ** 2                                   # 0.09, bounded at 1
both = (p - outcome) ** 2 + ((1 - p) - (1 - outcome)) ** 2     # 0.18, bounded at 2

Fatebook's source computes the second one. A score copied into a spreadsheet without its provenance is a number you will not be able to interpret in a year, which is the failure this whole exercise exists to avoid.

What the options are

A tracker you own. Fatebook above, hosted by somebody else and free, with a one-click export of everything you have ever forecast. Confido is the version you run: rooms, members and questions on your own server under AGPL, which is the answer when the questions themselves are the sensitive part. Two things about it are unlike everything else here. Its score is a rescaled Brier — the source computes (0.25 − (outcome − p)²) × 4, which runs from −3 to 1, sits at 0 for a 50% forecast and is higher-is-better, the opposite direction from every other score on this page. And automatic scoring covers binary questions only: ask for a numeric range and the scoring function returns nothing, leaving the CSV export and your own arithmetic. Confido also has a score time on each question, and a prediction counts only if it was submitted before that moment.

Somebody else's questions, and a score with a public method. Metaculus resolves against criteria published on the question, annuls a question whose criterion turns out not to be evaluable, and scores on the log score under two rescalings. The Baseline score compares your forecast to a fixed chance benchmark. The Peer score compares it to the geometric mean of the other forecasters live at the same moment, scaled by the number of them, which makes it the harsher and more informative figure. Both are weighted by coverage — the share of the question's life your forecast was standing — and both have a spot variant that reads a single moment instead. The API now needs an account token on every request, so plan for the record to live on the site rather than in your own store.

Good Judgment Open is the Brier-based equivalent, organised into Challenges with their own leaderboards and sponsors. Its Relative Brier Score is your average daily Brier minus the crowd's average daily median, multiplied by your participation rate; lower is better, negative beat the crowd, and a question you skipped scores zero rather than counting against you. The property worth the most here is procedural rather than statistical: there is no way to withdraw from a question or delete a forecast. A record you cannot tidy afterwards is the only kind worth keeping.

Practice, while the real record accrues. Quantified Intuitions serves you questions that already resolved, presented as they stood on a vantage date with the crowd forecast hidden and a search engine that filters out anything published after it. You learn in ninety seconds instead of eighteen months, scored by a log score relative to the crowd of the day. It is training, not a track record, and the site does not pretend otherwise.

Where this breaks

A Brier score without its convention is not a score. Fatebook sums the squared error over both outcomes and is bounded at 2; the single-outcome convention is bounded at 1; Confido rescales to −3 to 1 and flips the direction of "good". Three numbers, three meanings, and none of them prints the convention beside itself. What a forecasting platform's score actually measures is the page for the arithmetic; the operational version is to store the platform name in the same row as the figure, forever.

A crowd-relative score describes the crowd, not you. Fatebook subtracts the median of the other forecasters on that question; Good Judgment Open subtracts the crowd's daily median and scales by participation; Metaculus divides by the geometric mean of everyone live at that moment. Change the crowd and your number moves while your judgement stands still, which is why none of these travels to another site and why a good relative score on a thin question set is worth less than it looks. It has a sharper edge on a private tracker: a question only you forecast has no crowd to subtract, so the relative score is simply absent. Which score you are chasing is the head to head.

A record made of easy questions is not a better record. Nothing in a Brier or log score knows how hard the question was — that is precisely the work a crowd-relative score does, and it is the thing you give up by asking your own questions. Twenty instances of "will I ship this by Friday" resolving YES produce a calibration chart with a confident-and-correct bar and no information in it. The cheap defences are to tag questions and read the chart per tag rather than in aggregate, to write the question when it is genuinely uncertain rather than when it is nearly settled, and to keep some fraction of your volume on a question set somebody else chose.

On your own questions you are the forecaster and the judge. Fatebook's author resolves, with no moderator, no adjudication and no appeal; on Confido whoever runs the room enters the outcome. Nothing stops a question being resolved generously and nobody will notice. The discipline that replaces the missing referee is written before the fact: name the source that will decide, in the question, and use the AMBIGUOUS button — the equivalent of a Metaculus annulment — rather than forcing an unanswerable question to the answer that flatters the record. Who decides how a prediction market resolves is the same problem where money is involved.

Profit on a play-money platform is not accuracy. Manifold is the place this bites, because its profile number is mana profit and it reads like a score. Profit is position size multiplied by price movement, summed; your stated probability appears nowhere in it, and it rewards finding a mispriced question, being early in a thin one and sizing well. Those are real skills and none of them is stating a well-calibrated probability. Manifold does publish a platform-wide calibration curve, but that is a claim about the platform rather than about you — and on 19 September 2026 the figure on that page carried no unit, which is the first problem on this page all over again.

If you outgrow this

If the record has to be read by a research audience, the shape of the artefact changes. Scientific and government forecast hubs are open, scored against surveillance data and published as dated CSVs — but the unit of submission there is a team-and-model pair, not a person, so there is no personal record to build. The Social Science Prediction Platform is the nearer thing: forecast what an unpublished study will find, and be ranked by error against the estimate it eventually reports.

If the record is meant to lead somewhere, Good Judgment Open is the one with a published destination — recruitment each autumn from forecasters who have answered at least 100 questions, average accuracy per closed question weighted most heavily, then comment quality, then a three-month probation.

And if the question you actually want answered is whether the prices were any good rather than whether you were, that is a different measurement entirely: Brier.fyi scores four platforms against each other on hand-matched questions, with its pipeline paused and its method published. Everything here that computes a score is the wider listing, and it is worth reading before assuming that two of those numbers can be put side by side.