What to use instead of Replication Markets

The product

Replication Markets asked a question most forecasting sites never touch: will a finding that has already been published hold up when somebody runs the study again? It was a research project inside DARPA's SCORE programme — Systematizing Confidence in Open Research and Evidence — and its forecasts were data, collected to help train machine models that score the reliability of research.

It ran in rounds. Each round drew claims from behavioural and social-science journals published between 2009 and 2018. For one week forecasters filled in a survey, giving their own estimate of each claim's chance of replicating; then a market opened on the same claims, and the crowd moved a shared probability with its trades. The project wanted both from the same people, so it could compare the survey with the market. The survey week alone carried $4,800 in prizes a round, paid in batches of nine or ten claims — $80 to the top forecaster, $40 to the second, $20 each to the third and fourth. The home page promised at least $4,000 in payouts each month and counted over $100,000 awarded through 2020.

The schedule page shows rounds 6 to 11 running through 2020. Round 11, on COVID-19 claims, closed its market on 28 September. A separate market on 400 COVID-19 preprints followed, funded in part by the Fetzer Franklin Fund on the SCORE platform. For each preprint it asked four questions: where the paper would rank by citations a year on, and whether within a year it would be published in a journal with an impact factor of 10 or more, published below that, or not published at all. It carried $14,520 in prizes.

How it ended, with dates. The preprint market's surveys ran from 28 October to 10 November 2020 and its market from 11 November to 20 November 2020 at 12:00 UTC; afterwards its page said data collection was complete. The home page then said the project "does not plan to run any forecasts in 2021" and would resolve outstanding claims, pay prizes and analyse the results. On 16 November 2021 it announced 121 resolutions for the SCORE markets — 79 regular replications, 30 COVID-19 replications and 12 meta-claims — and contacted 258 winners to claim $142,623; the claim window has since closed. Preprint winners were announced on 3 February 2022. The home page's last word on the project is "RM is done". On 25 October 2026 the domain answered with an HTTP 522 error.

This page records 20 November 2020 as the end, because that is the last moment anyone could forecast on it.

What to use instead

It was three things at once: a subject (whether social science holds up), a mechanism (a market with cash for accuracy), and a score that followed you. No product here does all three, so pick by the one you came for.

If you came for the subject, go to the Social Science Prediction Platform, the one Replication Markets named itself. The object is close but not the same. There you forecast what a study will find before its results are reported, and you are ranked by your error against the estimate the researchers publish. That is a prior on a new result, not a forecast of whether an old result survives a second run. Anyone 18 or over can forecast free; the paid panel is limited to graduate students, faculty and researchers.

If you came for the market with prizes, Hypermind is the same shape. You trade a play currency on an order book, and a fixed prize per contest is split among the forecasters who finished with a positive performance, paid in Amazon gift certificates. The questions are Hypermind's — politics, economics, business, science and sports — and none of them is a replication claim.

If you came for a public score, Metaculus scores every forecast on a proper rule and keeps a track record on your profile that a stranger can read. Money arrives only through tournaments, each with its own prize pool and rules. Its questions include science, but it does not run replications, so a claim resolves when the source named in its criteria reports.

If you have one paper in mind, Manifold lets anyone open a question and resolve it. You can have a market on a specific replication by tomorrow. The price is play money and the resolution is whoever opened it, so the number is only as good as the people who turn up.

What you give up

Cash for being right about replication, open to anybody. The SCORE markets offered $142,623 in prizes to 258 winners, with no credential asked for. The nearest subject match pays only its researcher panel, and the nearest mechanism match pays in gift certificates on questions about something else.

Resolution by a second run of the study. A Replication Markets claim was settled when somebody actually tried to reproduce the finding. The Social Science Prediction Platform scores you against the study's own reported estimate, which can take years, and some studies never report. Metaculus settles on whatever source the question names. Nothing here pays you for predicting the outcome of a replication that someone has been funded to run.

The survey beside the market. Replication Markets asked the same crowd the same question twice, once alone and once through prices, so the two could be compared. Every pick here does one or the other.

Migration notes

Nothing carries over. Your Replication Markets history does not become a score anywhere else, and every pick starts you at zero. What a track record is worth off the venue is the honest version of how much that history can still do for you.

The archived pages are the only record left. The domain was not answering on 25 October 2026. Copies of the schedule, the rules and the resolution posts are in the Internet Archive; there is no account to log into and no export to request.

Scoring is not the same unit. A market price and a survey estimate on Replication Markets, a ranked error on the Social Science Prediction Platform, and a log score on Metaculus are three different numbers. What a forecasting score measures explains why a good number on one does not translate to the next.