Logbook · 3 of 4. Written 3 November 2026; the numbers are as of that date and will age.
The previous entry measured arbitrage on Polymarket: real, persistent and worth about $20 a day. This one is about Faro's second question: do outside forecasters know something market prices don't? If a well-calibrated crowd says 30 % where the market says 50 %, and the crowd is usually right, that gap is a signal. It is a statistical signal, not arbitrage, and Faro labels it that way.
The test was fixed before we looked: compare each source's forecast with the market's price at the same moment, on questions that have since settled, and score both against what happened. To pass, a source needed at least 300 questions, a skill score at least 3 % better than the market's with a confidence interval clear of zero, and a forecast that does not simply copy the price.
The hard part was not the forecasts
Outcomes turned out easy. For a question linked to a market we use what the market actually paid, at the moment the settlement became public. That is the outcome a trade would have earned, and it is known for far more questions than any forecasting site's own resolution files: seven times as many in our data.
Finding independent forecasters was harder. The benchmark datasets we hold mostly compare the market with itself. The genuinely independent crowds we could reach were Manifold, a large play-money prediction site, and Metaculus, a forecasting community whose research access we have asked for but not yet received. Neither says which Polymarket or Kalshi market asks the same question, so every question had to be matched first.
Matching is where the work went. Against nearly a million markets, a question like "Will the Fed cut rates in December?" has dozens of near-twins. Our first matcher, built on word overlap, numbers and dates, was confident about 51 pairs. The second learned to read names ("Fed" is "Federal", "GOP" is "Republicans"), to refuse a market that names someone the question does not, and never to pair a two-event question with a one-event market. We pulled every Manifold market since July 2024 with at least ten traders: 18,992 markets and 2.1 million bets, nine hours of polite fetching. The matcher then proposed 212 confident pairs.
Then we read all 212
On questions worded like their market, the matcher was right 95 % of the time. Across sources it was right 80 % of the time. Every one of its mistakes had the same shape: almost the same words, a different event.
| The question asked | The market asked |
|---|---|
| Will the Thunder win the NBA Finals? | Will the Thunder reach the NBA Finals? |
| Unemployment at least 4.5 %? | Unemployment exactly 4.5 %? |
| A US–Iran nuclear deal? | An Iran nuclear test? |
| France will NOT win the World Cup | Will France win the World Cup? |
No rule about names or numbers can see that; a person can, in a few minutes. So the test scores only pairs a person has confirmed: 169 the same, 3 the exact opposite (scored with the price flipped), the rest dropped.
Reading them showed something bigger. Much of Manifold's overlap with Polymarket is Manifold copying Polymarket. Twenty-five questions say so in their title ("[Polymarket]"), many more repeat the market's headline word for word, and several Manifold markets often copy the same Polymarket one. We call these mirrors. People forecasting on a mirror can see the real-money price, so their forecast may be that price handed back. Mirrors and independently written questions were scored separately.
What the scores showed
Every week from July 2024 we took each source's latest forecast and the market's latest price, and scored both once the question settled. 113 markets had both and had settled. The Brier score measures error (lower is better); the skill score is how much better the source did than the market.
| Group | Markets | Source error | Market error | Skill vs market (95 % range) |
|---|---|---|---|---|
| Manifold, independent questions | 48 | 0.095 | 0.081 | −17 % (−37 % to −4 %) |
| Manifold, mirrors | 62 | 0.069 | 0.062 | −11 % (−58 % to +11 %) |
| Metaculus | 3 | — | — | too few to say anything |
- On the questions Manifold wrote for itself, it was clearly worse than the market, at every horizon from a week to beyond three months. The whole range of plausible values is below zero, and a second test agrees (p < 0.001).
- On mirrors it was also worse, but too noisy to be sure.
- Neither group copies the price outright. Both follow it closely (correlation 0.93) and add noise.
The independent questions were the best chance: a crowd forecasting something on its own terms could have known something the market did not. That is exactly where it lost most clearly. On these events, the play-money crowd was the weaker forecaster and the real-money market the stronger one.
What we decided
Formally, the result is inconclusive: 113 markets is well short of the 300 the test asked for, and the rule for stopping a line of research needs 100 in one group. We report it as inconclusive rather than lower the bar.
But the evidence inside it points one way, and we acted on it: we will not build a forecast-driven strategy on Manifold. What stays is the machinery: the matcher, the hand review and the scoring are built and tested, and a better source can be put through them in hours. If Metaculus answers, or Faro produces forecasts of its own, the question reopens.
The caveats, plainly: 48 markets is a small sample, and they were chosen by the people who wrote the questions, not at random. The market's price came in 12-hour steps and we compared once a week. Neither rescues Manifold as a source of edge; together they mean the size of the gap is uncertain, not its sign.
What comes next
Two experiments have now answered, and neither found an edge worth building on. That is the tool doing its job.
- Recording the venues ourselves. There is no public archive of Kalshi's order books, and the one we used for Polymarket has gaps. A lean recorder starts our own history now, so the next questions have data that was not there before. Work on it starts this week.
- Q3, cross-venue. Polymarket and Kalshi have different users, rules and fees, so they should disagree about the same event more than one venue disagrees with itself. Matching the same question across venues is the problem we just spent a week on, and the matcher and the hand review carry straight over.
The last entry in this series starts when paper trading does: what a live feed and a ledger that settles real outcomes show that replayed history could not.
Sources
- Faro's PoC-2 run of 3 November 2026 and its decisions: ADR-042 to ADR-047 in the repository.
- Manifold API, the source of the 18,992 markets and their bets.
- ForecastBench, whose question sets carry Metaculus community predictions.
- brier.fyi, a public cross-platform accuracy comparison that matched about 390 questions across prediction sites; it set our expectation that matched questions would number in the hundreds, not thousands.