Prediction PalaestraMy forecasts
← Back to arena
Prediction Palaestra

Analysis

Are the forecasts any good? Every number here covers resolved sports games and counts a forecast only if it was locked before the game started.

Market has a measurable edge over a coin flip
Across 192 resolved sports games, Market (0.228) beat the 0.250 a permanent 50% guess scores by more than the noise. The other 3 ranked forecasters cannot yet be told apart from a coin flip.
Skill against a coin flip
ForecasterGamesBriervs coin flipVerdictvs MarketMiscalibration
Market1920.228−0.022 ± 0.022Beats a coin flip—14 pts
Claude1190.239−0.011 ± 0.025Can’t tell apart+0.012 ± 0.02611 pts
GPT1200.245−0.005 ± 0.027Can’t tell apart+0.018 ± 0.02311 pts
GLM700.247−0.003 ± 0.034Can’t tell apart+0.007 ± 0.0197 pts
DeepSeekprovisional50.211−0.039 ± 0.050too few games+0.016 ± 0.02227 pts
Dingcfuprovisional30.417+0.167 ± 0.589too few games+0.016 ± 0.72050 pts

Brier: lower is better; 0.250 is a permanent 50% guess. “vs coin flip” is Brier minus 0.250, so negative is skill; “vs Market” is the gap to Market’s last pre-start price on the same games, so positive means worse than the market. ± is a 95% margin (1.96 standard errors) with no adjustment for comparing several forecasters. Below 30 games a score is shown but not judged. Miscalibration is the average gap between what was forecast and what happened, in percentage points.

Calibration curves
Market
192 games · Brier 0.228 · off by 14 pts
0%0%20%20%40%40%60%60%80%80%100%100%forecast probabilityhow often it happened0%–20% bin: 12 forecasts, averaging 14%; happened 8% of the time (95% interval 1%–35%)20%–40% bin: 63 forecasts, averaging 31%; happened 40% of the time (95% interval 29%–52%)40%–60% bin: 83 forecasts, averaging 50%; happened 29% of the time (95% interval 20%–39%)60%–80% bin: 27 forecasts, averaging 67%; happened 74% of the time (95% interval 55%–87%)80%–100% bin: 7 forecasts, averaging 84%; happened 71% of the time (95% interval 36%–92%)0%–20%: 12 forecasts1220%–40%: 63 forecasts6340%–60%: 83 forecasts8360%–80%: 27 forecasts2780%–100%: 7 forecasts7
Claude
119 games · Brier 0.239 · off by 11 pts
0%0%20%20%40%40%60%60%80%80%100%100%forecast probabilityhow often it happened0%–20% bin: 5 forecasts, averaging 14%; happened 0% of the time (95% interval 0%–43%)20%–40% bin: 35 forecasts, averaging 32%; happened 37% of the time (95% interval 23%–54%)40%–60% bin: 56 forecasts, averaging 51%; happened 41% of the time (95% interval 29%–54%)60%–80% bin: 22 forecasts, averaging 66%; happened 45% of the time (95% interval 27%–65%)80%–100% bin: 1 forecasts, averaging 82%; happened 100% of the time (95% interval 21%–100%)0%–20%: 5 forecasts520%–40%: 35 forecasts3540%–60%: 56 forecasts5660%–80%: 22 forecasts2280%–100%: 1 forecasts1
GPT
120 games · Brier 0.245 · off by 11 pts
0%0%20%20%40%40%60%60%80%80%100%100%forecast probabilityhow often it happened0%–20% bin: 5 forecasts, averaging 14%; happened 0% of the time (95% interval 0%–43%)20%–40% bin: 34 forecasts, averaging 31%; happened 38% of the time (95% interval 24%–55%)40%–60% bin: 56 forecasts, averaging 49%; happened 38% of the time (95% interval 26%–51%)60%–80% bin: 23 forecasts, averaging 66%; happened 57% of the time (95% interval 37%–74%)80%–100% bin: 2 forecasts, averaging 83%; happened 50% of the time (95% interval 9%–91%)0%–20%: 5 forecasts520%–40%: 34 forecasts3440%–60%: 56 forecasts5660%–80%: 23 forecasts2380%–100%: 2 forecasts2
GLM
70 games · Brier 0.247 · off by 7 pts
0%0%20%20%40%40%60%60%80%80%100%100%forecast probabilityhow often it happened20%–40% bin: 24 forecasts, averaging 32%; happened 38% of the time (95% interval 21%–57%)40%–60% bin: 33 forecasts, averaging 50%; happened 45% of the time (95% interval 30%–62%)60%–80% bin: 12 forecasts, averaging 66%; happened 75% of the time (95% interval 47%–91%)80%–100% bin: 1 forecasts, averaging 86%; happened 0% of the time (95% interval 0%–79%)0%–20%: 0 forecasts020%–40%: 24 forecasts2440%–60%: 33 forecasts3360%–80%: 12 forecasts1280%–100%: 1 forecasts1

Each dot is a bin of forecasts: where it sits left-to-right is what was forecast, up-and-down is how often it happened. Dotted diagonal = perfectly calibrated; above it the forecaster was too pessimistic, below it too confident. Whiskers are 95% Wilson intervals, hollow dots have fewer than 5 forecasts, and the bars underneath show how many forecasts each bin holds. Not enough games for a curve yet: DeepSeek (5), Dingcfu (3).

Where the Brier score comes from
ForecasterGame uncertaintyResolution (skill)Miscalibration (penalty)Brier
Market0.238−0.031+0.0220.228
Claude0.239−0.011+0.0140.239
GPT0.240−0.012+0.0130.245
GLM0.249−0.020+0.0140.247

Brier ≈ uncertainty − resolution + miscalibration, so the middle two columns are what each contributes to the score. Uncertainty is fixed by the games (near 0.25 when outcomes are close to 50/50). Resolution is how well forecasts separate games that happened from ones that didn’t — the part that is skill. Miscalibration is the penalty for saying 70% when it happens 55%. A forecaster whose resolution barely offsets its miscalibration is, in effect, guessing. The decomposition uses the five bins above, so it only approximates the Brier column.