Accuracy

walk-forward record of single_race_latent/0.8.1 over 230 scored races

Every figure on this page is walk-forward: each cycle was forecast using only what was estimable before it, and scored against the certified result. Nothing here is fitted to the races it reports on. Champion specification single_race_latent/0.8.1, cycles 2016,2018,2020,2022,2024.

The record

The model is 0.95 points better than the polls it is built from, and leans 1.64 points toward the Democrats where the polling average leans 2.86 — so it removes part of the polling error rather than adding to it. That is the comparison that matters: a model that cannot beat its own inputs is an expensive way to repeat them.

Brier score 0.0543, log loss 0.1710 on the same 230 races.

Against the people who do this for a living

Two numbers, because they mean different things. Wrong calls counts only races a rater actually called — a rater who rates more races tossup has fewer chances to be wrong, so the column is not comparable across raters on its own. AUC uses the full ordinal ranking including the tossups. The model calls every race; it has no tossup category.

Senate races, single_race_latent/0.7.0

Ratings are read from Wikipedia's transcription of each rater's published table, not from the rater. Where the two disagree this is the one that is wrong. Concurrent same-state contests and special elections are dropped rather than joined by state name alone. A rating is ordinal and is never converted to a probability, so no Brier or log loss is reported against a rater.

Governor races, single_race_latent/0.7.0

Ratings are read from Wikipedia's transcription of each rater's published table, not from the rater. Where the two disagree this is the one that is wrong. Concurrent same-state contests and special elections are dropped rather than joined by state name alone. A rating is ordinal and is never converted to a probability, so no Brier or log loss is reported against a rater.

Head to head with 538, 2018

Read the p column before the Brier column. The model scores better against all three 538 versions and none of the differences is statistically distinguishable from zero. On one cycle of 59 races this comparison cannot separate them, and reporting the Brier difference without that would be claiming a win the data does not support.

2018 is the only cycle 538 published a per-race review for, and it is this model's best cycle. Cook and RealClearPolitics are prohibited in source_registry; Sabato and Silver Bulletin have no registered licence-clean bulk source. 538 'deluxe' includes Cook's and Sabato's ratings as inputs, which is the nearest available comparison to them.

Where the error is

By office

By region

The regional spread is mostly polling density and the national lean, not geography: the fitted regional term is 0.00 and cannot detect a regional sd below about 2 points.

By how much the race was polled

The sparse column is where the model is weakest, and it is why the blend was refitted in 0.7.0 — safe-state polls are wrong by far more than their sample sizes imply.

By how close the race was

Error and lean are both largest where the race is least close. The obvious reading — that the forecast is compressed toward the middle — is wrong and was tested: compression would make the lean negative where the Democrat is ahead, and it is positive in seven of eight cells. What this column is showing is the partisan gradient below, seen through a column that happens to correlate with it (experiments/lopsided_or_partisan.md).

By cycle

What is still wrong with it

This section is not an appendix. A track record that reports only the wins is advertising, and every item here is measured, reproducible and currently uncorrected.

• It leans about two points toward the Democrats. +2.08 over 435 scored races. Five pre-registered attempts to correct the level have been refused by the promotion gate, because the cycle mean has a coefficient of variation near 4 and flips sign in half the cycles measured: nothing estimable before a cycle predicts that cycle's lean. The lean is quoted on every chamber page as a sensitivity rather than silently corrected.
• There is a partisan gradient it does not correct. The polls overstate the Democrat more in redder states, in proportion to how red the state is — negative in 9 of 9 cycles, coefficient of variation 0.52, independent of the level. In the reddest quarter of states the lean is +5.9 points whether the model has the Democrat or the Republican ahead — which side is favoured makes no difference, only how red the state is. Correcting it improves MAE from 5.46 to 5.24 and 90% coverage from 0.887 to 0.906. It was pre-registered under log loss, which it does not improve, and the gate refused it (q = 0.685). Switching metric after seeing the result would make the pre-registration worthless, so it stays uncorrected and marked suggestive_only until 2026, its declared confirmatory cycle.
• 2014 is the worst cycle and is not fully explained. Bias +6.50, 90% coverage 0.723 against a nominal 0.90.
• The regional term is 0.00 and that is a bound, not a measurement. The estimator cannot detect a regional sd below about 2 points, so "no regional structure" means "none large enough for this design to see".
• The 50% band is the weakest. Measured 0.471 against nominal 0.500 — inside tolerance, but it is the band that has failed before and the one to watch.
• The evidence base is ten cycles. Cycle-level effects have ten observations, not 435. Anything this page reports per cycle is noisier than its decimal places suggest.

The full list runs to 132 entries in docs/FAILURE_MODES.md, each one a defect found in this system, what it cost, and what stopped it recurring.