FutureBallotU.S. election forecasts, with every number traced to its source

Adversarial review

Findings from independent attempts to break the model's conclusions, and the answer to each one.

Adversarial review package

Prepared: 2026-09-26 · Reviewed: 2026-09-27, see experiments/adversarial_review_2026-09-27.md For: a second reader, ideally a human, who has not built this What is asked: attack the claims below in the order given. Each names the evidence, and the attack that would refute it, so a review can be efficient without reproducing the pipeline.

One review has now happened (AR-2026-09-27, two models, recorded in experiments/). Both reviewers voted to withdraw the published champion; the ground they gave was a false premise about this system's scoring target, which the review itself is what caused me to go and check — finding a real defect (FM-87) and, on the corrected target, a compression slope of 0.999 against the champion's 1.115. Read that record before this document: it says which of the questions below are still open, which the review closed, and where a reviewer was simply wrong. Four objections are on a standing list and are not closed.

Phase 0 asked for an independent adversarial review by a person and that has still never happened. The case for it is this project's own record: on 2026-09-26 I found a defect that had halved every published interval by attacking my own work (FM-79), and then, within the hour, introduced a different defect while fixing the consequences of the first (FM-81). Self-review found both. It found the second only because a number failed to move when it should have, which is the kind of luck a reviewer should not have to rely on.

How to read the numbers. Everything below is reproducible from the repository: experiments/*.md holds each measurement with its method, docs/FAILURE_MODES.md holds 85 defects with what each cost, and docs/DECISIONS.md holds 18 decisions with a falsifier attached to each. The database is not needed to check the arguments.

1. What the system claims

Two products per race, never conflated: a nowcast ("if ballots were cast today, by the electorate projected for Election Day") and an election-day forecast (the nowcast plus integrated future drift). Both are published for every race, and the chambers are simulated with a correlated error structure so a national polling miss moves every race together.

The claim under review is not "the forecasts are good". It is: every number is accompanied by an honest account of how wrong it could be, and every published probability is one the project's own evidence supports.

2. The champion's scored record

single_race_latent/0.6.0, 414 races over 2010–2024 — eight cycles, 734 scored race-products, election-day product, certified results only. The evidence base tripled on 2026-09-28 by ingesting 538's pre-2004 poll compilation, which made sigma_nat estimable for cycles that previously refused (experiments/evidence_extension_538.md), and the calibration claim weakened when it did:

0.4.0 0.5.0 0.6.0 nominal
MAE 5.24 5.34 4.68
bias, toward Democrats +3.53 +3.60 +2.40 0
CRPS — 3.827 3.333
log loss — 0.169 0.160
50% interval coverage 0.439 0.427 0.475 0.50
80% 0.768 0.756 0.810 0.80
90% 0.886 0.875 0.919 0.90

On eight cycles the same champion reads MAE 5.16, bias +1.99, coverage 0.446 / 0.771 / 0.884 — and the 50% band FAILS the ADR-0018 floor by 0.054 against a 0.05 tolerance. It passed at 0.475 on five. About half the movement is the walk-forward laws being refitted on longer history (drift a 1.698 → 1.477) and about half is the new cycles being harder. 2014 is the worst cycle in the corpus — bias +6.16, 90% coverage 0.736 — and it is the same cycle the chamber PPC fails on at tail p = 0.037 and the same one FM-82 named. Three independent measurements point at 2014 and none explains it.

The champion is not demoted and the intervals are not widened: the floor is a promotion criterion, and widening after seeing the number is what pre-registration exists to prevent. A reviewer should treat the 0.475 figure below as superseded.

(0.4.0 and 0.5.0 are quoted over the races each actually published, which differ slightly; the like-for- like comparison of 0.5.0 against 0.6.0 is the 295 races above, scored identically.)

Before 2026-09-26 the coverage figures were 0.259 / 0.518 / 0.680, and the cause was a single term missing from the variance budget (FM-79). 0.6.0 is the scale correction below, promoted the same day, and it is the first specification here to bring all three bands inside the ADR-0018 floor. Promotions to date: two (c7_sparse_blend, c15_scale_calibration). Refusals: nine not_supported, seven inconclusive.

The most useful thing a reviewer can do with this table is attack 0.6.0 rather than 0.5.0. Every number in the 0.6.0 column is better than the one beside it, which is the condition under which a change is least likely to be examined.

3. The claims, in the order they should be attacked

3.1 That the variance budget is now complete — highest cost if wrong

The race budget is state + systematic + idiosyncratic + drift, where systematic is σ_nat (2.49) and idiosyncratic is σ_race (5.04), the two components of a variance-components fit on the error of a polling average. It carried σ_nat and three unfitted placeholders worth 1.89 pts² where σ_race's 25.3 belonged.

Evidence: restoring it takes coverage to 0.886 against a nominal 0.90 with nothing tuned, and the realised error with each cycle's mean removed — which is what the residual is — has an rms of 5.42 against a fitted 5.04.

Asked and answered, 2026-09-27, and the objection lands. This section predicted that "a cleaner refutation would show that σ_race, estimated from polling-average errors, is not the right residual for a model whose estimator is better", and that is what happened. On the 295 scored races the published budget's standardised errors have sd 0.863 — 14% too wide — and the residual conditional on the model's own var_state is 4.12 points against σ_race's 5.23, an over-count of 38% in variance. Substituting it gives a standardised sd of 0.999. var_state is not redundant: it correlates with the model's error at +0.272 and rms error rises monotonically across its quartiles, 4.23 to 7.68. This is FM-89.

And correcting it does not help, which is the part worth attacking. Substituting the conditional residual alone fails the ADR-0018 floor on two bands (0.393 / 0.685 / 0.861) because the errors are heavier-tailed than the Student-t ν=5 the simulator draws: matched on variance, the distribution is too narrow through the middle. So variance and tail weight were registered together as c16_conditional_residual_shape and evaluated walk-forward. The refitted tail comes out near-normal — ν of 41 to 60 — and the gate refused it on a dead heat: mean CRPS difference +0.0000 over 2018–2024, q = 0.542, coverage better at 50% (0.488 against 0.476) and 90% (0.897 against 0.921), worse at 80%, log loss better by 0.0063. experiments/preregistration_residual.md, holdout experiment 50c48659-d9f5-46cd-82cc-c69417d240f9.

So attack it here: two specifications with mean predictive sds of 6.3 and 5.4 and tail weights of 5 and 30–60 are empirically indistinguishable on 252 races. The published intervals rest on two errors cancelling; nothing guarantees they keep cancelling on a cycle whose tails differ; and the correctly decomposed alternative forecasts no better, so this project cannot currently say which of the two is right. That is the real fragility, it is stated rather than hidden, and four cycles of evidence cannot resolve it.

3.2 That the lean is the polls' and not the model's

Evidence: on the same 246 races, a recency- and size-weighted polling average has a bias of +3.43 where the model has +3.53 — the model adds 0.08 — and beats the model on nothing: MAE 5.34 against 5.24.

Attack it by asking: the baseline shares the model's inputs, its candidate pairing, and its two-way renormalisation. A defect in any of those would appear in both and look like a property of polling. The specific thing to check is the renormalisation: both quantities are margins as a share of the two scored candidates' votes, and a systematic error in who counts as the two candidates would bias both identically.

3.3 That the forecast is compressed toward the middle — and that the correction for it belongs in the published model

Evidence: realised = -2.02 + 1.098 × average on 411 races over ten cycles, with the slope above one in 9 of 10 and the outcomes more dispersed than the averages in 9 of 10. Survives restricting to ≥30 polls (1.093), to ≥99% two-way share (1.064), and holds in all three offices. Regression dilution predicts the opposite sign on the forecast and zero on the outcome; the measured slope on the outcome is negative in all four model-scored cycles.

This is no longer a hypothesis in this system: it is c15_scale_calibration, promoted 2026-09-26 and shipped as 0.6.0. Every central estimate is now a + b·mean with a and b fitted walk-forward on the polling average's compression over strictly prior cycles, and the spread untouched. Five folds, CRPS −0.5301 against a pre-registered threshold of 0.073, q = 0.000, 4 of 5 won, log loss improved.

Asked and answered, 2026-09-27, and this paragraph used to be wrong. It said "the realised margin is a two-party share", and invited a reviewer to settle the question with a total-vote target. Two reviewers took the invitation and both voted to withdraw the champion on it. The premise was false: the law is fitted on total-vote margins already — election_result.pct holds shares of the whole vote and Observation.margin renormalises nothing. Split by third-party strength the slope is 1.094 where there is no third party at all and 1.059 where they are strongest, which is the opposite of what the artefact requires, and 2016's third-party share (5.14%) is lower than 2012's or 2010's. experiments/compression_mechanism.md has it. What the exercise did find is that the scoring target is renormalised while the forecast is not, worth 0.031 of slope and −0.09 of lean (FM-87), and that on the forecast's own scale the corrected slope is 0.999 against 0.5.0's 1.115.

So attack it somewhere else. The mechanism registered for c15 was behavioural — "polls understate the leader in uncompetitive races" — and it is probably the wrong mechanism even though the correction works. Undecided allocation is arithmetically sufficient for the whole magnitude: a 9.23% mean undecided share predicts a slope of 1.102 and the measured slope on those races is 1.104. But the cross-sectional gradient runs backwards — more undecideds, less compression — so either the level agreement is a coincidence or regression dilution is flattening the slope exactly where undecideds are highest. That is the open question, and separating dilution from the mechanism is the measurement that decides it.

The strongest challenge to c15 is now a measurement, not an argument. With the history extended to 1998 (experiments/evidence_extension_538.md), the law has been fitted on twelve folds instead of five, and the slope reverses before 2010:

fold pre2004 pre2006 pre2008 pre2010 pre2012 ... pre2026
slope 0.903 0.909 0.974 1.105 1.108 ... 1.098

Every fold from 1998 to 2006 is below one — polls overstated the leader — and every fold from 2008 on is above one, tightly. So the compression c15 corrects is a property of post-2008 polling, not of polling. The promotion survives on its own terms (all five evaluated folds were post-2008, and the law is fitted walk-forward, so a 2004 forecast correctly receives the 2004-era slope), but the stability the registration implied is refuted. A reviewer should treat the mechanism as contingent on an era.

And attack the fold set, which is the part I am least sure of, and which AR-2026-09-27 hit hardest. Four of the five evaluation folds — 2018, 2020, 2022, 2024 — are the cycles on which the compression was discovered. The law's parameters are out-of-sample on every fold, because they come from the polling average over strictly prior cycles; the hypothesis is out-of-sample only on 2016. My defence is that 2016, the one clean fold, is the largest single winner of the five (−0.8065, against a threshold of 0.073), so the result is not carried by the contaminated folds. A reviewer should decide whether that defence holds. The full argument, the objection stated in its strongest form, and a pre-registered 2026 falsifier are in experiments/preregistration_scale.md; how I came to write the opposite conclusion three times first is FM-85. A reviewer called the amendment "post-hoc adjustment dressed as principle"; that is on the standing-objections list and is not closed.

3.4 That the promotion gate controls what it claims to

The gate does a cycle-clustered bootstrap with Benjamini–Hochberg FDR control, calibrated against 1,200 null challengers, with a minimum effect anchored per metric and, since ADR-0018, a calibration floor.

Attack it by asking: the gate has promoted once in its lifetime and refused sixteen times, and the one change that most improved the model — FM-79 — was classified as a defect and bypassed it entirely, by a rule (ADR-0017) written the same day, by the same author, in the same session that made the change. That is the weakest link in the governance and it deserves the most hostile reading. The specific question: does ADR-0017's test ("stateable without reference to any score") actually exclude anything a motivated author wants to ship?

3.5 That the walk-forward discipline has no leaks

Every fitted law records the cycles it used, publish_forecasts.py refuses terms fitted on the cycle being forecast, the repository layer is as_of-filtered throughout, and a leakage test parses the SQL rather than trusting it.

Attack it by asking: poll timing is rigorously controlled and poll content only partly. As of 2026-09-27 and rising: of 13,202 polls, 4,270 verified against a primary source, 3,073 unverifiable, 5,055 incomplete, 804 disputed. So a majority has never been checked against the pollster's own release, and a value mistranscribed from a Wikipedia table is invisible to every leakage guard here — those guards control when a poll became visible and say nothing about whether its number is right.

The figures move because verification now runs on a schedule rather than by hand, so quote them with a date. scripts/verify_polls.py reports the current split.

Two corrections to earlier versions of this line, both found by counting rather than by reading:

4. What has already been attacked and survived

Listing these so a reviewer does not spend time re-deriving them. Each has its working in experiments/:

5. Known weaknesses, unfixed and stated

  1. A +2.40-point lean remains after the scale correction, attributed to the polls. It is concentrated in the cycle means — +5.03 in 2016 and +4.99 in 2020 against −1.27 in 2018 and −0.40 in 2022 — which is mu_c, the quantity no walk-forward correction can know in advance and which sigma_nat exists to carry. Whether any of it is forecastable is open.
  2. All three interval bands now pass the calibration floor (0.475 / 0.810 / 0.919), by 0.025 on the tightest. That is a pass with little margin, on five cycles of effective sample size.
  3. sigma_nat is not estimable before 2016 in this corpus, so no cycle before then can be forecast or scored (FM-82). Five cycles is the whole of the evidence base for every band figure quoted here.
  4. Most polls have never been checked against a primary source; verification now runs on the scheduler, so the figure moves. 4,270 of 13,202 verified as of 2026-09-27.
  5. Eight generic-ballot pollsters unread; two of them prohibited by their own terms.
  6. The scheduler timer is not installed, so "continuous operation" is manual.

6. The three questions worth most

  1. Is var_state + var_idiosyncratic a double count? (§3.1) If yes, the intervals are now too wide and the fix that looked like the session's best work was half wrong.
  2. Does ADR-0017's defect/hypothesis distinction have teeth? (§3.4) If it does not, the gate is decorative and one author's judgement is the only real control.
  3. Is c15's fold set honest, and is the undecided allocation its real mechanism? (§3.3) The two-party artefact is settled and was not it. What remains is the fold-set sequence — four of five folds are the discovery cycles — and the fact that the correction works for a reason other than the one it was registered with. If the fold set does not hold up, the published champion has to be withdrawn, not a pending hypothesis.

6a. A fourth question, added 2026-09-28

  1. Should a fix that becomes the champion require an adversarial review? ADR-0017 enumerates what a fix owes — a semver bump, a walk-forward refit, republication, a FAILURE_MODES entry with the numbers before and after, and the measurement reported whether or not it flatters the change — and an adversarial review is not on the list. scripts/promote_fix.py implements the ADR as written and records adversarial_review_id null with criteria.kind = "fix".

    FM-104's fix moved every thinly polled forecast in the published record, which is a larger change to the product than most challengers make, and nothing adversarial was pointed at it. The argument on the other side is that a fix is checkable in a way a hypothesis is not: its defect is stated without reference to any score, so a reader can verify it by reading the estimator rather than by trusting a comparison. The argument against is that FM-104's own fix has now been measured three times with three different numbers — 3.0x, then 2.2x, after two faults were found inside the measuring script (FM-102, FM-103) — which is exactly the pattern a review exists to catch.

    tests/integration/test_champion_matches_published.py used to assert unconditionally that the champion carried a review. It now accepts a fix that names a resolvable defect and an artifact that exists. That was a weakening and it is flagged as one, though the assertion it replaced was giving false assurance: 0.4.0 and 0.5.0 were fixes that changed every published forecast and never became champion at all, so the old test passed twice while the product changed underneath it (FM-106). I wrote the ADR's implementation, the test, and this paragraph, and I am the wrong person to decide it.