DATA SOURCES

All reachability and licence statements below were probed on 2026-09-22 from this machine and are recorded with what was actually observed. Nothing here is assumed. A source is not fetched by the pipeline until its row in source_registry has enabled = true, which requires a human licence decision (DATABASE_SCHEMA §2).

---

1. The finding that changes the plan

The FiveThirtyEight raw poll feed is gone.

GET https://projects.fivethirtyeight.com/polls/data/president_polls.csv
  -> HTTP 200, Content-Type: text/html
  -> body is an ABC News HTML page, not CSV

The endpoint that nearly every open-source election model in the last decade was built on now returns a marketing page. The GitHub archive fivethirtyeight/data is still live and still CC BY 4.0 + MIT (verified by reading its README), but its polls/ directory contains only:

Last commit touching polls/: 2024-09-13. The pollster-ratings/ directory is present and is the most useful surviving artefact there.

Consequence

The polling database cannot be built by downloading one CSV. It must be assembled, and the assembly is the largest single work item in the project. Ranked by defensibility:

1. Primary pollster releases — the authoritative source, and the only one that supplies the methodology fields in build spec §5 (mode, weighting, LV screen, question wording). Highest fidelity, highest labour. This is where the LLM extraction agents earn their keep.
2. Surviving third-party archives with a clear licence. Each must be verified for provenance; a mirror is not an authority, and a fork of 538's CSVs is evidence of what 538 published, not evidence of what the pollster published.
3. Academic archives (Roper iPoll, ICPSR) — high quality, licence-restricted, need the user's authorisation.
4. Wikipedia election poll tables — broad coverage, CC BY-SA, crowd-maintained. Usable only as a discovery layer that points at primary documents, never as a terminal source. Every Wikipedia-sourced row must resolve to a primary artefact before it is verified. (Wikipedia's other uses here, as cited facts, are in §2b.)

This ordering is itself a design decision (ADR-0009) and it is why the architecture treats extraction-from-primary-documents as a first-class pipeline rather than an afterthought.

---

2. Verified-reachable sources

2.1 Data.gov catalog API — discovery, verified 2026-09-22

No authentication. Documentation at resources.data.gov/catalog-api. Endpoints confirmed by probe:

Other parameters documented: org_id, org_type, keyword, after, spatial_filter, sort. Pagination is cursor-based on the after token, not offset-based — the ingest client must follow the cursor rather than compute page numbers. The legacy CKAN path /api/3/action/package_search returns 404, so this is a distinct API and CKAN client libraries will not work against it.

Why this matters more than it first appears. A query for election results returned, in the first eight hits, precinct-level results from King County (WA) and statewide results from Connecticut, alongside the NIST Election Results Reporting XML Schema. Precinct-level results are the scarce input for redistricting disaggregation (§8 of STATISTICAL_MODEL, FM-16) — the thing that makes district baselines comparable across boundary changes. A no-auth, queryable index of state and county open-data portals is therefore a genuine discovery channel for the project's hardest data problem, not merely a directory of federal CSVs.

Role. This is the backbone of the Source Discovery Agent (build spec §25). It is a discovery layer: it returns metadata and pointers, and the pointed-at data is then fetched, archived, and verified through the normal L0 path.

Three limits, stated plainly.

1. It does not settle licensing. Most returned DCAT records carried an empty license field. Discovery via the catalog does not flip source_registry.enabled to true; a licence decision still has to be recorded per dataset. The catalog tells us a dataset exists, not that we may use it.
2. Coverage is uneven and metadata quality varies. Many records point at municipal Socrata portals with inconsistent schemas and no methodology documentation. Discovery volume is not data quality.
3. It does not replace the API keys. catalog.data.gov (the dataset catalogue) and api.data.gov (the shared federal API gateway that issues keys for OpenFEC) are different services. The FEC and Congress.gov keys are still needed for those APIs at volume.

2.2 API credentials — resolved 2026-09-22

One api.data.gov key serves both Congress.gov and OpenFEC. Both sit behind the api.data.gov umbrella gateway (visible in the x-api-umbrella-request-id response header), and the same key authenticates against both, via the X-Api-Key header. The user's existing key was located and installed as ELAB_CONGRESS_GOV_API_KEY and ELAB_DATA_GOV_API_KEY in the git-ignored .env.

Verified live:

The two services rate-limit the same key by more than two orders of magnitude, and the header does not state its window. A limiter that assumed "requests per hour" from one service would be wrong for the other by a wide margin in whichever direction the window differs. This is direct vindication of the fetcher design decision in PHASE1_PLAN M1.4: read x-ratelimit-limit / x-ratelimit-remaining from each response and adapt per service, rather than hardcode a rate per source. The source_registry.rate_limit_rps column becomes a conservative ceiling, not the operative control.

Still needs a human before it can proceed

Build spec §58 lists "credentials required" and "licensed data require authorization" as legitimate reasons to interrupt. These are those.

---

2b. Sources and the basis for their use: cited facts (owner's decision, 2026-10-20)

The rights basis. These sources supply facts: a rater's category call for a race, a chamber's party counts, a governor's party, a candidate's name, who holds a seat. Facts are not protected by copyright in the United States (Feist v. Rural Telephone, 1991), and reporting another firm's published call with attribution is the established practice of news sites, Wikipedia and other forecasters (in Barclays v. Theflyonthewall.com, 2nd Cir. 2011, a site reporting banks' stock recommendations defeated the banks' claim against it). So each fact is used with its source named where it is shown, nobody's text, analysis or table layout is republished, and none of these sources' own sites is read where its terms forbid automated access. Wikipedia's compilation is CC BY-SA 4.0 and is credited where its facts appear. Where a rater's call might be argued to be protected expression rather than a fact, the use rests on fair use: reporting, comparison and commentary (each call sits beside FutureBallot's own forecast and the other raters'), a brief conventional label rather than any analysis, and no substitute for the raters' own product, whose reasoning is never reproduced. The hardest question is scale: /raters and the race pages show every rated 2026 race, so they display substantially all of each rater's calls, continuously. That is still reporting and comparison, but it is the part a lawyer would look at first, and the owner can narrow it (show the calls only on each race's page, or only in the comparison index) without touching the model. This is the owner's decision of 2026-10-20, recorded in the licence registry (config/sources.yaml, migration 0073); it is not legal advice. No lawyer's review is planned (owner, 2026-10-20): the display stands as decided, and narrowing it remains available without touching the model.

Every fetch passes the licence gate (elab.provenance.licence) before robots.txt: a host whose source is prohibited or unverified is refused by the shared fetcher and by every monitor, list and results reader (FM-215).

Reading the news (owner's decision, 2026-10-20). Three different things, three rules:

1. Automated collection (scheduled fetchers, feeds, archives, crawlers) obeys each site's terms and robots.txt, through the licence gate above. Politico, Fox News, the New York Times' feeds and the rest stay refused to our programs.
2. A one-off read to check a fact, by the owner or by Claude for the owner, is fine for any outlet, prohibited hosts included: a single article is read, the fact is stated with credit to the outlet that did the original reporting and a link, quotations stay brief, and the article's text is neither stored nor republished. A partisan page (a party, a campaign committee, Media Matters) is a pointer to the original report, never the source credited: on NV-2, Rep. Amodei's "a guy who I think is a fraud" is the Washington Post's reporting (Dan Merica, 2026-09-24), which the DCCC and Raw Story relayed; it is credited to the Post.
3. GDELT's index entries (outlet, date, address, title, people, themes, tone) are used for every outlet under GDELT's terms (any use, with citation), prohibited hosts included, without reading the article site's robots.txt: the site is never visited. This replaces the earlier rule that applied FM-175's archived-copy test to an index's headline (which kept Politico's 2026-10-17 NV-2 story out of the race monitor); elab.monitor.gkg, elab.monitor.gdelt and the press-read study follow it.

2c. Published poll results from any public report (owner's decision, 2026-10-26)

The rule. A poll's published result (the firm, field dates, population, sample size and the parties' shares) is a fact (§2b). It may be recorded from any public page that states it: the pollster's or sponsor's own public release, or a press report, newsletter or public post that quotes it, with the page cited where the poll is used. Three things are never used:

1. Results known only from someone else's collection: an aggregator's database or poll list (Decision Desk HQ, RealClearPolitics, Silver Bulletin, FiftyPlusOne, 270toWin and the like). Such a list may tell us a poll exists; the figures are always taken from the public report.
2. Anything behind a login, a form or a paywall, or anything that must be paid for. Nothing is bought (owner: "we sure as fuck arent paying anyone").
3. Unreleased or private polls (internal polls nobody published).

Reading discipline. Automated reading still obeys each site's terms of access and robots.txt (§2b, rule 1): a press report is fetched by the project's own collector only where its site allows it; where it does not, another outlet's report of the same poll is used, and a poll with no readable public report stays out. The loader (scripts/ingest_generic_ballot_press.py) archives each page, requires the quoted sentence and figures to be on it, and records a second independent report where one exists. Morning Consult's weekly figures come from its own public tracker page (marked free to read), which refuses automated readers: they were read once through the Wayback Machine's public copies and are stored as cited facts (§2b rule 2), the page never fetched by the collector (owner, 2026-10-26). Before this rule, national generic-ballot polls were read only from firms whose own sites allowed automated reading, which left about 80% of 2026's polls out (TODO H89).

2d. Bluesky's public post stream: mention counts for the H36 test (2026-10-26)

What and why. TODO H36 (owner, 2026-10-22): a prospective test of social signals, collected from now to Election Day and scored after certification (experiments/social_signals_2026_preregistration.md, written before any post was read). The source is Bluesky's public Jetstream, the v2 US East instance, endpoint copied from Bluesky's own page https://bsky.network/docs/jetstream (docs.bsky.app redirects there): wss://jetstream.us-east.bsky.network/xrpc/network.bsky.jetstream.subscribeEvents?collections=app.bsky.feed.post&kinds=commit (the page lists the older v1 host jetstream2.us-east.bsky.network as legacy and recommends v2 for new projects). Registry bluesky_jetstream (restricted: counts only).

Terms, read 2026-10-26. The Jetstream page: "No authentication is required for the live tail; these instances serve the full network." The Bluesky Developer Guidelines (https://bsky.network/docs/developer-guidelines): no spam and no automated or bulk interactions; services must have a method for deleting content a user has requested to be deleted; developers keep a public, monitored contact address (contact@futureballot.com); reasonable security for any end-user information; Bluesky's Terms of Service (https://bsky.social/about/support/tos, last updated 14 August 2025, with no clause on reading the public stream), Privacy Policy and Community Guidelines apply. How the collector meets them: it never posts, follows, likes or sends anything (it opens the socket and reads); it stores no post text, handle, DID or URI, only per candidacy and UTC day the counts of matched posts, of matched posts carrying the pre-registered trouble and split keywords, of rater posts excluded, and an estimate of distinct authors; a matched post deleted while its day is open (that day and the next) is removed from the counts, using a salted hash kept only in the private state file (mode 0600) and discarded when the day closes; the day files are git-ignored research data on Skynet-Three.

Reading discipline. scripts/social/bluesky_collect.py (logic elab.social.bluesky; a standard-library WebSocket reader, elab.social.ws) passes the licence gate before connecting, resumes from its cursor with backoff, and stops by itself at 2026-11-04 12:00 UTC. It writes files only, no raw_artifact rows, so it is not in the health monitor's cadence table (QUIET_DAYS_BY_SOURCE) and its silence cannot withhold a forecast (FM-196); its job is watched by jobs-watch instead (docs/OPERATIONS.md). Nothing from it reaches the model or any page; the pre-registered result will be published with per-race window totals only. Reddit's and X's terms (the rest of H36) were not assessed in this change.

3. Sources deliberately not used

---

4. Prediction markets

Status: not used (2026-10-20). Kalshi's Developer Agreement prohibits any use of its API data beyond a member's own trading (§3), so Kalshi is prohibited. Polymarket's two sites are prohibited too (terms read 2026-10-20 from the text the owner supplied, since both pages render it client-side): polymarket.com forbids "data mining tools, robots, crawlers, or similar data gathering and extraction tools", "any manual process to monitor or copy any of the material" without written consent, and distributing any portion of the Interfaces; polymarket.us §5 limits market data to "personal, non-commercial use in connection with your trading activities" and prohibits "Redistribution, resale, scraping, bulk downloads". All three hosts are refused by the licence gate. Their prices return only with the company's written permission or licence.

Regardless of availability, markets are not ingested as a model input until they pass the test in build spec §41: do they add out-of-sample predictive value beyond polls and fundamentals? Markets partly price public forecasts, so naive inclusion creates circularity (FM-23). They enter the experiment registry as a hypothesis, and the null result is an acceptable outcome.

---

5. Fundamentals series

The FEC rule is worth stating twice: a campaign finance report becomes knowable on its filing date, not on the last day of the period it covers. The repository enforces this.

---

6. Coverage gaps, stated honestly

• Pre-1998 polling is sparse, inconsistently documented, and methodologically alien (landline-era RDD). It will be ingested where available and down-weighted or excluded by the concept-drift analysis, not silently pooled.
• House district polling is extremely thin. Most of the 435 will have zero polls in any given cycle. This is a property of reality, and the model must express the resulting uncertainty rather than manufacture a number.
• Precinct-level results for boundary disaggregation do not exist uniformly across states or years. boundary_lineage.method = 'unmapped' exists for exactly this case.
• Methodology metadata (weighting scheme, LV screen) is unavailable for a large fraction of historical polls. The schema distinguishes "not reported" from "not yet extracted" from "not applicable", and the model treats missing methodology as a source of extra variance rather than as a default value.

No gap is filled by fabrication. A missing value stays missing, and its absence propagates into the uncertainty.

---

7. Fetching discipline

1. Read robots.txt and terms before first fetch; record robots_checked_at.
2. Prefer official APIs and bulk files over HTML scraping.
3. Honour rate limits from source_registry.rate_limit_rps; exponential backoff on 429/5xx.
4. Identify the client honestly in the User-Agent, with contact information.
5. Archive raw bytes before parsing, always (DATABASE_SCHEMA §2). Parsing is re-runnable; fetching is not.
6. Cache by content hash — re-fetching identical bytes is a no-op.
7. All fetched content is untrusted input. See SECURITY.md.

Poll sponsors: what partisan_sponsor means (decided 2026-10-05)

The rule. poll.partisan_sponsor is true when the poll was paid for by

• a candidate's campaign (any candidate, including one running for another office),
• a party committee (DSCC, NRSC, DGA, RGA, DAGA, a state party, the Working Families Party), or
• a partisan-aligned group: a PAC or super PAC, or an advocacy organisation working to elect one party's candidates or a named candidate in the race polled.

It is false for

• a poll the pollster released and paid for itself, including a Republican- or Democratic-leaning firm's. A firm's lean is not a sponsor; it is what the pollster house effects measure;
• media and university polls (an outlet owned by an advocacy group, such as the John Locke Foundation's Carolina Journal, is still media);
• nonpartisan sponsors: AARP, chambers of commerce, trade and professional associations, think tanks and 501(c)(3) foundations, and issue or advocacy groups that do not campaign for candidates.

poll.sponsor_id records who paid whenever a sponsor is named, partisan or not (AARP, Fox News, a chamber).

How sponsored polls are used (owner, 2026-10-23). A poll with partisan_sponsor true is used, with the lean corrected: for Senate and governor races the pollster's house effect absorbs it; for House districts the deployed poll layer subtracts the sponsor party's bias measured on past elections (2026 calibration: Democratic-sponsored polls about 2 points more Democratic, Republican- sponsored about 4 points more Republican, than independent polls). Leaving them out was tested on 2016-2024 and did not help: House forecasts got worse and Senate and governor forecasts no better (experiments/partisan_polls_backtest_2026_10_23_result.md). Only a campaign's or party committee's own in-house polling operation is excluded, by name, in config/excluded_pollsters.yaml (kind internal).

poll.is_campaign_internal is narrower (decided 2026-10-06). It is true only when the poll was conducted by a pollster working for the campaign, as its pollster of record, and not when a campaign bought results from an independent survey. It sets the "campaign internal" tag on /polls; the forecast reads partisan_sponsor, not this column. Alaska Survey Research's 2026 Senate polls are the case that set it. Ivan Moore wrote that the Peltola campaign buys the Senate results (so partisan_sponsor is true and the payer is recorded), but that ASR is "not embedded in campaigns" and that its surveys "have multiple clients from across the political spectrum". So they are not internals.

Applying it. Who paid is read from the poll's own release, or from the sponsor's or campaign's release, or from two independent news reports. Wikipedia's footnote ("Poll sponsored by X's campaign") is a lead, not evidence. Rules used when the payer is a group:

• A union or issue group counts as partisan only if a source shows it endorsed or campaigned for a candidate in the race polled before or during the fieldwork, or that it exists to elect one party's candidates. An endorsement made after the poll does not count.
• Chambers of commerce, trade associations and AARP are nonpartisan whoever they endorse. The coordinator named them, and an endorsement by a chamber's PAC is the PAC's act, not the poll's payer.
• A pollster that sells a question to a campaign has been paid by that campaign. Ivan Moore wrote on 3 September 2026 that "the Peltola campaign purchases the Senate results" of Alaska Survey Research's 2026 Senate polling, so those polls are flagged.
• When the payer cannot be found in any source, the flag is set false only if the release itself names no sponsor and is the pollster's own poll. Otherwise the poll keeps its flag and is listed in experiments/poll_audit_2_corrections_2026-10-05.md.

Why it changed. Until 2026-10-05 the ingest set the flag from the "(R)"/"(D)" that Wikipedia writes after a pollster's name (WikiPollRow.is_partisan_marked). FM-171 then added sponsors named in footnotes, and its rule also marked any interest group, AARP included. Wikipedia's mark often labels only the firm's lean. Independent poll audit 2 found that 61 of the 137 flagged polls in its sample were the pollster's own releases or a nonpartisan sponsor's (FM-174). The ingest now takes the flag from the sponsor footnote alone (elab.ingest.poll_sponsors.classify, Sponsor.partisan); the mark is kept as WikiPollRow.has_party_mark/partisan_mark and is not used as the flag. Every live 2026 Senate and governor poll was re-judged under the rule (scripts/fix_partisan_sponsor_flags_2026_10_05.py).

What reads it. The statewide model (single_race_latent 0.7.0 and the 0.8.1 blend) does not read the flag; sponsored polls are kept and their lean is left to the house effects (docs/ADJUSTMENTS.md, sponsor_handling). The House district model and --exclude-partisan backtests read it, and the site labels sponsored polls.

House and historical polls (2026-10-05, experiments/pending_poll_items_2026-10-05.md). The 132 flagged 2026 House polls were researched one by one: 52 keep the flag with the payer recorded, 4 lost it (the pollster's own release), 76 are undetermined and keep it. The 2,404 flagged pre-2026 polls were judged from evidence already archived (scripts/pending_poll_items_rules_2026_10_05.py): 299 lost the flag and 12 had the payer recorded; 763 keep it because Wikipedia's footnote names a campaign, party committee or PAC (a lead: the payer is not recorded); 1,330 are undetermined and keep it. The corrected versions are recorded on 2026-10-05, so a backtest standing in an earlier cycle reads the flag as it was then: the corrections reach a 2026 run only.

Evidence search for a poll correction (2026-10-05, FM-176)

Any pass that checks a stored poll against its release follows these rules: a sweep, a pending-item pass, an audit's corrections or a release check. They are added after poll audit 4. In that audit two errors the sweep had already found were dropped, because the search stopped early (FM-176).

1. A closed primary is not "no admissible source". The release may be robots-refused, on a prohibited host, or have its archived bytes removed (FM-175). Then search for permitted reports that print the disputed figure itself, not just the headline shares. Search by the figure: the n, the candidate's name, the field dates. Record every URL and query tried, with its result.
2. Re-check every refusal at the time of the pass. Judge each URL with RobotsGate and the registry's prohibited hosts (robots_remediation_2026_10_05.prohibited_hosts, which reads source_registry) as they stand now. Never reuse a "robots refused" or "not fetched" verdict from an earlier pass's notes. robots.txt files change, and the matcher has changed (FM-175).
3. Prohibited hosts come from the registry, not a fixed list. A fetch helper with a fixed list (for example scripts/poll_audit_4_fetch.py) misses hosts the registry prohibits later. Newsweek is one: its terms were reviewed on 2026-10-05.
4. Sibling questions of one survey. NYT/Siena and similar state surveys print one likely-voter n per state sample. A stored round n can be corrected on one permitted report of the n, when the other race's question from the same survey already stores that n. The sibling is named in the decision.
5. A material finding that is not applied stays open. It goes on the pass's undetermined list with the sources tried and what would settle it. It is never logged only as "checked; not changed".

FEC biennial report: the workbook, not the PDF

The FEC publishes Federal Elections as both a PDF and a workbook. This project reads the workbook. The two carry the same data, but the PDF carries it as typography: the earlier parser needed a tuned column offset, a wrapped-row joiner and a regular expression for candidate names, and each of those is a guess whose failure mode is a plausible number rather than an error. In the workbook, fusion totals, runoff rounds and ranked-choice rounds are each a labelled column, so the only question left is which column the law says decides the election.

Document ids, verified against the FEC's own per-cycle index pages:

2024 is not listed because it is not published. Both URL patterns were checked on 2026-09-23: /election-results-and-voting-information/federal-elections-2024/ returns 404 and /election-and-voting-information/federal-elections-2024/ returns 200 with no document links. That is a checked absence, not an assumed one.

Do not guess document ids. An earlier attempt brute-forced roughly 1,400 of them, which got this client rate-limited for half an hour in violation of section 7 above, and produced five wrong ids that looked right: four served MP3 recordings of open meetings, and the fifth served a valid PDF of a 2003 press release about campaign spending, filed under the id guessed for 2014. The content-type header did not catch the last one, because it really was a PDF. The fix is in two parts: Fetcher(expect_magic=...) checks the body rather than the header, and ids come from the index pages.

What the report settles

Against the Clerk of the House's official statistics, the parser reproduces the House split exactly for all five cycles -- 435 seats, correct D/R division, no third-party or unresolved seats. Rules that each fix a real contest:

• Fusion (NY, CT, SC): a candidate appears once per endorsing party. 2022 adds a Combined Parties: line with the sum; 2014 has exactly one such line in the whole sheet. The total is used when present and the components summed when it is not -- never both, which once produced a 144% vote share.
• Ranked choice (AK 2022, ME): decided by the last round in which two candidates hold votes. In 2018 Maine had no round columns, and the winner trailed first preferences by 2,171 votes, so the report's own winner flag takes precedence over the counts.
• Runoffs (LA, GA, MS): a runoff decides the whole contest. Applied per candidate, it compared one candidate's runoff count against another's first-round count.
• Unopposed seats: the word "Unopposed" stands where a count would be; reading it as a missing value dropped the district entirely.
• Full vs unexpired terms: IN-02 in 2022, OK in 2014 and 2022, MS 2018, GA 2020. The qualifier lives in the district cell and its spelling drifts between cycles.
• Delegates (AS, DC, GU, MP, PR, VI): in the report, not among the 435.
• NC-09 in 2018: the report flags no winner, because the election was never certified.

Governors: three sources, none of them federal

The FEC's remit stops at Congress, so there is no single authority for governors and every one of this project's 1,180 governor results rested on one Wikipedia pipeline. Three independent sources were found; between them they certify 561 results (47.5%), up from none.

MEDSL mirrors the same returns on GitHub, without the guestbook. Three of the five Dataverse state-returns files sit behind guestbookID 458; the GitHub organisation publishes the same data openly, and checking it turned two of those three into ordinary downloads:

So the guestbook blocks exactly one cycle, 2020, not three. Finding that took one call to the GitHub API, after I had already written the gap up as a decision for the user to make. It is the fourth instance this project has recorded of a gap that was really an unsearched source (FM-43), and the first where I had gone as far as asking someone else to act on it.

2026-10-13: the owner authorised answering the guestbook as FutureBallot (name "Phil", contact@futureballot.com, institution FutureBallot, position "Election forecasting") after reading the Dataverse community norms: the data are CC0 and the norms ask only for a data citation and no re-identification of human subjects (not applicable to precinct totals), and are stated to be non-binding. scripts/stateleg/fetch_dataverse.py --guestbook submits those answers through the access API; each dataset's manifest records that it was guestbook-gated. MEDSL is credited with the dataset citation wherever its data are used.

Where it does apply, the guestbook is not worked around (it is answered, as above, only with the owner's authorisation). It is a condition the depositor set and a representation made to a named institution. The datasets themselves carry CC0 1.0 and no terms of use at all, so the question is one of access rather than licensing.

The last of these is not usable as it stands. It lists who held office in each year rather than who won each election, so it includes appointed and successor governors — Jeff Colyer appears for Kansas in 2018 and 2019 without ever winning an election. Comparing it to election results would manufacture disagreements wherever a governor resigned.

Two guards decide what is usable

Neither source is trusted on its own, because both are compiled by summing smaller units and summing is where a plausible wrong number comes from.

• Turnout. A statewide contest shares a ballot with that state's House races, so its total should resemble theirs; anything outside 0.85–1.15× is rejected. This is not hypothetical. MEDSL's Georgia rows for 2022 total exactly twice the true vote and its Indiana rows about 40% of it. OpenElections's Delaware file for 2004 puts Sussex County at 219,819 votes in a county that cast about 60,000. None of these is visible in the vote share, and all three would otherwise have been certified.
• Agreement. Nothing is certified on one source's word. A result is certified when the new source and the existing record agree within half a point, so an error on either side costs coverage rather than correctness.

What each source needed

MEDSL codes a fusion candidate's minor line as "OTHER" under party_simplified, so Richard Blumenthal appears twice in Connecticut and summing only the first put his share 0.8 points low. OpenElections needed three fixes, each found by a state that looked empty or impossible: several states elect a joint ticket and label the office "Governor and Lieutenant Governor", so excluding anything mentioning a lieutenant governor dropped four states entirely; Massachusetts's "Governor's Council" is a different body and matched on the bare word; and states spell parties their own way, "DEMOCRATIC PARTY" in New Mexico against a bare "D" elsewhere, where an unrecognised spelling silently drops a candidate from the two-party vote.

The file for each state and year comes from data/openelections/file_index.json, built by listing each repository's tree once rather than by guessing paths. A guessed path returns 404, and a 404 read as "no data" is how a gap gets recorded that is not there — which this project has done often enough to name it, as FM-43.

Generic ballot after 2016

The 538 trendline this project uses (generic_topline_historical.csv) ends 2016-11-06. Without a staleness limit that produced a confident zero swing for every later cycle, which is FM-45.

Searched, and not found in a current source:

Found in the Internet Archive. The 538 endpoint was captured while it was live: web.archive.org/web/<ts>id_/https://projects.fivethirtyeight.com/polls/data/generic_ballot_polls.csv with snapshots in 2022, 2023, 2024 and 2025. The 2022 capture is in data/generic_ballot/generic_ballot_polls_2022.csv — 1,207 polls with pollster, field dates, sample size and D/R shares. The 2024 capture was not retrieved because archive.org went offline mid-session; the CDX query to repeat is recorded above.

Retrieved, 2026-09-25. The archive came back up. The CDX listing for the endpoint has snapshots through 2025-03-05, and the last useful one is 20241127022426 (42kB, taken after the 2024 election, so it covers that whole cycle). The 2025 captures are 1.6kB and 3.4kB — the endpoint was already returning almost nothing before 538 was shut down in March 2025.

scripts/ingest_generic_ballot.py fetches the captures through the provenance layer rather than reading a file out of data/, so every poll traces to a raw_artifact row holding the bytes that were retrieved. The polls land in fundamental_series as one row per poll — the generic ballot is a national quantity with no contest attached, so it cannot go in poll_question, which requires a race — with the field dates as the reference period and the publication date as the vintage. FundamentalsRepository.series_window reads them vintage-correct, because a poll published after the forecast date is a poll the forecast could not have seen.

CORRECTION, 2026-09-25: there were two files, and this section described one of them. 538's own polls/README.md says it plainly — "Current polls files contain data since the most recent election. Historical files contain data prior to the most recent election" — and names polls-page/data/generic_ballot_polls_historical.csv. That file, captured at 20250118200335, holds 4,832 national generic-ballot polls from 2016 to 2024, across the 2018, 2020, 2022 and 2024 cycles. Everything above was written after reading only the current file, and the conclusion that the corpus began in November 2020 was a statement about the lookup, not about the data (FM-63).

The corpus is now 3,815 polls over four cycles rather than 1,055 over one and a half, which is what makes an environment drift law estimable at all: experiments/environment_drift.json fits Var(h) = 0.112·h^0.59 from 371 non-overlapping increments of the generic-ballot path, and a law fitted on one cycle's path would have been a description of that cycle.

Two further corrections to what this section claimed.

The 2022 capture was truncated. 20221109051953, taken the day after the 2022 election, parsed to 388 polls whose newest was fielded to 3 August. So was 20221013105339. Every Wayback copy of the current file is cut off mid-body, which is why the September and October 2022 polls were missing and were put down to the source. ingest_generic_ballot.py now refuses a capture whose newest poll is more than 21 days from both the capture date and the election before it (FM-62). The local data/generic_ballot/generic_ballot_polls_2022.csv is a complete copy of that file and is not used: it has no provenance, and a CSV in data/ cannot say where it came from.

The rows are questions, not polls. One survey appears up to six times in these files — registered voters, likely voters, variant screens — so 4,832 rows are 3,815 polls. The ingest now takes one record per poll_id, preferring likely voters, then registered voters, then adults (FM-64).

Still missing, and named rather than smoothed over: generic-ballot polls fielded between 12 October and 8 November 2022. No capture of either file covers them. They are fillable from pollsters' own releases, which is what ingest_generic_ballot_direct.py does for the current cycle, and its --since flag is what would extend it backwards.

CORRECTION, 2026-09-25: the paragraph below was wrong, and wrong in the way this project has a failure mode for. I checked FiveThirtyEight's dead endpoint and a handful of guessed Wikipedia titles, found nothing, and reported a failed lookup as an absence — FM-43 and FM-46 again, a third time, and the owner caught it a second time. I never ran a Wikipedia search, and I never looked at a pollster's own site.

There are 680 individual generic-ballot polls for the 2026 cycle from 105 firms, spanning January 2025 to 22 September 2026: Emerson, Marquette, Quinnipiac, YouGov/Economist, Fox News, Echelon, AtlasIntel, McLaughlin, ActiVote and the rest. Every one is published by the firm that conducted it.

Routes, with the licence position on each.

src/elab/ingest/pollster_direct.py holds one extractor per firm, because each writes its release in its own prose. Seven are read — Quinnipiac, Rasmussen, The Economist/YouGov, Marquette, Emerson, Cygnal and Echelon Insights — each returning field dates, sample size, population and a cross-check against whatever else the release says about its own numbers. MISSING_EXTRACTORS names the six firms seen publishing a generic ballot and not yet read, with a reason each, because "no poll exists" and "nobody has written the twenty lines that read this firm's prose" are the distinction this project keeps getting wrong.

Echelon Insights is the safest read of the seven, and the reason is the document. They publish a topline: every question with its answer percentages, labelled by party in words, so there is no orientation to infer and no chart to read. The sub-items are the cross-check — "Definitely", "Probably" and "Lean" must reproduce each party's leaner-inclusive total, which is arithmetic the pollster did rather than a sentence they wrote. Their pages carry no date and their feed does, which is why a source may now declare a published_index: knowability is publication (FM-02) and this firm states it in one place only. Seven polls, April to September 2026.

Cygnal is the first source here whose numbers are not on a web page. Its poll page carries no vote shares at all; the public slide deck it links carries them, so reading it needed a second fetch and elab.parse.pdf_text. Three of the eight 2026 national releases state the ballot in prose with their own margin ("they lead 49% to 42%, a D+7 margin"), which is both the cross-check and the thing that decides which share is which; the other five describe movement without stating a level and produce nothing. The deck's chart page is not used although it is the obvious place to look — page 22 of the July deck extracts as eight percentages from two panels with nothing to say which is the topline, and a number read off that would be right some months. Provenance points at the deck, not the page: an archived page that does not state a figure is not evidence for that figure.

That also settled a reason that was being given for four other firms — and then checking the licences before writing the parsers changed two of them from technical to final (FM-77). "The release is a PDF" was the first obstacle recorded and not the operative one:

All three are registered in config/sources.yaml with the quoted language. What remains a purely technical barrier is a form (Morning Consult), and two are refusals by the server: ActiVote answers a bot challenge and Quantus Insights serves a certificate that does not verify.

robots.txt checked 2026-09-25 for Quinnipiac (no Disallow rules at all), Emerson (one rule), Marquette (36, none blanket), Cygnal (/wp-admin/, /wp-includes/, /CLIENTS/, /dev/ — neither /polls/ nor /wp-content/uploads/), Echelon and Fox News (preview and search paths only). None of them restricts its published releases.

The superseded paragraph, kept because being wrong in public is part of the record:

2026 has no generic ballot, and this is now a settled conclusion rather than a pending search. The source stopped publishing before the cycle began. What remains on Wikipedia's 2026 House page is a table of aggregators — Decision Desk HQ, FiftyPlusOne, RealClearPolitics — which is not an acceptable input twice over: RealClearPolitics is disabled in the source registry, and an aggregate is another forecaster's model, which this project treats as a benchmark and never as ground truth. Also checked and absent: a "Nationwide opinion polling for the 2026 House elections" article, a Generic ballot article, individual-poll sections or labelled transclusions on the 2026 House page (only GenericBallotAgg exists), and the fivethirtyeight/data repository's congress-generic-ballot/ and polls/ folders, which the GitHub contents API shows hold one 1995–2016 trendline and presidential averages respectively.

Ingested, 2026-09-25: 1,055 individual polls spanning 2020-11-21 to 2024-11-04. Two captures with verified provenance, in fundamental_series with the field dates as the reference period, the publication date as the vintage, and the sample size in observation_n (migration 0022 — added after checking the spread rather than assuming it: 813 at the 5th percentile and 17,010 at the 95th, an interquartile ratio of 11.7, so equal weighting would throw away most of the information in the large polls).

Two things about the archive worth knowing before repeating this.

538 rotated the file per cycle. A capture from December 2023 holds 411 rows, all cycle 2024, starting the week after the 2022 election. There is no single file with every cycle in it.

A capture's timestamp is not the content's date. The snapshot at 20221109051953 — the day after the 2022 election — contains 454 rows ending 8 August 2022. The archive crawled a cached copy. So the complete 2022-cycle file, which must have carried September and October polls, exists only in captures taken between the 2022 election and the rotation, and the CDX endpoint returns 504 on every query narrowed to that window. Retry:

https://web.archive.org/cdx/search/cdx?url=projects.fivethirtyeight.com/polls/data/
generic_ballot_polls.csv&output=json&fl=timestamp,length&filter=statuscode:200
&from=20221110&to=20230601&collapse=digest

Consequence, stated rather than smoothed over. The 2022 cycle has generic-ballot coverage only through 8 August, three months before the election. Environment.staleness_days makes that visible and estimate returns UNAVAILABLE rather than a stale number at the 2022 election date, which is FM-45's lesson working rather than failing.

Not ingested: data/generic_ballot/generic_ballot_polls_2022.csv, 1,207 rows covering the full 2022 cycle, written by an earlier session with no raw_artifact row — its sha256 is not in the archive and which capture it came from is unknown. Bytes whose provenance is "somebody downloaded this" do not enter the corpus.

Priority note. This mattered less than expected. house_prior_too_wide.md shows the 2016 House forecast received a national swing of +4.63 against a realised +4.82 — essentially exact — and still missed by 38 seats. The environment is a real gap for 2018 onward and it is not what breaks the House forecast.

MEDSL per-state precinct files: what each state's file actually contains

Read for the 2024 cycle on 2026-09-25, while building district presidential baselines and certifying House results. These are not complaints about MEDSL, which publishes what the states publish; they are the shapes a consumer has to handle, and each was found because a number that looked reasonable disagreed with something.

The checks that catch these live in scripts/build_district_baselines.py and are worth stating because the order they were added in is the order the failures were found: precinct match rate (caught OK, RI), statewide margin agreement (caught WA, NJ, AZ, KS, MS, LA), vote coverage floor and ceiling (caught LA's luck and IN's duplication), and crosswalk unambiguity (caught OH). Each one caught something the previous ones could not.