feat: agenda ingestion - migration + ingest_agenda.py (MON-210)

[Walid-peach]

What

Adds the agenda ingestion pipeline for MON-110: a new agenda_items table and scripts/ingest_agenda.py, which pulls the AN agenda export and lands séance publique ODJ points in the database.

Why

ADR-030 (MON-209) fixed the table shape and refresh semantics for the agenda feature.
This PR implements exactly that: one denormalized agenda_items table, upserted only, scoped to séance publique.
It unblocks MON-212 (GET /agenda endpoints) and MON-211 (plain-French summaries).

Changes

• data/migrations/009_agenda.sql — creates agenda_items per the ADR-030 column set, CREATE TABLE IF NOT EXISTS (safe to re-run), with indexes on sitting_start (date-window queries) and dossier_id (the join to votes).
• scripts/ingest_agenda.py — downloads/reads the Agenda.json.zip export, filters to @xsi:type = seance_type (séance publique only), flattens each réunion's ODJ point(s) (pointODJ is a bare object for a single point and a list for several — both handled), and upserts on point_uid with last_seen_at stamped on every touched row. Cancellations are reflected via reunion_etat/point_etat and cancelled_at, never via DELETE (upsert-only rule, CLAUDE.md decision 8).
• scripts/run_ingestion_prod.py — wires in an "Agenda" step via run_step(..., critical=False), matching the party-fix/summaries pattern: an agenda-feed outage must not block core deputies/votes/positions ingestion. No workflow file changes needed — .github/workflows/ingest_prod.yml already invokes run_ingestion_prod.py.
• tests/test_agenda_parsers.py + tests/fixtures/agenda_sample.json — pure-function tests over a committed JSON fixture covering: the single-point vs list pointODJ shape, séance-vs-commission filtering, a cancelled réunion (Annulé), a point with no dossierRef, and the --since window.
• CLAUDE.md / README.md — documented the new table and script.

Out of scope by design (ADR-030 / follow-up issues): summary_plain/theme population (MON-211), GET /agenda (MON-212).

Testing

• python3 -m pytest tests/ -m "not integration" -q — 352 passed (baseline unchanged).
• python3 -m ruff check . and python3 -m ruff format --check . — clean.
• Manually built a zip fixture from tests/fixtures/agenda_sample.json and ran fetch_all_reunions + parse_reunions against it end-to-end — correctly parses 4 séance ODJ points, filters out the commission réunion, and reflects the cancelled réunion's état.
• Not tested: the live --zip-path upsert against a running Postgres and the make migrate idempotency check — Docker wasn't available in this environment. The parsing/transform logic (the part with real risk of AN-feed-shape bugs) is covered by the unit tests above; the upsert SQL itself mirrors the already-proven ingest_votes.py pattern.

Risks / Notes

• New migration (009_agenda.sql) — additive only (CREATE TABLE IF NOT EXISTS + two indexes), no changes to existing tables.
• Reviewers should verify the live DB path (make migrate then python scripts/ingest_agenda.py --zip-path <fixture> twice) before merge, since it wasn't possible to run here.
• run_ingestion_prod.py's summary log block now has a 5-line body instead of 4; purely cosmetic.

Breaking Changes

None.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
mon-elu	Ready	Preview	Aug 3, 2026 8:50pm

[github-actions]

🧪 dbt data-health check — ✅ passed

20:48:55  52 of 62 PASS source_not_null_raw_votes_vote_id ................................ [PASS in 1.15s]
20:48:55  56 of 62 START test source_unique_raw_deputies_deputy_id ....................... [RUN]
20:48:55  53 of 62 PASS source_not_null_raw_votes_voted_at ............................... [PASS in 1.07s]
20:48:55  57 of 62 START test source_unique_raw_votes_vote_id ............................ [RUN]
20:48:56  54 of 62 PASS source_relationships_raw_vote_positions_deputy_id__deputy_id__source_raw_deputies_  [PASS in 1.15s]
20:48:56  58 of 62 START test unique_mart_deputy_scorecard_deputy_id ..................... [RUN]
20:48:56  56 of 62 PASS source_unique_raw_deputies_deputy_id ............................. [PASS in 1.09s]
20:48:56  59 of 62 START test unique_mart_party_alignment_deputy_id ...................... [RUN]
20:48:56  55 of 62 PASS source_relationships_raw_vote_positions_vote_id__vote_id__source_raw_votes_  [PASS in 1.39s]
20:48:56  60 of 62 START test unique_mart_vote_summary_vote_id ........................... [RUN]
20:48:56  57 of 62 PASS source_unique_raw_votes_vote_id .................................. [PASS in 1.09s]
20:48:56  61 of 62 START test unique_stg_deputies_deputy_id .............................. [RUN]
20:48:57  58 of 62 PASS unique_mart_deputy_scorecard_deputy_id ........................... [PASS in 1.07s]
20:48:57  62 of 62 START test unique_stg_votes_vote_id ................................... [RUN]
20:48:57  59 of 62 PASS unique_mart_party_alignment_deputy_id ............................ [PASS in 1.08s]
20:48:57  61 of 62 PASS unique_stg_deputies_deputy_id .................................... [PASS in 1.07s]
20:48:57  60 of 62 PASS unique_mart_vote_summary_vote_id ................................. [PASS in 1.10s]
20:48:58  62 of 62 PASS unique_stg_votes_vote_id ......................................... [PASS in 1.06s]
20:48:59  
20:48:59  Finished running 62 data tests in 0 hours 0 minutes and 23.46 seconds (23.46s).
20:48:59  
20:48:59  Completed successfully
20:48:59  
20:48:59  Done. PASS=62 WARN=0 ERROR=0 SKIP=0 TOTAL=62

Validates prod data health. A failure means current prod data is stale/broken, not necessarily that this PR is wrong.

[Walid-peach]

Attention Score

82/100 - SAFE TO SKIM

• Base 100
• Should Fix count: 1 → -6
• Live-DB path (idempotency, migration apply) not executed by the author → -8
• Non-empty Testing/Validation Gaps section → -4
  Reason: The change is additive (new table + new script), the parsing logic is thoroughly unit-tested against real AN-shaped fixtures, and the only real gap is a DB-environment limitation the author already flagged.

Summary

Implements the agenda_items table and ingest_agenda.py exactly as scoped by ADR-030/MON-209: séance-publique-only ODJ points, soft-state upsert with last_seen_at, cancellations reflected via reunion_etat/point_etat/cancelled_at rather than DELETE. Wired into run_ingestion_prod.py as a non-critical step, matching the existing party-fix/summaries pattern. Parsing logic (single-vs-list pointODJ, séance/commission filtering, cancelled réunion, missing dossierRef) is covered by a new pure-function test file against a committed JSON fixture, and the full non-integration suite plus ruff pass.

Must Fix

None.

Should Fix

• ingest_agenda.py's standalone --since default (90 days) differs from run_ingestion_prod.py's effective default (365 days, inherited from args.since). Since the orchestrator always passes --since explicitly, this only bites someone running ingest_agenda.py directly without a flag — worth a one-line comment or aligning the default so it's not a surprise later.

Nice to Have

• _first_ref silently takes only the first dossierRef when a point carries several. Given the ADR's dossier_id is a single TEXT column this is the right call, but a short comment noting the multi-dossier case is intentionally collapsed would save a future reader from wondering.
• point_etat can end up stale relative to reunion_etat if the AN feed cancels the réunion without updating each point's own cycleDeVie — not a bug in this script (it stores what the feed says), but worth a note for MON-212 since the ADR's freshness/state filtering leans on both fields.

Testing / Validation Gaps

• The acceptance criteria "make migrate applies cleanly" and "a second run over the same fixture leaves the row count unchanged" were not verified against a live Postgres — Docker wasn't available in this environment (already called out in the PR description). Recommend a reviewer runs make migrate + python scripts/ingest_agenda.py --zip-path <fixture> twice before merging, to confirm the upsert/idempotency behavior end-to-end.

Documentation / Reviewer Notes

• CLAUDE.md and README.md were updated with the new table/script — accurate and consistent with the ADR-030 column set.
• No dbt mart added, consistent with the ADR's explicit "no mart" decision.

Verdict

Ready with minor changes