Integration v4-cohort: 8-worker pipeline quality uplift (Session 1 + Workers A–H) - #4
Closed
mdmurphy822 wants to merge 19 commits into
Closed
Integration v4-cohort: 8-worker pipeline quality uplift (Session 1 + Workers A–H)#4mdmurphy822 wants to merge 19 commits into
mdmurphy822 wants to merge 19 commits into
Conversation
Promotes Ed4All out of "prototype with accurate-looking metrics" into "v0.1.x with honest self-evaluation." Nine defects surfaced by the WCAG_201 package are the roadmap. Priority 1 - Pipeline self-trust (Trainforge/process_course.py, align_chunks.py): - html_preservation_rate now measures tag balance via stdlib HTMLParser - learning_outcome_coverage measures referential integrity (not field presence) - Adds footer_contamination_rate, follows_chunk_boundary_violations, factual_inconsistency_flags, html_balance_violations, broken_refs, orphan_week_scoped_refs - quality_report.json now carries metrics_semantic_version=2 + methodology block - PipelineIntegrityError + strict_mode refuse to write when core invariants fail (off by default in v0.1.x per severity-flip trigger in VERSIONING.md) - follows_chunk resets at every lesson boundary Priority 2 - Architectural commitments: - Orphan week-scoped outcome IDs preserved in new pedagogical_scope_refs with parent_id=null, status=orphan - the plan's Option 2 explicitly - Concept graph edges carry relation_type=co-occurs (forward-compat for v1.0 typed extractor); pedagogy/logistics tags partitioned into pedagogy_graph.json - Defensive boilerplate detector at Trainforge/rag/boilerplate_detector.py Priority 3 - Narrative honesty: - README concept-graph claim scoped to what the pipeline actually ships - New VERSIONING.md with explicit v1.0 exit criteria (including >=3 domain regression runs outside accessibility) and NSF grant-narrative framing - Investigation-first §4.4a slot for the 47% enrichment miss rate Priority 4 - Mechanical fixes: - WCAG SC name canonicalization (Trainforge/rag/wcag_canonical_names.py) applied to chunk text, key_terms, misconceptions, and concept_tags - ContentFactValidator (lib/validators/content_facts.py) flags mismatched numeric claims (WCAG SC count, Section 508 SC count) and internal arithmetic contradictions - LeakChecker.check_corpus_boilerplate extends leak detection to cross-chunk repeated boilerplate - Enrichment fallback helpers (derive_bloom_from_verbs, extract_key_terms_from_html, extract_misconceptions_from_text) shipped as unwired module-level utilities pending §4.4a investigation New workflow gates (warning, flip trigger documented in VERSIONING.md §3): - outcome_ref_integrity - content_fact_check - leak_check threshold.max_boilerplate_chunk_fraction = 0.10 Test fixtures split into mini_course_clean/, mini_course_defective/, mini_course_edge/ so every failure names exactly one defect class. 33 regression tests green; ruff clean; ci/integrity_check 7/7. https://claude.ai/code/session_017bY9JPjoGjeGxaY38JtfWY
…deferral Four follow-up items from the post-merge review: 1. archive/v0.1.0-baseline/ scaffold with explicit ARCHIVE_README naming what must be populated and why it can't be deferred. Empty in this environment — the WCAG_201 artifact is not in this checkout. The repo owner must populate the slot from their copy before the v1.0 branch begins, or invoke the documented fallback (rebuild from 18c6613). Committing the empty scaffold makes the obligation structural rather than a todo item that can drift. 2. VERSIONING.md §4 hypothesis list: add H5 (JSON-LD parser silent failure on malformed JSON / unexpected schema variants). Distinguished from H2 because the fix is in the parser, not the source. Without this, the §4.4a investigation could attribute parser failures to H1/H3/H4 and miss the real root cause. 3. VERSIONING.md §3 Severity flip trigger: require BOTH the synthetic mini_course_clean/ floor AND a real-domain clean regeneration. Either alone is a weaker bar than the NSF narrative implies. The follow-up PR cannot cite "CI green on synthetic" alone as justification. 4. VERSIONING.md §4b: explicit deferral table for Courseforge §2.1 + §2.3. The "ownership: both" architectural decision is partially fulfilled, not retracted — Trainforge half ships, Courseforge half is named for a follow-up branch (claude/courseforge-template-chrome-and-dual-ids). The Trainforge defensive layer is acknowledged as load-bearing, which is the principal reason strict_mode=True is not on by default. 5. lib/validators/content_facts.py: negative-context suppressor for the wcag_2_2_sc_count and arithmetic checks. Suppresses when the claim sentence contains "WCAG 2.0", "WCAG 2.1", "historically", "previously", "formerly", "used to", "section 508", or hypothetical framings ("if there were", "suppose", "imagine"). Section 508 SC count check is preserved (it has its own pattern + expected value). Without this, the post-flip critical-severity gate would false-positive on legit historical mentions. Tests: 37 green (33 prior + 4 new suppressor cases). Lint clean, integrity 7/7. https://claude.ai/code/session_017bY9JPjoGjeGxaY38JtfWY
Add five flow metrics to the base-pass quality_report.json under `metrics`: content_type_label_coverage, key_terms_coverage, key_terms_with_definitions_rate, misconceptions_present_rate, interactive_components_rate. These surface silent metadata drops between the HTML parser and _create_chunk that the prior metrics didn't see. Scope is observability-only — metrics do not feed overall_quality_score. Follows ADR-001 Contract 2: only the base pass bumps METRICS_SEMANTIC_VERSION; alignment is untouched. Integrity fields attached where per-chunk failure lists are useful: chunks_with_empty_definitions, chunks_missing_misconceptions. The misconceptions denominator is threaded via self._pages_with_misconceptions populated in _chunk_content (set of lesson_ids whose JSON-LD declared ≥1 misconception); falls back to all-chunks when no page declared any, with methodology string announcing the fallback. FOLLOWUP-WORKER-B-1: interactive_components live on parsed_items but not on chunks; this metric uses a regex fallback against chunk HTML (HTMLContentParser.COMPONENT_PATTERNS). Promoting interactive_components to a first-class chunk field belongs to Worker E. WCAG_201 regeneration confirms metrics_semantic_version == 4 and the five new keys land in the report. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
…SION v4)
Adds a 2–3 sentence per-chunk `summary` field plus an optional
`retrieval_text` (summary + key_terms) for dense-retrieval recall
augmentation. Declares `CHUNK_SCHEMA_VERSION = "v4"` per ADR-001
Contract 1; stamps it on every chunk (`schema_version`) and on
`manifest.json` (`chunk_schema_version`).
Summary generator (`Trainforge/generators/summary_factory.py`) is pure,
deterministic, and biased toward the opening sentence plus any
LO-tag-bearing sentence. LLM path is opt-in behind `mode="llm"` with a
caller-supplied callable; the extractive path MUST ship without it.
Benchmark (`Trainforge/rag/retrieval_benchmark.py`) reuses LibV2's
`LazyBM25` and compares BM25 recall@{1,5,10} across three chunk-text
variants (text, summary, retrieval_text). Activated via the new
`--benchmark-retrieval` CLI flag; writes `quality/retrieval_benchmark.json`.
Held-out question set is derived from `course.json::learning_outcomes`:
every LO statement is a query whose relevant chunks are those whose
`learning_outcome_refs` contain the LO id. Synthetic but reproducible.
WCAG_201 regeneration numbers (131/131 chunks carry `summary`):
- text recall@5 = 0.0369
- summary recall@5 = 0.0376
- retrieval_text recall@5 = 0.0399 (ship)
`retrieval_text` is emitted on chunks that carry `key_terms` (69/131 on
WCAG_201); for chunks without key_terms the benchmark's variant builder
falls back to summary alone.
Fixture `Trainforge/tests/fixtures/mini_course_summaries/` (6 chunks,
2 LOs) exercises both the generator and the benchmark. 18 new tests
covering determinism, length bounds, LO-tag sentence selection, schema
version stamping on chunks and manifest, LLM opt-in + fallback,
recall@k arithmetic, question-set derivation, and write-artifact shape.
All 130 Trainforge tests pass (18 new + 112 existing; no regressions).
Per ADR-001 Contract 1: this is the first worker of B/D/E to land the
CHUNK_SCHEMA_VERSION declaration; Workers B and E branch from
`chunk-schema-v4` and amend `docs/schema/chunk-schema-v4.md` in place
when they ship. Human consolidates the rebase after both return.
Add a new Trainforge stage, `Trainforge/synthesize_training.py`, that reads
an already-enriched `corpus/chunks.jsonl` and emits two training artifacts
under `training_specs/`:
- `instruction_pairs.jsonl` (SFT format)
- `preference_pairs.jsonl` (DPO format)
plus the matching counts on `training_specs/dataset_config.json` under
`statistics.instruction_pairs` / `statistics.preference_pairs`.
Pipeline surface
- New CLI flags on `process_course.py`: `--synthesize`,
`--synthesis-provider {mock,anthropic}`, `--synthesis-seed`.
- The stage runs after alignment and before LibV2 import so synthesized
artifacts are copied verbatim by the importer.
- Deterministic mock provider is the MVP. The `anthropic` provider slot
exists in both factories but raises `NotImplementedError` — follow-up.
Factories
- `Trainforge/generators/instruction_factory.py` — one-pair-per-call
factory with a Bloom-level × content-type template catalog
(6 × 5 = 30 templates) and deterministic seeding.
- `Trainforge/generators/preference_factory.py` — draws `rejected` from
`chunk.misconceptions[0]` when present, else synthesizes via
deterministic negation swaps; enforces the Jaccard-delta gate.
Quality gates (all deterministic, asserted in tests)
- prompt 40-400 chars, completion 50-600 chars
- no 50+-char verbatim span from `chunk.text` leaks into the prompt
- `chosen != rejected` with token-Jaccard delta ≥ 0.3
- every pair carries a `decision_capture_id` that resolves in the
capture log
ADR-001 Contract 3
- Establishes `lib.decision_capture.ALLOWED_DECISION_TYPES` as a new
tuple constant (advisory; not enforced in `log_decision` because many
legacy call sites still emit uncatalogued types). Adds
`instruction_pair_synthesis` and `preference_pair_generation`.
Tracked follow-up `FOLLOWUP-ADR001-5` for enforcement.
Schemas
- `schemas/instruction_pair.schema.json`
- `schemas/preference_pair.schema.json`
Fixture
- `Trainforge/tests/fixtures/mini_course_training/` — 15 pre-enriched
chunks covering misconception-backed pairs, rule-synthesized pairs,
an orphan chunk (no LO refs) for the eligibility filter, and four
Bloom × content-type combinations. README documents CI assertions.
Tests (12, all passing)
- factory determinism under seed (instruction + preference)
- schema validation on emitted pairs
- malformed-pair rejection diagnostic
- 50-char verbatim-span rule
- chosen != rejected with Jaccard delta ≥ 0.3
- length-gate enforcement
- decision_capture_id resolves for every pair
- LO filter skips orphan chunks
- integration: volume floor on mini_course_training fixture
- stage idempotence at same seed
- dataset_config.json statistics update
Regeneration (WCAG_201, 131 chunks):
131 instruction pairs, 131 preference pairs, 0 rejected.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
…LM pass) Adds graph/concept_graph_semantic.json alongside the existing concept_graph.json. Edges carry one of three types — is-a, prerequisite, related-to — plus per-edge provenance (rule name, rule version, evidence). Rule modules (Trainforge/rag/inference_rules/): - is_a_from_key_terms: parses key_terms[].definition for "is a type of X" phrasings; emits edges only when both child and parent resolve to nodes in the co-occurrence graph. - prerequisite_from_lo_order: uses course.json learning_outcomes ordering and chunks[].learning_outcome_refs; when concept A first appears at an earlier LO than B's first chunk, emits B --prerequisite--> A. - related_from_cooccurrence: re-emits co-occurrence edges above a configurable threshold (default 3) as typed related-to edges. Orchestrator (Trainforge/rag/typed_edge_inference.py) applies precedence is-a > prerequisite > related-to on collisions and gates the optional LLM escalation pass behind llm_enabled (OFF by default). The default path is byte-identical across runs when generated_at is held fixed. Process_course.py wires a new stage that builds the semantic graph from chunks + course.json + concept_graph.json and writes the artifact in _write_metadata. CLI flag --typed-edges-llm opt-in; deterministic when off. Artifact shape validated by schemas/concept_graph_semantic.schema.json. Smoke fixture at Trainforge/tests/fixtures/mini_course_typed_graph/ exercises all three rules plus precedence; 9 unit tests (including the eight mandated in the Worker F spec) pass. Follows ADR-001 Contract 3: adds a typed_edge_inference decision type as a free string (ALLOWED_DECISION_TYPES enum still doesn't exist in tree; Worker C owns creation per the contract). Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
…ase) Four pipeline defects identified from the published LibV2 WCAG_201 package, closed on Trainforge's base pass before the multi-worker coordination phase. All regeneration improvements land without touching alignment pass or chunk schema; Workers B-F rebase on top of this commit. 1. Bloom fallback wiring. _create_chunk now runs the full chain: section JSON-LD -> page JSON-LD -> parsed LOs -> derive_bloom_from_verbs -> "understand" default. Emits bloom_level_source when below lo_inherited confidence. Closes 17/131 chunks that previously shipped with no bloom. 2. Domain concept extraction. Loads optional domain_concepts from objectives JSON, compiles word-boundary patterns, merges JSON-LD keyTerms into concept_tags (existing NON_CONCEPT_TAGS filter still applied). Closes the sparse-graph failure mode where CONCEPT_PATTERNS was pedagogy- only and domain vocabulary (WCAG/ARIA/POUR) was absent from the graph. 3. Uncovered outcomes in quality_report. New metric outcome_reverse_coverage + integrity.uncovered_outcomes list catches LOs with zero chunk refs (WCAG_201 was silently dropping 4 of 28 LOs past the chunk-coverage metric). 4. Rewritten _build_pedagogy_summary. Now consumes chunks and emits module_sequence, bloom_progression, prerequisite_chain, and prerequisite_violations grounded in real chunk data. Previous stub was a 5-key static dict. Also: feedback moved to LOGISTICS_TAG_SET so it routes to pedagogy_graph rather than polluting concept_graph. METRICS_SEMANTIC_VERSION 2 -> 3. WCAG_201_objectives.json already carried the domain_concepts seed list. 21 new tests across 6 classes in test_generator_defects.py. Full suite: 55 pass in this file, 130 across Trainforge/tests, zero regressions. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Adds a LibV2-level indexer that scans every course's graph/concept_graph.json and emits LibV2/catalog/cross_package_concepts.json -- a catalog of which concepts appear across which courses, with optional typed-edge cross- references when Worker F's concept_graph_semantic.json is present. When no course carries a semantic graph (the case for every course built before Worker F), cross_package_edges degrades gracefully to an empty list on every concept. Deterministic ordering is guaranteed: concepts sorted by total_courses desc then alphabetically; per-concept edges sorted by (target, type, course_slug). Wires a new check_cross_package_index_freshness() into lib/libv2_fsck.py's top-level runner (LibV2Fsck.check_all), comparing the catalog's generated_at against every course graph's mtime so a stale catalog surfaces as a warning-severity FsckIssue with category "stale_catalog". Adds the 'libv2 cross-index' subcommand (click-style, matches existing CLI) with auto-detected --repo-root and a default output path under LibV2/catalog/. Committed example artifact covers the two LibV2 courses with graphs in this repo (best-practices-in-digital-web-design-for-accessibi, foundations-of-digital-pedagogy) -- 92 unique concepts, zero cross-package edges since neither course was built with Worker F yet. A follow-up tracks regenerating LibV2 with the integration pipeline so typed edges start appearing. Also whitelists LibV2/catalog/cross_package_concepts.json in .gitignore (LibV2/catalog/* is otherwise ignored). 10 new tests pass; no regressions in LibV2/tests/. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
…LO-fanout)
The content-generator agent was writing course_data JSON with week-local
LO IDs (W01-CO-01..W12-CO-04). Each week independently numbered its CO
list 01..04, and Trainforge strips the W0N- prefix during
learning_outcome_refs normalization, so chunks from all 12 weeks
collapsed onto the same four canonical IDs (co-01..co-04). Against the
28 declared outcomes in course.json this drove
outcome_reverse_coverage=0.143 with 24 uncovered outcomes, capping
training-pair diversity, recall@k, and prerequisite-edge inference for
every downstream worker.
Fix (deterministic, script-only; no agent invocation):
- generate_course.py gets load_canonical_objectives() and
resolve_week_objectives(), plus a new --objectives CLI flag. When
supplied, each week's objectives are overridden with the canonical
subset for that week (all TOs plus the COs whose "Week N-M" chapter
range covers the week number). The week->chapter regex matches the
one Trainforge.process_course.load_objectives already uses.
- validate_page_objectives.py walks generated HTML and asserts every
JSON-LD learningObjectives block references only IDs declared for
the page's inferred week, so the defect cannot silently reappear.
- tests/test_generate_course_lo_specificity.py (9 tests) covers the
selection function (weeks 1/3/0/unmapped), the end-to-end emitter
(week 3 pages carry canonical IDs, no week-local leakage), the
legacy behaviour when --objectives is omitted, and the validator
failing on the exact buggy pattern.
- docs/per-week-learning-objectives.md explains the rule, the
emit/validate loop, and the before/after metrics.
Verified against WCAG_201: regenerated 90 pages + IMSCC, re-ran
Trainforge. outcome_reverse_coverage 0.143 -> 1.0, uncovered_outcomes
24 -> 0, 28 distinct LO refs distributed across the corpus.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
WCAG_201 regenerated fully (Courseforge generate_course.py --objectives +
Trainforge with integration branch). Results:
- outcome_reverse_coverage: 0.143 -> 1.0
- uncovered_outcomes: 24 -> 0
- distinct LO refs: 4 -> 28
- all 131 chunks carry summary, html_xpath, char_span, schema_version=v4
- concept_graph_semantic.json: 1451 related-to edges
DIGPED_101 re-run through Trainforge on its existing (pre-H) IMSCC. Picks
up B/D/E/F improvements; the LO-fanout defect persists on DIGPED because
its Courseforge regen is gated on inputs/course-data/ files that don't
ship today. Tracked as FOLLOWUP-WORKER-H-2.
Cross-package catalog rebuilt:
- course_count: 2 -> 2 (unchanged)
- concept_count: 92 -> 282
- typed cross_package_edges: 0 -> 5 (accessibility x4, cognitive-load x1)
- scaffolding pollution ('due') no longer in top shared concepts
LibV2/courses/*/ content itself is gitignored (repository by design);
only catalog/cross_package_concepts.json tracks as the index artifact.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: cde1a13d60
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Comment on lines
+743
to
+744
| existing_broken = integrity.get("broken_refs", []) | ||
| integrity["broken_refs"] = existing_broken + broken_refs |
There was a problem hiding this comment.
update_quality_report appends new broken_refs to whatever was already in quality_report.json, so stale base-pass violations remain even after alignment fixes those references. In a normal rerun flow this produces false integrity failures (the report can still show old broken refs while learning_outcome_refs_coverage is 1.0), which makes downstream quality checks and dashboards unreliable. The integrity list should be replaced (or at least deduplicated against current chunks), not cumulatively appended.
Useful? React with 👍 / 👎.
Comment on lines
+480
to
+482
| graph_file = course_dir / "graph" / "concept_graph.json" | ||
| if not graph_file.is_file(): | ||
| continue |
There was a problem hiding this comment.
The staleness detector only looks at graph/concept_graph.json, but cross_package_concepts.json also incorporates typed edges from graph/concept_graph_semantic.json. If a course updates only the semantic graph (for example, after typed-edge regeneration), check_cross_package_index_freshness reports the catalog as fresh and misses a genuinely stale cross-package index.
Useful? React with 👍 / 👎.
5 tasks
Owner
Author
|
Superseded by dev-v0.2.0 which merges integration/v4-cohort + worker-i + worker-j + worker-k. |
7 tasks
mdmurphy822
pushed a commit
that referenced
this pull request
May 2, 2026
Lay the scaffolding for the generalized remediation prompt-suffix builder per `plans/phase3_5_post_rewrite_validation.md` §A Subtask 1 (pre-resolved decision #3 + #4): module docstring, imports of `GateResult` / `GateIssue` from `MCP.hardening.validation_gates`, the 8-key `_REMEDIATION_DIRECTIVES_BY_GATE_ID` directives table (4 outline + 4 rewrite gate IDs) per the cross-tier overlap pattern the plan specifies, and stub bodies for `_append_remediation_for_gates`, `_append_preserve_remediation`, `_missing_preserve_tokens` raising `NotImplementedError`. Subtasks 2 and 3 fill in the bodies; Subtask 4 lands ≥8 tests. https://claude.ai/code/session_01Se8hZK7RFScb6R4mfe9FgX
mdmurphy822
pushed a commit
that referenced
this pull request
May 3, 2026
…lock_types + document two-pass env var (Phase 5-prep) Apply 6 surgical amendments to plans/phase5_independent_stages.md so it matches HEAD e5603ac before execution dispatch: 1. (MAJOR) Drop `courseforge-classify` subcommand. Phase 4 wired Bloom classification inline as the `bloom_classifier_disagreement` validator gate (config/workflows.yaml:1129, 1334), not as a standalone tier. There is no `_run_classification` handler in pipeline_tools.py. Operators re-check Bloom alignment via `courseforge-validate`. Subcommand list shrinks 5 -> 4. §12 sequencing, §13 risks, §14 open questions, and `03_classification/` output directory all updated. 2. (MINOR) `--blocks` vocabulary aligned with the actual 16 singular `BLOCK_TYPES` enum at Courseforge/scripts/blocks.py:77 (objective, concept, example, assessment_item, explanation, prereq_set, activity, misconception, callout, flip_card_grid, self_check_question, summary_takeaway, reflection_prompt, discussion_prompt, chrome, recap). Plan §3 originally listed plural names (assessments, examples, etc.) that don't match the enum. Open Question #4 resolved. 3. (MINOR) Document `COURSEFORGE_TWO_PASS=true` env-var requirement. The 4 target phases (content_generation_outline, inter_tier_validation, content_generation_rewrite, post_rewrite_validation) carry `enabled_when_env: 'COURSEFORGE_TWO_PASS=true'`. Recommended approach: auto-set inside each subcommand handler with a `--no-two-pass` opt-out for operators wanting the legacy single-pass code path. 4. (MINOR) Refresh line citations against MCP/core/workflow_runner.py that drifted post-Phase 6 + 7a/b/c: - run_workflow at :802 (was :798) - _completed check at :860 (was :798) - _synthesize_dart_skip_output at :1324 (was :1171) - _synthesize_course_planning_reuse_output at :1400 (was :1247) - _dependencies_met at :1643 (was :811) 5. (MINOR) Expand the upstream-phase pre-populate list with phases that landed post original plan authoring: chunking (Phase 7b), concept_extraction (Phase 6), imscc_chunking (Phase 7c). Document the full post-Phase-7c dependency chain. 6. (MINOR) Document `02_validation_report/report.json` as a NEW writer that Phase 5 must add. The shipped `_run_inter_tier_validation` emits JSONL only (blocks_validated_path, blocks_failed_path); the operator-facing per-block aggregation report is a Phase 5 deliverable. Schema documented inline in §6. Plan-only — no production code, schema, workflow YAML, or test files touched. Plan is now ready for the recommended 2-wave execution dispatch. https://claude.ai/code/session_01Se8hZK7RFScb6R4mfe9FgX
mdmurphy822
pushed a commit
that referenced
this pull request
May 3, 2026
…tion note Two surgical amendments per the plan review worker's findings: 1. Lock cip29/bert-blooms-taxonomy-classifier as the chosen replacement for the deleted kabir5297/bloom_taxonomy_classifier in pre-resolved decision #4. Open Question §1 closes (RESOLVED). Documents the _CIP29_TO_BLOOM label-map translation table requirement (contents determined at SHA-pin time by reading the model's id2label config) and records the rejected alternatives (Malithi200, phoenix28). 2. Add ST 3 + ST 5 merge-coordination note to Wave B execution sequencing. Both subtasks edit MCP/core/workflow_runner.py::_LEGACY_PHASE_PARAM_ROUTING — ST 3 adds libv2_root entries to three phases; ST 5 adds Phase 6/7c.5 routing to course_planning + libv2_archival. Different keys + different phases but adjacent lines in the same dict. Sequencing ST 3 → ST 5 prevents merge conflicts. ST 5's "Depends on" line carries the same note for the executor. Plan is now execution-ready. Waves A (HIGH parallel: ST 1, 2, 6, 7), B (MEDIUM dep-chained: ST 3 → ST 5), C (LOW user-resolved + config: ST 4, 8, 9, 10) ready for dispatch. https://claude.ai/code/session_01Se8hZK7RFScb6R4mfe9FgX
mdmurphy822
pushed a commit
that referenced
this pull request
May 3, 2026
…T 9) Logs the operator's Phase 8 Subtask 9 decision to replace the deleted kabir5297/bloom_taxonomy_classifier ensemble member with cip29/bert-blooms-taxonomy-classifier (414 downloads, apache-2.0, BERT-based 'generated_from_trainer' provenance). Subtask 4 will consume this capture's selection to: 1. Replace the 'main' placeholder revision with a concrete HuggingFace SHA. 2. Add a _CIP29_TO_BLOOM label-map translation table mirroring _SST2_TO_BLOOM at lib/classifiers/bloom_bert_ensemble.py:100-103. Adds bert_ensemble_replacement_selection to the canonical decision_type enum in schemas/events/decision_event.schema.json (alphabetical insertion between bert_ensemble_member_loaded and block_escalation, preserving the Phase 4.5 alphabetical contract from commit 3184f1a). Alternatives considered (rejected): Malithi200/bloom-taxonomy-classifier (no 'bert' name match), phoenix28/bloom-taxonomy-classifier (only 18 downloads — insufficient usage signal). JSONL artifact validates against schemas/events/decision_event.schema.json. Plan reference: plans/phase8_cleanup.md::Subtask 9 (pre-resolved decision #4). https://claude.ai/code/session_01Se8hZK7RFScb6R4mfe9FgX
mdmurphy822
pushed a commit
that referenced
this pull request
May 3, 2026
…p29 + add _CIP29_TO_BLOOM (Phase 8 ST 4) Replaces the deleted upstream kabir5297/bloom_taxonomy_classifier with cip29/bert-blooms-taxonomy-classifier (per operator decision logged 2026-05-03 — see plan ST 4 Decision #4) and pins all three ensemble members to concrete 40-hex-char HuggingFace commit SHAs. Resolved SHAs (huggingface_hub.HfApi().model_info(repo_id).sha): - cip29/bert-blooms-taxonomy-classifier: ae343e4f - distilbert-base-uncased-finetuned-sst-2-english: 714eb0fa - MoritzLaurer/DeBERTa-v3-base-mnli-fever-anli: 6f5cf0a2 The cip29 model emits generic LABEL_0 ... LABEL_5 labels (verified by fetching its config.json — id2label is the default LABEL_N form, no semantic mapping). New _CIP29_TO_BLOOM translation table (mirrors the _SST2_TO_BLOOM pattern at :100-103) maps the generic labels onto canonical Bloom levels following standard hierarchical ordering: LABEL_0 -> remember, LABEL_1 -> understand, LABEL_2 -> apply, LABEL_3 -> analyze, LABEL_4 -> evaluate, LABEL_5 -> create. The translation runs inside _classify_with_member against the model's raw argmax output before the resulting Bloom level is returned to the ensemble vote aggregator. Module docstring updated: removes the "Phase 4 followup" callout (SHAs are now concrete, not placeholder "main") and documents the Phase 8 ST 4 swap rationale plus the new 40-hex-char regex contract. Test additions (lib/classifiers/tests/test_bloom_bert_ensemble.py): - test_default_ensemble_revisions_are_concrete_commit_shas: every default member's revision matches ^[0-9a-f]{40}$ (locks the no-rollback-to-tag-style-refs contract) - test_default_ensemble_first_member_is_cip29_replacement: first member is cip29; kabir5297 string nowhere in registry - test_cip29_to_bloom_covers_all_canonical_levels: every value in _CIP29_TO_BLOOM is a canonical Bloom level + every canonical level appears in the table values (full coverage) - test_cip29_to_bloom_keys_are_label_n_form: every key matches ^LABEL_[0-5]$ + exactly 6 entries All 10 ensemble tests + 12 disagreement validator regression tests pass (22 total). https://claude.ai/code/session_01Se8hZK7RFScb6R4mfe9FgX
mdmurphy822
added a commit
that referenced
this pull request
May 8, 2026
…m_support_check rationale fields Per W-D11 debt sweep finding #4: the rationale-content description for ``pair_claim_support_check`` documented only the W5.D structured- attribution signals and was silent on the three W-D11 evidence_quote rates that T11.1 (commit 58650de) and T11.2 (commit 5b9445e) added to the block-side and pair-side validators respectively. Refreshes the description string to enumerate ``evidence_quote_coverage_rate``, ``evidence_quote_substring_fail_rate``, and ``evidence_quote_char_span_mismatch_rate`` with the canonical interpolation format, and adds a parallel signal-contract clause for the previously-undocumented ``claim_support_check`` block-side mirror. Documentation-only change inside a JSON Schema description field — no enum changes, no shape changes; ``schemas/tests/`` stays green. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
mdmurphy822
added a commit
that referenced
this pull request
Jun 23, 2026
… + B15 Resources (loop cycle 1)
Works through the first batch of the FRAMEWORK.pdf progress-review action set. No stubs
(real artifacts/contracts only); slug-free (dynamic discovery); byte-stable when the IB
flags are off; warning-day-1 gates with # TODO(calibration) deferred flips.
Calibration harness (KEYSTONE — unblocks the ~9 deferred gate-family critical-flips):
- scripts/calibration_harness.py — pure measurement (never mutates/flips). Slug-free
discovery of LibV2 rollup reports + Courseforge per-block GateResult reports + decision
JSONL; corpus-IDENTITY collapsing (the 10 timestamped runs of one textbook = 1 distinct
corpus); a 15-gate-family flip-criteria table with auditable expected-bands; emits
calibration_report.json with per-gate fire-rate + sample + flip_ready. On this box: 1
distinct corpus → every family flip_ready=false (honest). Flipping needs a 2nd
distinct-corpus run (+ ED4ALL_BLOCK_QUALITY_RUBRIC/IB flags on). 9 tests.
Keyboard + time-based-media a11y (real WCAG holes, not flag-flips) — lib/validators/rewrite_html_shape.py:
- BLOCK_KEYBOARD_OPERABLE (2.1.1): custom click-only non-native control w/o keyboard
affordance flagged; native button/a/input/details = escape hatch.
- BLOCK_FOCUS_VISIBLE (2.4.7): inline outline:none w/o replacement focus indicator flagged.
- B04 per-piece: MULTIMEDIA_{CONTROLS,CAPTIONS,AUDIO_DESC,TRANSCRIPT}_MISSING (1.2.2/1.2.4/
1.2.5) — AD check matches the actual renderer output (class="audio-description"). 19 tests.
B15 Resources (closes the last canonical-catalog gap — every B-code now has an Ed4All primary):
- resources block_type → framework_block B15 (block_catalog.yaml), IB5 default-off posture;
_render_resources_section (accessible descriptive links, never bare-URL); new
ResourceLinkPurposeValidator (2.4.4, RESOURCE_LINK_PURPOSE_UNCLEAR) wired warning-day-1 at
post_rewrite_validation in both two-pass workflows; planner nudge (further-reading shape);
JSON-LD enum + router-policy matrix + count tests 28->29; IB2 reconciliation table +
gate-count table re-derived (62/107/169). 19 tests.
Combined sweep: 723 passed. Pre-existing-unrelated: test_outline_seam_uses_block_validators
(fails on clean stash — IB3 outline-seam, not touched here).
Remaining in the action set: #4 anchored-rubric producer + #6 rollup->final_status wiring
(next iteration); #2 verb-triple flip + #6 enforcement are blocked on a 2nd-corpus FP
measurement (a GPU cycle). pipeline_tools.py decomposition stays deferred (monkeypatch
contract). .gitignore: extracted/ + calibration_report.json.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01QfgokVwS8UMGNAmWR1SohW
mdmurphy822
added a commit
that referenced
this pull request
Jun 23, 2026
…ate wiring (loop cycle 1 cont.) Continues the FRAMEWORK.pdf action set. No stubs (grounded/deterministic); byte-stable when the IB flags are off; the new gate is warning-day-1 with a # TODO(calibration) deferred flip. #4 — Anchored-rubric PRODUCER (closes the IB3.4 validator-without-producer gap): - lib/generation/anchored_rubric.py — deterministic, grounded (no LLM): authors Block.anchored_rubric for scored Evaluate/Create blocks (criteria + per-band exemplars + published_before_task), grounding criteria/exemplars in the objective's resolved Bloom verb + the assessed concept (abcd.action_object → statement phrase → key_terms); returns None (validator flags honestly) when nothing grounds — never fabricates. Gated on ED4ALL_ALIGNMENT_VERB_TRIPLE (same flag as the consumer), field hash-excluded. - Wired into MCP/tools/pipeline_tools.py: producer hook in _run_content_generation_outline (after _align_outline_blocks_to_objectives), serializer emits anchored_rubric only-when-non-None, + the 4 _entry_to_block rehydrators carry it through to post_rewrite_validation. Tests prove AnchoredRubricValidator now PASSES on a produced block. #6 — Rollup → workflow gate (FR-07/13 / §6.5 course-level): - lib/validators/block_quality_rollup.py::BlockQualityRollupValidator — scores the rewrite-tier blocks via the canonical IB6.1 rubric scorer (single owner), feeds BlockQualityRollupAggregator (single owner of the rollup math), returns a GateResult enumerating course/block hard-gate fails. Wired at post_rewrite_validation in both two-pass workflows, severity: warning, with a # TODO(calibration) flip path (→ critical + passed=course_pass once scripts/calibration_harness.py confirms FP rate on >=2 corpora). Byte-stable no-op when ED4ALL_BLOCK_QUALITY_RUBRIC is off. - Gate count re-derived: 62 critical / 109 warning / 171 total (cg 31 warn, ttc 75 warn). Hygiene: deleted the stale untracked scripts/wave76_*.py (long-pending), resolved 5 stale unmerged index flags (content==HEAD, no-op). Combined sweep: 59 passed across the touched suites. Pre-existing-unrelated: test_outline_seam_uses_block_validators (IB3 outline seam, fails on clean HEAD). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01QfgokVwS8UMGNAmWR1SohW
mdmurphy822
added a commit
that referenced
this pull request
Jul 1, 2026
Self-review of the Wave 63 SHACL implementation surfaced six issues. This wave closes all of them, with regression tests for each, plus one pyld interaction bug caught by the end-to-end round trip. Fixes (by Wave 63 review item): #1 Closed-set vocab validation. bloomLevel / cognitiveDomain / hierarchyLevel / bloomRange now use sh:in against the canonical concept IRI set instead of sh:pattern prefix matching. Wave 63's prefix check accepted typos like <https://ed4all.dev/vocab/bloom#aplly> because they shared the namespace prefix; Wave 67 rejects them. #2 parentObjectiveId now carries sh:pattern ".*[/#](TO|CO)-\\d{2,}$" on the IRI's lexical form. Wave 63 only required sh:nodeKind sh:IRI, letting any URL through — including non-LO URIs. #3 New cfshapes:SectionShape targeting ed4all:Section. Requires schema:name (heading) and validates bloomRange items. Wave 63 had no Section shape at all; malformed sections validated silently. #4 NodeShape target classes now point at ed4all:CourseModule / ed4all:LearningObjective / ed4all:Section rather than their Schema.org equivalents. Wave 63 over-targeted — a Pearson-emitted schema:LearningResource in a mixed graph would trigger our required-predicate constraints and fail. Wave 67 stays scoped to OUR emit; Schema.org inference is still available via the Wave 65 vocabulary's rdfs:subClassOf axioms. #5 Property shapes now declare sh:node references (schema:teaches → LearningObjectiveShape, schema:hasPart → SectionShape, ed4all:hasMisconception → MisconceptionShape, ed4all:bloomDistribution → BloomDistributionShape, ed4all:targetsConcept → TargetedConceptShape). Wave 63 relied on class-based targeting alone; the parent-child shape relationship was implicit. Wave 67 makes it declarative. #6 CourseModuleShape now has an ed4all:hasMisconception property constraint (was absent in Wave 63). Ripple fixes surfaced along the way: * pyld @vocab + @container:@list/@set interaction bug: when two @context terms share the same @id and one uses @vocab + @container, the container-scoped @vocab resolution fails, producing literal @values instead of IRI references in the expanded RDF. This broke the end-to-end SHACL round trip. Wave 67 mitigation: bloomLevels and bloomVerbs (plural convenience fields for Wave 58) are suppressed from RDF projection via "bloomLevels": null / "bloomVerbs": null in the @context; the singular bloomLevel / bloomVerb carry the authoritative RDF predicate. bloomRange drops @type:@vocab and emits as string literals — SHACL uses sh:in against the string enum. The JSON wire format is unchanged; only the RDF projection of these three fields changes. * sections: @container switched from @list to @set so sh:node constraints traverse per-item (SHACL doesn't recurse rdf:List heads by default). JSON wire stays an array; order is preserved in practice by pyld compaction. RDF consumers that need strict ordering can add a schema:position field per section. Wave 62 / 64 test updates: * test_course_module_type_expands_to_schema_learning_resource → renamed to *_ed4all_course_module with the updated expansion assertion. Schema.org inference is now via vocabulary subClassOf, not direct @type alias. * test_jsonld_context_loader asserts ed4all:CourseModule in the expanded @type as proof-of-loader-served-context. New regression tests (test_courseforge_shacl_shapes.py): - External schema:LearningResource doesn't fire CourseModuleShape (over-targeting closed). - Typo bloom IRI in correct namespace fails (sh:in closed-set check). - Empty-fragment bloom IRI fails. - Non-canonical parentObjectiveId fails the pattern. - Canonical TO-01 parent ID passes. - Section without heading fails SectionShape. - Well-formed section with bloomRange passes. - bloomRange with typo value fails (string-literal sh:in). - Empty-string statement fails sh:minLength 1. Test suite: 2694 passed, 7 skipped (up from 2685; +9 new tests). Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
mdmurphy822
pushed a commit
that referenced
this pull request
Jul 1, 2026
Lay the scaffolding for the generalized remediation prompt-suffix builder per `plans/phase3_5_post_rewrite_validation.md` §A Subtask 1 (pre-resolved decision #3 + #4): module docstring, imports of `GateResult` / `GateIssue` from `MCP.hardening.validation_gates`, the 8-key `_REMEDIATION_DIRECTIVES_BY_GATE_ID` directives table (4 outline + 4 rewrite gate IDs) per the cross-tier overlap pattern the plan specifies, and stub bodies for `_append_remediation_for_gates`, `_append_preserve_remediation`, `_missing_preserve_tokens` raising `NotImplementedError`. Subtasks 2 and 3 fill in the bodies; Subtask 4 lands ≥8 tests. https://claude.ai/code/session_01Se8hZK7RFScb6R4mfe9FgX
mdmurphy822
pushed a commit
that referenced
this pull request
Jul 1, 2026
…lock_types + document two-pass env var (Phase 5-prep) Apply 6 surgical amendments to plans/phase5_independent_stages.md so it matches HEAD 4ad72e0 before execution dispatch: 1. (MAJOR) Drop `courseforge-classify` subcommand. Phase 4 wired Bloom classification inline as the `bloom_classifier_disagreement` validator gate (config/workflows.yaml:1129, 1334), not as a standalone tier. There is no `_run_classification` handler in pipeline_tools.py. Operators re-check Bloom alignment via `courseforge-validate`. Subcommand list shrinks 5 -> 4. §12 sequencing, §13 risks, §14 open questions, and `03_classification/` output directory all updated. 2. (MINOR) `--blocks` vocabulary aligned with the actual 16 singular `BLOCK_TYPES` enum at Courseforge/scripts/blocks.py:77 (objective, concept, example, assessment_item, explanation, prereq_set, activity, misconception, callout, flip_card_grid, self_check_question, summary_takeaway, reflection_prompt, discussion_prompt, chrome, recap). Plan §3 originally listed plural names (assessments, examples, etc.) that don't match the enum. Open Question #4 resolved. 3. (MINOR) Document `COURSEFORGE_TWO_PASS=true` env-var requirement. The 4 target phases (content_generation_outline, inter_tier_validation, content_generation_rewrite, post_rewrite_validation) carry `enabled_when_env: 'COURSEFORGE_TWO_PASS=true'`. Recommended approach: auto-set inside each subcommand handler with a `--no-two-pass` opt-out for operators wanting the legacy single-pass code path. 4. (MINOR) Refresh line citations against MCP/core/workflow_runner.py that drifted post-Phase 6 + 7a/b/c: - run_workflow at :802 (was :798) - _completed check at :860 (was :798) - _synthesize_dart_skip_output at :1324 (was :1171) - _synthesize_course_planning_reuse_output at :1400 (was :1247) - _dependencies_met at :1643 (was :811) 5. (MINOR) Expand the upstream-phase pre-populate list with phases that landed post original plan authoring: chunking (Phase 7b), concept_extraction (Phase 6), imscc_chunking (Phase 7c). Document the full post-Phase-7c dependency chain. 6. (MINOR) Document `02_validation_report/report.json` as a NEW writer that Phase 5 must add. The shipped `_run_inter_tier_validation` emits JSONL only (blocks_validated_path, blocks_failed_path); the operator-facing per-block aggregation report is a Phase 5 deliverable. Schema documented inline in §6. Plan-only — no production code, schema, workflow YAML, or test files touched. Plan is now ready for the recommended 2-wave execution dispatch. https://claude.ai/code/session_01Se8hZK7RFScb6R4mfe9FgX
mdmurphy822
pushed a commit
that referenced
this pull request
Jul 1, 2026
…tion note Two surgical amendments per the plan review worker's findings: 1. Lock cip29/bert-blooms-taxonomy-classifier as the chosen replacement for the deleted kabir5297/bloom_taxonomy_classifier in pre-resolved decision #4. Open Question §1 closes (RESOLVED). Documents the _CIP29_TO_BLOOM label-map translation table requirement (contents determined at SHA-pin time by reading the model's id2label config) and records the rejected alternatives (Malithi200, phoenix28). 2. Add ST 3 + ST 5 merge-coordination note to Wave B execution sequencing. Both subtasks edit MCP/core/workflow_runner.py::_LEGACY_PHASE_PARAM_ROUTING — ST 3 adds libv2_root entries to three phases; ST 5 adds Phase 6/7c.5 routing to course_planning + libv2_archival. Different keys + different phases but adjacent lines in the same dict. Sequencing ST 3 → ST 5 prevents merge conflicts. ST 5's "Depends on" line carries the same note for the executor. Plan is now execution-ready. Waves A (HIGH parallel: ST 1, 2, 6, 7), B (MEDIUM dep-chained: ST 3 → ST 5), C (LOW user-resolved + config: ST 4, 8, 9, 10) ready for dispatch. https://claude.ai/code/session_01Se8hZK7RFScb6R4mfe9FgX
mdmurphy822
pushed a commit
that referenced
this pull request
Jul 1, 2026
…T 9) Logs the operator's Phase 8 Subtask 9 decision to replace the deleted kabir5297/bloom_taxonomy_classifier ensemble member with cip29/bert-blooms-taxonomy-classifier (414 downloads, apache-2.0, BERT-based 'generated_from_trainer' provenance). Subtask 4 will consume this capture's selection to: 1. Replace the 'main' placeholder revision with a concrete HuggingFace SHA. 2. Add a _CIP29_TO_BLOOM label-map translation table mirroring _SST2_TO_BLOOM at lib/classifiers/bloom_bert_ensemble.py:100-103. Adds bert_ensemble_replacement_selection to the canonical decision_type enum in schemas/events/decision_event.schema.json (alphabetical insertion between bert_ensemble_member_loaded and block_escalation, preserving the Phase 4.5 alphabetical contract from commit 3f35ae5). Alternatives considered (rejected): Malithi200/bloom-taxonomy-classifier (no 'bert' name match), phoenix28/bloom-taxonomy-classifier (only 18 downloads — insufficient usage signal). JSONL artifact validates against schemas/events/decision_event.schema.json. Plan reference: plans/phase8_cleanup.md::Subtask 9 (pre-resolved decision #4). https://claude.ai/code/session_01Se8hZK7RFScb6R4mfe9FgX
mdmurphy822
pushed a commit
that referenced
this pull request
Jul 1, 2026
…p29 + add _CIP29_TO_BLOOM (Phase 8 ST 4) Replaces the deleted upstream kabir5297/bloom_taxonomy_classifier with cip29/bert-blooms-taxonomy-classifier (per operator decision logged 2026-05-03 — see plan ST 4 Decision #4) and pins all three ensemble members to concrete 40-hex-char HuggingFace commit SHAs. Resolved SHAs (huggingface_hub.HfApi().model_info(repo_id).sha): - cip29/bert-blooms-taxonomy-classifier: ae343e4f - distilbert-base-uncased-finetuned-sst-2-english: 714eb0fa - MoritzLaurer/DeBERTa-v3-base-mnli-fever-anli: 6f5cf0a2 The cip29 model emits generic LABEL_0 ... LABEL_5 labels (verified by fetching its config.json — id2label is the default LABEL_N form, no semantic mapping). New _CIP29_TO_BLOOM translation table (mirrors the _SST2_TO_BLOOM pattern at :100-103) maps the generic labels onto canonical Bloom levels following standard hierarchical ordering: LABEL_0 -> remember, LABEL_1 -> understand, LABEL_2 -> apply, LABEL_3 -> analyze, LABEL_4 -> evaluate, LABEL_5 -> create. The translation runs inside _classify_with_member against the model's raw argmax output before the resulting Bloom level is returned to the ensemble vote aggregator. Module docstring updated: removes the "Phase 4 followup" callout (SHAs are now concrete, not placeholder "main") and documents the Phase 8 ST 4 swap rationale plus the new 40-hex-char regex contract. Test additions (lib/classifiers/tests/test_bloom_bert_ensemble.py): - test_default_ensemble_revisions_are_concrete_commit_shas: every default member's revision matches ^[0-9a-f]{40}$ (locks the no-rollback-to-tag-style-refs contract) - test_default_ensemble_first_member_is_cip29_replacement: first member is cip29; kabir5297 string nowhere in registry - test_cip29_to_bloom_covers_all_canonical_levels: every value in _CIP29_TO_BLOOM is a canonical Bloom level + every canonical level appears in the table values (full coverage) - test_cip29_to_bloom_keys_are_label_n_form: every key matches ^LABEL_[0-5]$ + exactly 6 entries All 10 ensemble tests + 12 disagreement validator regression tests pass (22 total). https://claude.ai/code/session_01Se8hZK7RFScb6R4mfe9FgX
mdmurphy822
added a commit
that referenced
this pull request
Jul 1, 2026
…m_support_check rationale fields Per W-D11 debt sweep finding #4: the rationale-content description for ``pair_claim_support_check`` documented only the W5.D structured- attribution signals and was silent on the three W-D11 evidence_quote rates that T11.1 (commit 2b66f77) and T11.2 (commit ded9cbb) added to the block-side and pair-side validators respectively. Refreshes the description string to enumerate ``evidence_quote_coverage_rate``, ``evidence_quote_substring_fail_rate``, and ``evidence_quote_char_span_mismatch_rate`` with the canonical interpolation format, and adds a parallel signal-contract clause for the previously-undocumented ``claim_support_check`` block-side mirror. Documentation-only change inside a JSON Schema description field — no enum changes, no shape changes; ``schemas/tests/`` stays green. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
mdmurphy822
added a commit
that referenced
this pull request
Jul 1, 2026
… + B15 Resources (loop cycle 1)
Works through the first batch of the FRAMEWORK.pdf progress-review action set. No stubs
(real artifacts/contracts only); slug-free (dynamic discovery); byte-stable when the IB
flags are off; warning-day-1 gates with # TODO(calibration) deferred flips.
Calibration harness (KEYSTONE — unblocks the ~9 deferred gate-family critical-flips):
- scripts/calibration_harness.py — pure measurement (never mutates/flips). Slug-free
discovery of LibV2 rollup reports + Courseforge per-block GateResult reports + decision
JSONL; corpus-IDENTITY collapsing (the 10 timestamped runs of one textbook = 1 distinct
corpus); a 15-gate-family flip-criteria table with auditable expected-bands; emits
calibration_report.json with per-gate fire-rate + sample + flip_ready. On this box: 1
distinct corpus → every family flip_ready=false (honest). Flipping needs a 2nd
distinct-corpus run (+ ED4ALL_BLOCK_QUALITY_RUBRIC/IB flags on). 9 tests.
Keyboard + time-based-media a11y (real WCAG holes, not flag-flips) — lib/validators/rewrite_html_shape.py:
- BLOCK_KEYBOARD_OPERABLE (2.1.1): custom click-only non-native control w/o keyboard
affordance flagged; native button/a/input/details = escape hatch.
- BLOCK_FOCUS_VISIBLE (2.4.7): inline outline:none w/o replacement focus indicator flagged.
- B04 per-piece: MULTIMEDIA_{CONTROLS,CAPTIONS,AUDIO_DESC,TRANSCRIPT}_MISSING (1.2.2/1.2.4/
1.2.5) — AD check matches the actual renderer output (class="audio-description"). 19 tests.
B15 Resources (closes the last canonical-catalog gap — every B-code now has an Ed4All primary):
- resources block_type → framework_block B15 (block_catalog.yaml), IB5 default-off posture;
_render_resources_section (accessible descriptive links, never bare-URL); new
ResourceLinkPurposeValidator (2.4.4, RESOURCE_LINK_PURPOSE_UNCLEAR) wired warning-day-1 at
post_rewrite_validation in both two-pass workflows; planner nudge (further-reading shape);
JSON-LD enum + router-policy matrix + count tests 28->29; IB2 reconciliation table +
gate-count table re-derived (62/107/169). 19 tests.
Combined sweep: 723 passed. Pre-existing-unrelated: test_outline_seam_uses_block_validators
(fails on clean stash — IB3 outline-seam, not touched here).
Remaining in the action set: #4 anchored-rubric producer + #6 rollup->final_status wiring
(next iteration); #2 verb-triple flip + #6 enforcement are blocked on a 2nd-corpus FP
measurement (a GPU cycle). pipeline_tools.py decomposition stays deferred (monkeypatch
contract). .gitignore: extracted/ + calibration_report.json.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01QfgokVwS8UMGNAmWR1SohW
mdmurphy822
added a commit
that referenced
this pull request
Jul 1, 2026
…ate wiring (loop cycle 1 cont.) Continues the FRAMEWORK.pdf action set. No stubs (grounded/deterministic); byte-stable when the IB flags are off; the new gate is warning-day-1 with a # TODO(calibration) deferred flip. #4 — Anchored-rubric PRODUCER (closes the IB3.4 validator-without-producer gap): - lib/generation/anchored_rubric.py — deterministic, grounded (no LLM): authors Block.anchored_rubric for scored Evaluate/Create blocks (criteria + per-band exemplars + published_before_task), grounding criteria/exemplars in the objective's resolved Bloom verb + the assessed concept (abcd.action_object → statement phrase → key_terms); returns None (validator flags honestly) when nothing grounds — never fabricates. Gated on ED4ALL_ALIGNMENT_VERB_TRIPLE (same flag as the consumer), field hash-excluded. - Wired into MCP/tools/pipeline_tools.py: producer hook in _run_content_generation_outline (after _align_outline_blocks_to_objectives), serializer emits anchored_rubric only-when-non-None, + the 4 _entry_to_block rehydrators carry it through to post_rewrite_validation. Tests prove AnchoredRubricValidator now PASSES on a produced block. #6 — Rollup → workflow gate (FR-07/13 / §6.5 course-level): - lib/validators/block_quality_rollup.py::BlockQualityRollupValidator — scores the rewrite-tier blocks via the canonical IB6.1 rubric scorer (single owner), feeds BlockQualityRollupAggregator (single owner of the rollup math), returns a GateResult enumerating course/block hard-gate fails. Wired at post_rewrite_validation in both two-pass workflows, severity: warning, with a # TODO(calibration) flip path (→ critical + passed=course_pass once scripts/calibration_harness.py confirms FP rate on >=2 corpora). Byte-stable no-op when ED4ALL_BLOCK_QUALITY_RUBRIC is off. - Gate count re-derived: 62 critical / 109 warning / 171 total (cg 31 warn, ttc 75 warn). Hygiene: deleted the stale untracked scripts/wave76_*.py (long-pending), resolved 5 stale unmerged index flags (content==HEAD, no-op). Combined sweep: 59 passed across the touched suites. Pre-existing-unrelated: test_outline_seam_uses_block_validators (IB3 outline seam, fails on clean HEAD). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01QfgokVwS8UMGNAmWR1SohW
mdmurphy822
added a commit
that referenced
this pull request
Jul 7, 2026
… + B15 Resources (loop cycle 1)
Works through the first batch of the FRAMEWORK.pdf progress-review action set. No stubs
(real artifacts/contracts only); slug-free (dynamic discovery); byte-stable when the IB
flags are off; warning-day-1 gates with # TODO(calibration) deferred flips.
Calibration harness (KEYSTONE — unblocks the ~9 deferred gate-family critical-flips):
- scripts/calibration_harness.py — pure measurement (never mutates/flips). Slug-free
discovery of LibV2 rollup reports + Courseforge per-block GateResult reports + decision
JSONL; corpus-IDENTITY collapsing (the 10 timestamped runs of one textbook = 1 distinct
corpus); a 15-gate-family flip-criteria table with auditable expected-bands; emits
calibration_report.json with per-gate fire-rate + sample + flip_ready. On this box: 1
distinct corpus → every family flip_ready=false (honest). Flipping needs a 2nd
distinct-corpus run (+ ED4ALL_BLOCK_QUALITY_RUBRIC/IB flags on). 9 tests.
Keyboard + time-based-media a11y (real WCAG holes, not flag-flips) — lib/validators/rewrite_html_shape.py:
- BLOCK_KEYBOARD_OPERABLE (2.1.1): custom click-only non-native control w/o keyboard
affordance flagged; native button/a/input/details = escape hatch.
- BLOCK_FOCUS_VISIBLE (2.4.7): inline outline:none w/o replacement focus indicator flagged.
- B04 per-piece: MULTIMEDIA_{CONTROLS,CAPTIONS,AUDIO_DESC,TRANSCRIPT}_MISSING (1.2.2/1.2.4/
1.2.5) — AD check matches the actual renderer output (class="audio-description"). 19 tests.
B15 Resources (closes the last canonical-catalog gap — every B-code now has an Ed4All primary):
- resources block_type → framework_block B15 (block_catalog.yaml), IB5 default-off posture;
_render_resources_section (accessible descriptive links, never bare-URL); new
ResourceLinkPurposeValidator (2.4.4, RESOURCE_LINK_PURPOSE_UNCLEAR) wired warning-day-1 at
post_rewrite_validation in both two-pass workflows; planner nudge (further-reading shape);
JSON-LD enum + router-policy matrix + count tests 28->29; IB2 reconciliation table +
gate-count table re-derived (62/107/169). 19 tests.
Combined sweep: 723 passed. Pre-existing-unrelated: test_outline_seam_uses_block_validators
(fails on clean stash — IB3 outline-seam, not touched here).
Remaining in the action set: #4 anchored-rubric producer + #6 rollup->final_status wiring
(next iteration); #2 verb-triple flip + #6 enforcement are blocked on a 2nd-corpus FP
measurement (a GPU cycle). pipeline_tools.py decomposition stays deferred (monkeypatch
contract). .gitignore: extracted/ + calibration_report.json.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01QfgokVwS8UMGNAmWR1SohW
mdmurphy822
added a commit
that referenced
this pull request
Jul 7, 2026
…ate wiring (loop cycle 1 cont.) Continues the FRAMEWORK.pdf action set. No stubs (grounded/deterministic); byte-stable when the IB flags are off; the new gate is warning-day-1 with a # TODO(calibration) deferred flip. #4 — Anchored-rubric PRODUCER (closes the IB3.4 validator-without-producer gap): - lib/generation/anchored_rubric.py — deterministic, grounded (no LLM): authors Block.anchored_rubric for scored Evaluate/Create blocks (criteria + per-band exemplars + published_before_task), grounding criteria/exemplars in the objective's resolved Bloom verb + the assessed concept (abcd.action_object → statement phrase → key_terms); returns None (validator flags honestly) when nothing grounds — never fabricates. Gated on ED4ALL_ALIGNMENT_VERB_TRIPLE (same flag as the consumer), field hash-excluded. - Wired into MCP/tools/pipeline_tools.py: producer hook in _run_content_generation_outline (after _align_outline_blocks_to_objectives), serializer emits anchored_rubric only-when-non-None, + the 4 _entry_to_block rehydrators carry it through to post_rewrite_validation. Tests prove AnchoredRubricValidator now PASSES on a produced block. #6 — Rollup → workflow gate (FR-07/13 / §6.5 course-level): - lib/validators/block_quality_rollup.py::BlockQualityRollupValidator — scores the rewrite-tier blocks via the canonical IB6.1 rubric scorer (single owner), feeds BlockQualityRollupAggregator (single owner of the rollup math), returns a GateResult enumerating course/block hard-gate fails. Wired at post_rewrite_validation in both two-pass workflows, severity: warning, with a # TODO(calibration) flip path (→ critical + passed=course_pass once scripts/calibration_harness.py confirms FP rate on >=2 corpora). Byte-stable no-op when ED4ALL_BLOCK_QUALITY_RUBRIC is off. - Gate count re-derived: 62 critical / 109 warning / 171 total (cg 31 warn, ttc 75 warn). Hygiene: deleted the stale untracked scripts/wave76_*.py (long-pending), resolved 5 stale unmerged index flags (content==HEAD, no-op). Combined sweep: 59 passed across the touched suites. Pre-existing-unrelated: test_outline_seam_uses_block_validators (IB3 outline seam, fails on clean HEAD). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01QfgokVwS8UMGNAmWR1SohW
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Consolidation of nine commits (Session 1 + Workers A–H) lifting Ed4All pipeline quality from silent-drops-allowed to multi-axis surfaced & fixed. See
docs/architecture/ADR-001-pipeline-shape.md(Worker A) for the architectural decisions and cross-worker contracts that gate this cohort.Work included (in order of commits)
5e4a0385d00780390e316b69426cALLOWED_DECISION_TYPESadvisory enum0f8bb075374201e02a27bc0f00b1libv2 cross-indexCLIc078ccacde1a13Plus five merge commits consolidating the parallel worker branches.
Headline metrics (WCAG_201 regenerated end-to-end)
outcome_reverse_coverageuncovered_outcomesbloom_level_coveragesummarysource.html_xpathArchitectural changes worth callouts
METRICS_SEMANTIC_VERSIONbumped to 4 (additive:outcome_reverse_coverage, 5 flow metrics,uncovered_outcomesintegrity list)CHUNK_SCHEMA_VERSION = "v4"declared; every chunk stampsschema_version; manifest stampschunk_schema_versioninstruction_pair.schema.json,preference_pair.schema.json,concept_graph_semantic.schema.jsonalignmentkey withalignment.base_metrics_semantic_version; does not mutate base-pass keys (to be enforced in code; tracked as FOLLOWUP-ADR001-4)ALLOWED_DECISION_TYPEStuple created inlib/decision_capture.pyas advisory; enforcement deferred via FOLLOWUP-ADR001-5 pending legacy type auditOpen follow-ups tracked (all non-blocking)
FOLLOWUP-ADR001-1..5— LibV2 importer filename collision; run_summarizer dead reader; align_chunks docstring drift; additive contract enforcement in code; legacy decision_type auditFOLLOWUP-WORKER-B-1— interactive_components not threaded to chunks (regex fallback works, but proper threading is better)FOLLOWUP-WORKER-E-1—LibV2/schema/chunk.schema.jsondoesn't exist; importer copies chunks.jsonl verbatim so nothing's blockedFOLLOWUP-INTEG-1..4— capture guard in _create_chunk; pre-existing html_balance_violations signal; decision-capture validator warningsFOLLOWUP-WORKER-G-1..2— full LibV2 regeneration sweep; rebuild-hook intolibv2 index rebuildFOLLOWUP-WORKER-H-1..3— rewrite legacy week-local IDs in course_data.json; regenerate other LibV2 courses through fixed path; wire validate_page_objectives.py as a build-time gate in package_multifile_imscc.pyTest plan
venv/bin/python -m pytest Trainforge/tests LibV2/tools/libv2/tests Courseforge/scripts/tests -q— expect 295 passedpython -m Trainforge.process_course --imscc Courseforge/exports/WCAG_201_COURSE/05_final_package/WCAG_201.imscc --course-code WCAG_201 --division STEM --domain computer-science --subdomain web-accessibility --output Trainforge/output/wcag_201 --objectives Courseforge/inputs/exam-objectives/WCAG_201_objectives.json --import-to-libv2— expect outcome_reverse_coverage=1.0python -m LibV2.tools.libv2.cli cross-index --repo-root . --output LibV2/catalog/cross_package_concepts.json— expect shared concepts accessibility/assessment/cognitive-loadpython Courseforge/scripts/validate_page_objectives.py Courseforge/exports/WCAG_201_COURSE/ --objectives Courseforge/inputs/exam-objectives/WCAG_201_objectives.json— expect 0 violations🤖 Generated with Claude Code