Adopt real-schema idiom + generate HTML schema reference with xs3p

.github/workflows/ci.yml

@@ -27,8 +27,9 @@ jobs:
 
       - name: Drift gate (generated models + schema docs match the schema)
         # Regenerate on the same Python 3.9 target and fail if the committed artifacts differ.
-        # xsdata is pinned (==25.7, the last 3.9-supporting line) so generation is byte-stable.
-        # See docs/decisions/0008.
+        # Models: xsdata is pinned (==25.7, the last 3.9-supporting line) so generation is
+        # byte-stable. Schema reference: HTML produced from the XSD by the vendored xs3p
+        # stylesheet (deterministic). See docs/decisions/0008 and 0010.
         run: |
           make generate
           make gen-schema-docs

.gitignore

@@ -222,3 +222,6 @@ site/
 
 # Local verification venvs
 .venv-check/
+
+# Private real material — real XSD/corpus/models. NEVER committed (proprietary).
+private/

Makefile

@@ -31,7 +31,7 @@ docs-serve: ## Live-preview the HTML docs at http://localhost:8000
 generate: ## Regenerate typed models from schema/*.xsd via xsdata.
 	python -m acoustic_dataset.cli generate
 
-gen-schema-docs: ## Generate schema reference pages + Mermaid ERD from the enriched XSD.
+gen-schema-docs: ## Generate the HTML schema reference from the XSD (via vendored xs3p).
 	python -m acoustic_dataset.cli gen-schema-docs
 
 pipeline: ## End-to-end: map example input -> objects -> XML -> validate -> round-trip.

docs/concepts/pipeline-data-flow.md

@@ -27,11 +27,11 @@ no pickle) is needed to get testability; that whole chain was removed (ADR 0002)
 
 ## The entities as an ER diagram
 
-This is the kind of **Mermaid ERD** the docs render — here drawn by hand for the data-flow story
-(it deliberately includes pipeline entities like the golden and reference files). The
-**[schema reference](../reference/schema/index.md)** ERD, by contrast, is produced
-**automatically from the schema** by `make gen-schema-docs`
-(ADR 0009).
+This **Mermaid ERD** is drawn by hand for the data-flow story (it deliberately includes pipeline
+entities like the golden and reference files). The
+**[schema reference](../reference/schema/index.html)**, by contrast, is produced
+**automatically from the schema** (as HTML) by `make gen-schema-docs`
+(ADR 0011).
 
 ```mermaid
 erDiagram

docs/concepts/schema-as-contract.md

@@ -34,8 +34,7 @@ flowchart TD
     XSD["Enriched XSD<br/>(schema/acoustic_dataset.xsd)"]
     XSD --> Models["Typed data classes<br/>(xsdata)"]
     XSD --> Validate["Validation gate<br/>(xmlschema)"]
-    XSD --> Docs["HTML schema reference<br/>(MkDocs Material)"]
-    XSD --> ERD["Mermaid ERD"]
+    XSD --> Docs["HTML schema reference<br/>(xs3p)"]
     XSD --> Bindings["Other-language bindings<br/>(Java, JSON Schema — later)"]
     Models --> XML["Emitted XML Acoustic Dataset"]
     Validate --> XML

docs/decisions/0009-mkdocs-material-mermaid-html-docs.md

@@ -1,9 +1,14 @@
 # ADR 0009: MkDocs Material + Mermaid for HTML docs and ERDs generated from the XSD
 
-- **Status**: Accepted
+- **Status**: Superseded (in part) by [ADR 0011](0011-xs3p-html-schema-reference.md)
 - **Date**: 2026-06-13
 - **Deciders**: Project team
 
+> **Note (superseded in part):** the decision to *generate the schema reference + Mermaid ERD as
+> Markdown by walking the XSD ourselves* is superseded by [ADR 0011](0011-xs3p-html-schema-reference.md),
+> which generates the reference as HTML with the off-the-shelf xs3p stylesheet (no generated ERD).
+> The choice of MkDocs Material + Mermaid for the rest of the site still stands.
+
 ## Context
 
 Two documentation needs exist:

docs/decisions/0011-xs3p-html-schema-reference.md

@@ -0,0 +1,44 @@
+# ADR 0011: Generate the schema reference as HTML with xs3p
+
+**Status:** Accepted (supersedes the schema-reference/ERD portion of
+[ADR 0009](0009-mkdocs-material-mermaid-html-docs.md))
+
+## Context
+
+[ADR 0009](0009-mkdocs-material-mermaid-html-docs.md) chose to generate the schema reference as
+Markdown — entity tables plus a hand-shaped Mermaid ERD — by walking the XSD in our own code
+(`schema_docs.py`). When that code was first pointed at a **real** XSD (no-namespace, XSD 1.1,
+"salami-slice": every element global, complex types built from `xs:all` and `xs:element ref="..."`,
+documentation on the global elements / ref sites), it produced almost nothing: the hand-rolled
+walker assumed `xs:sequence` + inline-named elements + `xs:extension` inheritance and silently
+skipped everything else.
+
+That is a direct violation of the project's **"configure, don't create"** principle: we were
+hand-implementing XSD semantics (ref resolution, model groups, type resolution, XSD 1.1) — the
+exact thing a real schema library or tool does correctly. Re-implementing it is both effort and a
+standing source of bugs.
+
+## Decision
+
+Generate the schema reference as **HTML** using the off-the-shelf **xs3p** XSLT stylesheet,
+driven through `lxml`'s XSLT engine (`make gen-schema-docs` → `schema_html.py`):
+
+- xs3p (`tools/xs3p/xs3p.xsl`, vendored unmodified) renders any XSD — including the real idiom —
+  into a single self-contained HTML reference. We own no XSD-parsing code.
+- Output is **byte-deterministic** (no embedded timestamp/path), so the committed
+  `docs/reference/schema/index.html` is **drift-checked** in CI exactly as the generated models are.
+- The hand-rolled `schema_docs.py` and its Mermaid ERD are removed. The remaining Mermaid diagrams
+  in the *concept* pages are hand-drawn illustrations and stay.
+
+xs3p is licensed under the **DSTC Public License (DPL) 1.1** (an MPL-1.1-derived weak copyleft);
+its `LICENSE.html` is vendored alongside the stylesheet and the file is kept unmodified
+(`tools/xs3p/README.md`).
+
+## Consequences
+
+- The reference is robust to real-world XSD idioms by construction — no parser to maintain.
+- We trade the MkDocs-integrated Markdown + Mermaid ERD for a standalone HTML page linked from the
+  site. No entity-relationship *diagram* of the contract is generated anymore.
+- A new, file-level-copyleft licence (DPL 1.1) enters the tree, scoped to `tools/xs3p/`.
+- ADR 0009's choice of MkDocs Material + Mermaid for the rest of the site still stands; only its
+  "schema reference + ERD generated as Markdown" portion is superseded here.

docs/decisions/index.md

@@ -31,7 +31,9 @@ To add one, follow [Add a decision record](../how-to/add-a-decision-record.md) u
 | [0006](0006-codespaces-with-local-fallback.md) | GitHub Codespaces as primary env, with a portable local fallback | Accepted |
 | [0007](0007-pin-python-3-9-4.md) | Pin to exactly Python 3.9.4 to match the target system | Accepted |
 | [0008](0008-generated-models-no-drift.md) | Treat bindings as generated artifacts; fail CI on drift | Accepted |
-| [0009](0009-mkdocs-material-mermaid-html-docs.md) | MkDocs Material + Mermaid for HTML docs and ERDs from the XSD | Accepted |
+| [0009](0009-mkdocs-material-mermaid-html-docs.md) | MkDocs Material + Mermaid for HTML docs and ERDs from the XSD | Superseded (in part) |
+| [0010](0010-build-schema-object-directly.md) | Build the schema's data object directly (no intermediate hierarchy) | Accepted |
+| [0011](0011-xs3p-html-schema-reference.md) | Generate the schema reference as HTML with xs3p | Accepted |
 
 !!! note
     Most decisions are **provisional in spirit** — the delivery plan records that the design

docs/glossary.md

@@ -44,9 +44,9 @@ catch *schema-valid-but-different* output during migration.
 validation yet differs from what a consumer depended on. Caught by the migration-safety
 comparison, not by validation alone.
 
-**ERD (Entity-Relationship Diagram)** — a diagram of entities and how they relate. We
-render these with **Mermaid** and generate them from the enriched XSD so they cannot drift
-from the contract. See the [generated schema reference](reference/schema/index.md).
+**ERD (Entity-Relationship Diagram)** — a diagram of entities and how they relate. We draw
+these by hand with **Mermaid** in the concept pages. The entity/type structure of the contract
+itself is in the [generated schema reference](reference/schema/index.html) (HTML, from the XSD).
 
 ## Tooling
 

docs/how-to/build-the-docs-site.md

@@ -44,15 +44,15 @@ erDiagram
 ```
 ````
 
-See the [pipeline ERD](../concepts/pipeline-data-flow.md) and the
-[generated schema ERD](../reference/schema/index.md) for working examples.
+See the [pipeline ERD](../concepts/pipeline-data-flow.md) for a hand-drawn example; the
+[generated schema reference](../reference/schema/index.html) is HTML produced from the XSD.
 
-## Schema reference & ERD are *generated*
+## The schema reference is *generated*
 
-The [schema reference](../reference/schema/index.md) — the entity tables and the Mermaid ERD —
-is produced from the **enriched XSD** by `make gen-schema-docs`, not hand-written, so it can't
-drift from the contract. Regenerate it after any schema change; the CI drift gate fails if the
-committed page is stale.
+The [schema reference](../reference/schema/index.html) — every element, type, facet and
+`xs:documentation` note — is produced from the XSD by the vendored xs3p stylesheet
+(`make gen-schema-docs`), not hand-written, so it can't drift from the contract. Regenerate it
+after any schema change; the CI drift gate fails if the committed page is stale.
 
 ## Linking rules (to keep `--strict` happy)
 

docs/how-to/change-the-schema.md

@@ -21,12 +21,12 @@ redesign*: models, validation, bindings and the schema docs all regenerate from
    (ADR 0008). Generation is pinned to the 3.9
    toolchain so the output is byte-reproducible for the drift gate.
 
-3. **Regenerate the schema docs + ERD.**
+3. **Regenerate the schema reference.**
    ```bash
    make gen-schema-docs
    ```
-   Produces the reference pages and the Mermaid ERD from the schema
-   (ADR 0009).
+   Produces the HTML reference from the schema via xs3p
+   (ADR 0011).
 
 4. **Update the builder.**
    `src/acoustic_dataset/build.py` is the **one place** that knows element names — update it
@@ -64,5 +64,5 @@ output that is schema-valid but differs from what a consumer depends on
 ## What you should *not* touch
 
 - Generated models (`src/acoustic_dataset/models/`) — regenerate instead.
-- Generated schema reference/ERD under `reference/schema/` — regenerate instead.
+- Generated HTML schema reference under `reference/schema/` — regenerate instead.
 - The generation code — it's schema-agnostic by design.

docs/how-to/run-the-migration-safety-comparison.md

@@ -47,11 +47,11 @@ The comparison canonicalises both sides first, so these **never** cause a failur
 |---|---|
 | Attribute order | `a="1" b="2"` vs `b="2" a="1"` |
 | Whitespace / indentation | pretty-printed vs single-line |
-| Namespace **prefix** | default `xmlns=…` vs `ds:` prefix (same URI) |
+| Namespace **prefix** | the same URI under a prefix vs the default namespace (the contract itself is no-namespace) |
 | Comments | a `<!-- trial file -->` banner |
 
 Anything that changes the **data** — an element value, a missing or extra element, a different
-structure — is reported as a difference. For instance, a radiated-noise `Level` of `134.000`
+structure — is reported as a difference. For instance, a radiated-noise `SectorLevel` of `134.000`
 that becomes `144.000` is still inside the schema's decibel band (so it stays schema-valid) but
 is surfaced as a one-line diff.
 

docs/index.md

@@ -28,8 +28,8 @@ different question — plus a set of **Architecture Decision Records (ADRs)** th
 ## The one-paragraph mental model
 
 The **schema (XSD) is the contract.** Everything derives from it: typed data classes
-(generated by `xsdata`), the validation gate (`xmlschema`), the HTML reference docs and
-Mermaid ERD, and the language bindings shipped to consumers. We never hand-write the
+(generated by `xsdata`), the validation gate (`xmlschema`), the HTML reference docs
+(generated by xs3p), and the language bindings shipped to consumers. We never hand-write the
 things the schema can generate — *configure, don't create*. When you write Python here you
 start from a structured set of parameters and the data stays in **typed objects end to end**,
 from that structure through to the XML — see
@@ -44,8 +44,8 @@ human semantic gate (is the science right?). See
 - Every significant choice is recorded as a numbered **ADR**, kept in the
   repository under `docs/decisions/` (not published to this site).
 - New understanding lands as a **concept** page; new recipes as **how-to** guides.
-- The **schema reference and ERD** are *generated from the enriched XSD* — see the
-  [generated schema reference](reference/schema/index.md) — so the docs can never drift from the contract.
+- The **schema reference** is *generated from the XSD* (by xs3p) — see the
+  [generated schema reference](reference/schema/index.html) — so the docs can never drift from the contract.
 
 !!! note "Status"
     The environment (Codespace, tests, docs site) and the full schema-driven pipeline are real

docs/onboarding.md

@@ -31,7 +31,7 @@ before the same `make verify` / `make pipeline`. Both paths reach the *same* gre
 | Example calculation input | `examples/calculation_input.json` |
 | Tests (unit / integration / golden) | `tests/` |
 | The plan & design artifacts | `specs/001-codespace-xml-scaffold/` (`spec.md`, `plan.md`, `tasks.md`) |
-| The generated schema reference + ERD | [reference/schema](reference/schema/index.md) (run `make gen-schema-docs`) |
+| The generated HTML schema reference | [reference/schema](reference/schema/index.html) (run `make gen-schema-docs`) |
 | Why each choice was made | `docs/decisions/` (ADRs, kept in the repo) |
 
 ## Build the mental model
@@ -45,5 +45,5 @@ Read these, in order — they're short:
 
 !!! tip "What's done, what's next"
     The full Phase 1 pipeline is in place — environment, end-to-end pipeline, migration-safety
-    `compare`, the generated schema reference/ERD, and the distribution `bundle` with a CI drift
+    `compare`, the generated HTML schema reference, and the distribution `bundle` with a CI drift
     gate. Remaining polish is tracked in `specs/001-codespace-xml-scaffold/tasks.md`.

docs/reference/commands.md

@@ -13,7 +13,7 @@
 | `make docs` | Build static HTML site (`mkdocs build --strict`) → `site/` | ✅ |
 | `make docs-serve` | Live-preview docs at <http://localhost:8000> | ✅ |
 | `make generate` | Regenerate models from `schema/*.xsd` via `xsdata` | ✅ |
-| `make gen-schema-docs` | Generate schema reference + Mermaid ERD from the enriched XSD | ✅ |
+| `make gen-schema-docs` | Generate the HTML schema reference from the XSD (via xs3p) | ✅ |
 | `make pipeline` | End-to-end: map → serialise → validate → round-trip | ✅ |
 | `make compare` | Migration-safety diff vs a known-good reference | ✅ |
 | `make bundle` | Distribution bundle (data + schema + models) | ✅ |

docs/reference/index.md

@@ -5,7 +5,7 @@
 | Page | What it gives you |
 |---|---|
 | [Commands](commands.md) | Every `make` target and CLI command, with exit-code meaning |
-| [Schema reference](schema/index.md) | Generated entity tables + Mermaid ERD, produced from the enriched XSD |
+| [Schema reference](schema/index.html) | Generated HTML reference (entities, types, facets, `xs:documentation`), produced from the XSD by xs3p |
 
 ## Authoritative specifications
 
@@ -19,7 +19,7 @@ the docs site, under version control):
 
 ## Generated reference
 
-`make gen-schema-docs` (re)generates the [schema reference](schema/index.md) from the enriched
-XSD — the entity tables, every field's `xs:documentation` prose, and the Mermaid ERD — produced
-*from the schema* so they cannot drift (ADR 0009).
+`make gen-schema-docs` (re)generates the [schema reference](schema/index.html) from the XSD via
+the vendored xs3p stylesheet — every element, type, facet and `xs:documentation` note — produced
+*from the schema* so it cannot drift (ADR 0011).
 The committed page is regenerated by CI; a stale copy fails the drift gate.