diff --git a/README.md b/README.md index 15c3650..6beed5f 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,7 @@ Schemas for LLM-maintained markdown wikis. Each variant is a self-contained template you copy into a new wiki repo: an always-loaded `CLAUDE.md` core, on-demand `workflows/` procedures, and a stdlib-only lint tool (`lint.py` holds the variant's configuration; the `wikilint/` package holds the shared engine, byte-identical across variants) that handles every mechanical health check so the model only spends judgment where judgment is needed. -Every wiki produced from these templates is a conformant [Open Knowledge Format (OKF) v0.1](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md) bundle, so it can be consumed by any OKF-aware agent or tool without translation. +Every wiki produced from these templates is a conformant [Open Knowledge Format (OKF) v0.2](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md) bundle, so it can be consumed by any OKF-aware agent or tool without translation. ## Where this sits in the OKF ecosystem @@ -43,7 +43,7 @@ The schemas follow a few rules, informed by how agent-maintained wikis actually - **Trust is structural, not remembered.** Immutable `raw/`, append-only `log.md`, a git commit per operation, a pre-commit hook that lints the staged snapshot (exactly the bytes that will land) and makes lint errors uncommittable, and commit-pinned verification fields. Agent-maintained judgment metadata decays, so `confidence:` is reduced to the two states a lint can actually check (`low` = uncited, `contested` = sources disagree and the body must explain), inferred claims are marked inline with `(inferred)`, and tags are validated against a `taxonomy.md`, whose `## Page types` section also describes every allowed page type with a one-line meaning, so each bundle self-describes its type vocabulary to OKF consumers. - **Contested is a state to exit.** The documented failure mode of agent wikis is contradictions accumulating faster than they resolve. Lint flags contested pages older than 30 days; the reconcile workflow rewrites in place, moving losing claims to a dated "Superseded claims" section instead of deleting them. - **Autonomous but reversible.** The maintenance workflow runs unattended on a `maintenance` branch with an exhaustively-listed set of safe actions (mechanical fixes, index rebuild, unambiguous cross-links); everything else becomes a proposal. The human reviews the branch diff and merges. Nothing automated ever lands on main directly. -- **Native OKF conformance.** Every wiki is an [OKF v0.1](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md) bundle: markdown files with YAML frontmatter, ordinary markdown links as the edge form (bundle-absolute `[title](/dir/page.md)` preferred), reserved `index.md` (stamped with `okf_version` frontmatter by `rebuild-index`) and `log.md` (date-grouped headings, bold-action-word entries). `check_okf` enforces the spec's three conformance rules: parseable frontmatter on every non-reserved `.md`, a non-empty `type`, and reserved-file structure. Scriptorium's schemas are a strict superset of OKF's (its recommended `timestamp` field is our `updated`; its `resource` is our `source_path`; its `title` is optional, with the index prettifying filenames when absent). One deliberate deviation: `raw/` is excluded from conformance because prime directive 1 makes those sources immutable, and OKF has no concept of a non-concept directory. +- **Native OKF conformance.** Every wiki is an [OKF v0.2](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md) bundle: markdown files with YAML frontmatter, ordinary markdown links as the edge form (bundle-absolute `[title](/dir/page.md)`, the form v0.2 §6.1 recommends), reserved `index.md` (stamped with `okf_version` frontmatter by `rebuild-index`, the mechanism §12 specifies) and `log.md` (date-grouped headings, bold-action-word entries). `check_okf` enforces the spec's three conformance rules: parseable frontmatter on every non-reserved `.md`, a non-empty `type`, and reserved-file structure. Provenance uses the spec's `sources` shape (§5.1): mapping entries with a required `resource`, enforced by `check_sources`, with pre-0.2 string entries downgraded to warnings. Scriptorium's schemas are a strict superset of OKF's (the spec's `generated.at` is our `updated`; its `resource` is our `source_path`; its `title` is optional, with the index prettifying filenames when absent). One deliberate deviation: `raw/` is excluded from conformance because prime directive 1 makes those sources immutable, and OKF has no concept of a non-concept directory. - **Opt-in extension points for non-wiki trees.** The engine can also lint markdown trees that aren't wikis (a findings folder, a labs journal): `okf_conformance: False` turns off the OKF rules for trees that aren't bundles, `non_page_allowed` accepts glob patterns, `index_file`/`index_body_fn` relocate and reshape the generated index (`index_file: None` disables it), `extra_secret_patterns`/`secret_allow_res` extend the secrets scan, and `extra_checks` runs custom callables. Every knob's default lives once in `wikilint/settings.py` (`DEFAULTS`) and preserves the original behavior; a variant's `lint.py` lists a key only to override it. Bad values (a malformed regex, an out-of-tree `index_file`, a non-callable check) are rejected at startup with a clear message rather than a mid-run traceback. ## Unattended maintenance diff --git a/codebase-large/CLAUDE.md b/codebase-large/CLAUDE.md index f198a52..8760e8b 100644 --- a/codebase-large/CLAUDE.md +++ b/codebase-large/CLAUDE.md @@ -73,7 +73,7 @@ updated: 2026-04-10 # bumped on every edit description: One line saying what this page covers, used for scanning. subsystem: auth # or 'global' for cross-cutting pages tags: [] -sources: [] +sources: [] # entries are mappings: - resource: /dir/page.md (bundle-absolute, OKF v0.2) confidence: low | contested # optional; absent means normal --- ``` diff --git a/codebase-large/lint.py b/codebase-large/lint.py index 95a668c..1728c5f 100644 --- a/codebase-large/lint.py +++ b/codebase-large/lint.py @@ -26,7 +26,7 @@ "inbox_warn_count": 10, "inbox_warn_age_days": 14, # Hot-core size guard: warn when CLAUDE.md exceeds this many lines. - "claude_md_max_lines": 145, + "claude_md_max_lines": 200, # Every page tag must appear in this file (one `- tag — meaning` line each). "taxonomy_file": "taxonomy.md", # Contested pages untouched for this many days get flagged for reconcile. diff --git a/codebase-large/wikilint/checks.py b/codebase-large/wikilint/checks.py index cfdec49..bc9884d 100644 --- a/codebase-large/wikilint/checks.py +++ b/codebase-large/wikilint/checks.py @@ -136,7 +136,10 @@ def check_links_and_orphans(pages, report, root): if target.rel != p.rel: inbound[target.rel] += 1 for field in CONFIG["path_fields"] + CONFIG["edge_fields"]: - for value in p.field_list(field): + for value in p.path_list(field): + # External resources (URLs) are provenance, not wiki edges. + if URI_SCHEME_RE.match(value): + continue target = resolve_link(value, by_stem, by_rel) if target is None: report.error( @@ -606,8 +609,39 @@ def check_log(root, report): ) +def check_sources(pages, report): + """OKF v0.2 sources shape (SPEC §5.1): each entry is a mapping whose + required resource names either an external URL or a bundle-absolute page + path. Plain-string entries (the pre-0.2 shape) still resolve as paths in + check_links_and_orphans, but warn until installs migrate.""" + if not CONFIG["okf_conformance"]: + return + for p in pages: + for i, entry in enumerate(p.field_list("sources"), 1): + if isinstance(entry, dict): + resource = entry.get("resource", "") + if not resource: + report.error( + "sources", p.rel, + f"sources entry {i} has no resource " + "(required per entry, OKF v0.2 §5.1)", + ) + elif not URI_SCHEME_RE.match(resource) and not resource.startswith("/"): + report.warning( + "sources", p.rel, + f"sources entry {i}: bundle paths should be " + f"bundle-absolute (/dir/page.md), got {resource!r}", + ) + else: + report.warning( + "sources", p.rel, + f"sources entry {i} is a plain string; the OKF v0.2 shape " + "is '- resource: /dir/page.md'", + ) + + def check_okf(pages, report, root): - """OKF v0.1 conformance: (1) every non-reserved .md parses as frontmatter, + """OKF v0.2 conformance: (1) every non-reserved .md parses as frontmatter, (2) with a non-empty type, (3) reserved files follow their structure. Pages are skipped here: check_frontmatter is strictly stricter. log.md's structure is owned by check_log, not re-validated here. raw/ is excluded diff --git a/codebase-large/wikilint/cli.py b/codebase-large/wikilint/cli.py index 48a18bf..84383ee 100644 --- a/codebase-large/wikilint/cli.py +++ b/codebase-large/wikilint/cli.py @@ -40,6 +40,7 @@ def gather_report(root): checks.check_mermaid(pages, report) checks.check_adrs(pages, report, root) checks.check_log(root, report) + checks.check_sources(pages, report) checks.check_okf(pages, report, root) check_index_drift(pages, report, root) for extra in CONFIG["extra_checks"]: diff --git a/codebase-large/wikilint/model.py b/codebase-large/wikilint/model.py index 6fc8484..4c2cbae 100644 --- a/codebase-large/wikilint/model.py +++ b/codebase-large/wikilint/model.py @@ -10,7 +10,7 @@ GENERATED_MARKER = "" # OKF (Open Knowledge Format) version this engine produces and validates. -OKF_VERSION = "0.1" +OKF_VERSION = "0.2" # Pages allowed to break kebab-case and to have no inbound links. FILENAME_EXEMPT = ("README.md", "OWNERS.md") @@ -43,6 +43,10 @@ ADR_FILE_RE = re.compile(r"^(\d{4})-[a-z0-9-]+\.md$") YAML_FENCE_RE = re.compile(r"```yaml\n(.*?)```", re.DOTALL) KEY_RE = re.compile(r"^([A-Za-z_][A-Za-z0-9_-]*):\s*(.*)$") +# A mapping line inside a block list ("resource: /sources/foo.md"). The +# required whitespace after the colon keeps bare URLs (scheme:// has none) +# parsing as plain strings. +LIST_MAP_RE = re.compile(r"^([A-Za-z_][A-Za-z0-9_-]*):\s+(\S.*)$") class Report: @@ -100,8 +104,11 @@ def as_list(value): def parse_frontmatter(text): """Minimal YAML frontmatter parser: flat keys, inline and block lists. - Returns (fields, error). Nested mappings beyond one list level are not - supported by design; the page schemas only use flat fields. + Block-list items may be one-level mappings ("- resource: /x.md" plus + same-item continuation lines), the OKF v0.2 sources shape. Deeper + nesting is not supported by design; the page schemas stay flat. + + Returns (fields, error). """ if not text.startswith("---"): return None, "missing frontmatter" @@ -115,6 +122,7 @@ def parse_frontmatter(text): return None, "unterminated frontmatter" fields = {} current_list_key = None + current_item = None # dict being extended by continuation lines, or None for line in lines[1:end]: stripped = line.strip() if not stripped or stripped.startswith("#"): @@ -125,6 +133,7 @@ def parse_frontmatter(text): return None, f"unparseable frontmatter line: {stripped!r}" key, raw_val = m.group(1), m.group(2).strip() current_list_key = None + current_item = None if raw_val == "": fields[key] = [] current_list_key = key @@ -137,8 +146,18 @@ def parse_frontmatter(text): else: fields[key] = unquote(raw_val) elif stripped.startswith("- ") and current_list_key is not None: - fields[current_list_key].append(unquote(stripped[2:])) - # Indented non-list lines (nested mappings) are tolerated but ignored. + item_text = stripped[2:] + m = LIST_MAP_RE.match(item_text) + if m: + current_item = {m.group(1): unquote(m.group(2))} + fields[current_list_key].append(current_item) + else: + current_item = None + fields[current_list_key].append(unquote(item_text)) + elif current_item is not None and LIST_MAP_RE.match(stripped): + m = LIST_MAP_RE.match(stripped) + current_item[m.group(1)] = unquote(m.group(2)) + # Other indented non-list lines are tolerated but ignored. return fields, None @@ -234,6 +253,20 @@ def type(self): def field_list(self, name): return as_list((self.fields or {}).get(name)) + def path_list(self, name): + """Path values of a field: OKF v0.2 mapping entries contribute their + resource, plain strings pass through (the legacy shape). Mappings + without a resource contribute nothing; check_sources reports them.""" + values = [] + for entry in self.field_list(name): + if isinstance(entry, dict): + resource = entry.get("resource", "") + if resource: + values.append(resource) + else: + values.append(entry) + return values + def discover_pages(root, report): # Normalize through PurePosixPath so "./x" and "x" compare equal to the @@ -304,6 +337,10 @@ def build_link_index(pages): def resolve_link(target, by_stem, by_rel): target = target.split("|")[0].split("#")[0].strip() + # Bundle-absolute (leading /) targets resolve against the root, where + # page rel paths have no leading slash. + if target.startswith("/"): + target = target[1:] if not target: return None if target in by_rel: diff --git a/codebase-large/wikilint/settings.py b/codebase-large/wikilint/settings.py index 98b8009..52b1143 100644 --- a/codebase-large/wikilint/settings.py +++ b/codebase-large/wikilint/settings.py @@ -17,7 +17,7 @@ # these bare as CONFIG[key] is safe because configure() merges DEFAULTS under # every variant config. DEFAULTS = { - # Enforce OKF v0.1 conformance (check_okf) and stamp okf_version + # Enforce OKF v0.2 conformance (check_okf) and stamp okf_version # frontmatter into the rebuilt index. Off for non-OKF markdown trees. "okf_conformance": True, # Report pages with no inbound links (plus unlinked-mention hints). diff --git a/codebase/CLAUDE.md b/codebase/CLAUDE.md index 8e3321f..6b69fce 100644 --- a/codebase/CLAUDE.md +++ b/codebase/CLAUDE.md @@ -60,7 +60,7 @@ wiki/ ## Page types and frontmatter -Every page starts with YAML frontmatter. All pages require `type`, `created`, `updated`, `description` (one accurate sentence; it is how pages are found without being opened), and `tags` (each tag must appear in `taxonomy.md`, which also describes the allowed page types; introducing a tag means adding it there in the same commit). Optional on any page: `sources` (wiki paths backing the page) and `confidence: low | contested` (absent means normal; a `contested` page must explain the disagreement in its body, and contested is a state to exit, not a resting place: reconcile it). Mark claims you inferred rather than verified against the code with `(inferred)` inline; a page containing any carries `confidence: low`. +Every page starts with YAML frontmatter. All pages require `type`, `created`, `updated`, `description` (one accurate sentence; it is how pages are found without being opened), and `tags` (each tag must appear in `taxonomy.md`, which also describes the allowed page types; introducing a tag means adding it there in the same commit). Optional on any page: `sources` (mappings backing the page, each `- resource: /dir/page.md`, bundle-absolute; the OKF v0.2 shape) and `confidence: low | contested` (absent means normal; a `contested` page must explain the disagreement in its body, and contested is a state to exit, not a resting place: reconcile it). Mark claims you inferred rather than verified against the code with `(inferred)` inline; a page containing any carries `confidence: low`. Extra required fields by type: diff --git a/codebase/lint.py b/codebase/lint.py index ed26a13..0c23116 100644 --- a/codebase/lint.py +++ b/codebase/lint.py @@ -26,7 +26,7 @@ "inbox_warn_count": 10, "inbox_warn_age_days": 14, # Hot-core size guard: warn when CLAUDE.md exceeds this many lines. - "claude_md_max_lines": 130, + "claude_md_max_lines": 200, # Every page tag must appear in this file (one `- tag — meaning` line each). "taxonomy_file": "taxonomy.md", # Contested pages untouched for this many days get flagged for reconcile. diff --git a/codebase/wikilint/checks.py b/codebase/wikilint/checks.py index cfdec49..bc9884d 100644 --- a/codebase/wikilint/checks.py +++ b/codebase/wikilint/checks.py @@ -136,7 +136,10 @@ def check_links_and_orphans(pages, report, root): if target.rel != p.rel: inbound[target.rel] += 1 for field in CONFIG["path_fields"] + CONFIG["edge_fields"]: - for value in p.field_list(field): + for value in p.path_list(field): + # External resources (URLs) are provenance, not wiki edges. + if URI_SCHEME_RE.match(value): + continue target = resolve_link(value, by_stem, by_rel) if target is None: report.error( @@ -606,8 +609,39 @@ def check_log(root, report): ) +def check_sources(pages, report): + """OKF v0.2 sources shape (SPEC §5.1): each entry is a mapping whose + required resource names either an external URL or a bundle-absolute page + path. Plain-string entries (the pre-0.2 shape) still resolve as paths in + check_links_and_orphans, but warn until installs migrate.""" + if not CONFIG["okf_conformance"]: + return + for p in pages: + for i, entry in enumerate(p.field_list("sources"), 1): + if isinstance(entry, dict): + resource = entry.get("resource", "") + if not resource: + report.error( + "sources", p.rel, + f"sources entry {i} has no resource " + "(required per entry, OKF v0.2 §5.1)", + ) + elif not URI_SCHEME_RE.match(resource) and not resource.startswith("/"): + report.warning( + "sources", p.rel, + f"sources entry {i}: bundle paths should be " + f"bundle-absolute (/dir/page.md), got {resource!r}", + ) + else: + report.warning( + "sources", p.rel, + f"sources entry {i} is a plain string; the OKF v0.2 shape " + "is '- resource: /dir/page.md'", + ) + + def check_okf(pages, report, root): - """OKF v0.1 conformance: (1) every non-reserved .md parses as frontmatter, + """OKF v0.2 conformance: (1) every non-reserved .md parses as frontmatter, (2) with a non-empty type, (3) reserved files follow their structure. Pages are skipped here: check_frontmatter is strictly stricter. log.md's structure is owned by check_log, not re-validated here. raw/ is excluded diff --git a/codebase/wikilint/cli.py b/codebase/wikilint/cli.py index 48a18bf..84383ee 100644 --- a/codebase/wikilint/cli.py +++ b/codebase/wikilint/cli.py @@ -40,6 +40,7 @@ def gather_report(root): checks.check_mermaid(pages, report) checks.check_adrs(pages, report, root) checks.check_log(root, report) + checks.check_sources(pages, report) checks.check_okf(pages, report, root) check_index_drift(pages, report, root) for extra in CONFIG["extra_checks"]: diff --git a/codebase/wikilint/model.py b/codebase/wikilint/model.py index 6fc8484..4c2cbae 100644 --- a/codebase/wikilint/model.py +++ b/codebase/wikilint/model.py @@ -10,7 +10,7 @@ GENERATED_MARKER = "" # OKF (Open Knowledge Format) version this engine produces and validates. -OKF_VERSION = "0.1" +OKF_VERSION = "0.2" # Pages allowed to break kebab-case and to have no inbound links. FILENAME_EXEMPT = ("README.md", "OWNERS.md") @@ -43,6 +43,10 @@ ADR_FILE_RE = re.compile(r"^(\d{4})-[a-z0-9-]+\.md$") YAML_FENCE_RE = re.compile(r"```yaml\n(.*?)```", re.DOTALL) KEY_RE = re.compile(r"^([A-Za-z_][A-Za-z0-9_-]*):\s*(.*)$") +# A mapping line inside a block list ("resource: /sources/foo.md"). The +# required whitespace after the colon keeps bare URLs (scheme:// has none) +# parsing as plain strings. +LIST_MAP_RE = re.compile(r"^([A-Za-z_][A-Za-z0-9_-]*):\s+(\S.*)$") class Report: @@ -100,8 +104,11 @@ def as_list(value): def parse_frontmatter(text): """Minimal YAML frontmatter parser: flat keys, inline and block lists. - Returns (fields, error). Nested mappings beyond one list level are not - supported by design; the page schemas only use flat fields. + Block-list items may be one-level mappings ("- resource: /x.md" plus + same-item continuation lines), the OKF v0.2 sources shape. Deeper + nesting is not supported by design; the page schemas stay flat. + + Returns (fields, error). """ if not text.startswith("---"): return None, "missing frontmatter" @@ -115,6 +122,7 @@ def parse_frontmatter(text): return None, "unterminated frontmatter" fields = {} current_list_key = None + current_item = None # dict being extended by continuation lines, or None for line in lines[1:end]: stripped = line.strip() if not stripped or stripped.startswith("#"): @@ -125,6 +133,7 @@ def parse_frontmatter(text): return None, f"unparseable frontmatter line: {stripped!r}" key, raw_val = m.group(1), m.group(2).strip() current_list_key = None + current_item = None if raw_val == "": fields[key] = [] current_list_key = key @@ -137,8 +146,18 @@ def parse_frontmatter(text): else: fields[key] = unquote(raw_val) elif stripped.startswith("- ") and current_list_key is not None: - fields[current_list_key].append(unquote(stripped[2:])) - # Indented non-list lines (nested mappings) are tolerated but ignored. + item_text = stripped[2:] + m = LIST_MAP_RE.match(item_text) + if m: + current_item = {m.group(1): unquote(m.group(2))} + fields[current_list_key].append(current_item) + else: + current_item = None + fields[current_list_key].append(unquote(item_text)) + elif current_item is not None and LIST_MAP_RE.match(stripped): + m = LIST_MAP_RE.match(stripped) + current_item[m.group(1)] = unquote(m.group(2)) + # Other indented non-list lines are tolerated but ignored. return fields, None @@ -234,6 +253,20 @@ def type(self): def field_list(self, name): return as_list((self.fields or {}).get(name)) + def path_list(self, name): + """Path values of a field: OKF v0.2 mapping entries contribute their + resource, plain strings pass through (the legacy shape). Mappings + without a resource contribute nothing; check_sources reports them.""" + values = [] + for entry in self.field_list(name): + if isinstance(entry, dict): + resource = entry.get("resource", "") + if resource: + values.append(resource) + else: + values.append(entry) + return values + def discover_pages(root, report): # Normalize through PurePosixPath so "./x" and "x" compare equal to the @@ -304,6 +337,10 @@ def build_link_index(pages): def resolve_link(target, by_stem, by_rel): target = target.split("|")[0].split("#")[0].strip() + # Bundle-absolute (leading /) targets resolve against the root, where + # page rel paths have no leading slash. + if target.startswith("/"): + target = target[1:] if not target: return None if target in by_rel: diff --git a/codebase/wikilint/settings.py b/codebase/wikilint/settings.py index 98b8009..52b1143 100644 --- a/codebase/wikilint/settings.py +++ b/codebase/wikilint/settings.py @@ -17,7 +17,7 @@ # these bare as CONFIG[key] is safe because configure() merges DEFAULTS under # every variant config. DEFAULTS = { - # Enforce OKF v0.1 conformance (check_okf) and stamp okf_version + # Enforce OKF v0.2 conformance (check_okf) and stamp okf_version # frontmatter into the rebuilt index. Off for non-OKF markdown trees. "okf_conformance": True, # Report pages with no inbound links (plus unlinked-mention hints). diff --git a/docs/adr/0001-bundle-absolute-links.md b/docs/adr/0001-bundle-absolute-links.md index ad28bfc..9b7d943 100644 --- a/docs/adr/0001-bundle-absolute-links.md +++ b/docs/adr/0001-bundle-absolute-links.md @@ -12,3 +12,7 @@ All internal links in every wiki variant use bundle-absolute targets (`[proxmox- - Wiki links 404 when browsed on github.com and produce edge-less graphs in Google's unpatched visualizer. Treat external rendering as a conversion/tooling concern (the visualizer needs a ~3-line patch to resolve leading-`/` targets against the bundle root), not an authoring concern. - Still conformant OKF: the spec supports both styles. If a future OKF version drops bundle-absolute links, revisit this ADR. - The convention is load-bearing in `wikilint/derived.py` (index generation emits `[Title](/path.md)`), link resolution in `wikilint/checks.py`, each variant's `CLAUDE.md` linking rule, and tests — a reversal is a coordinated migration across all four variants plus existing wiki content. + +## Addendum (2026-07-26): OKF v0.2 adopted bundle-absolute as the recommended form + +OKF v0.2 (published 2026-07-24) resolved the upstream tension in our favor: SPEC §6.1 now recommends the bundle-absolute form ("stable when documents are moved within their subdirectory") and the upstream proposal to recommend file-relative (PR #165) did not land. The decision above is unchanged and now spec-endorsed; the visualizer rendering gap remains a tooling concern. With the v0.2 migration, frontmatter path values (`sources[].resource`, `supersedes`, edge fields) also use the bundle-absolute spelling, so links and path fields share one canonical form per target. diff --git a/docs/research/2026-07-22-okf-type-vocabulary-position.md b/docs/research/2026-07-22-okf-type-vocabulary-position.md index 4936376..175a720 100644 --- a/docs/research/2026-07-22-okf-type-vocabulary-position.md +++ b/docs/research/2026-07-22-okf-type-vocabulary-position.md @@ -28,9 +28,11 @@ Therefore the v0.2 ask should be a **place, not a vocabulary**: a reserved, conv ## Predictions (falsifiable, 2026-07-22) -- If OKF v0.2 ships a controlled type vocabulary: adoption bifurcates within roughly a year. Asset-catalog bundles conform; wiki-style bundles ignore it; "OKF-conformant" stops naming one thing. -- If v0.2 ships a self-description convention (glossary file or frontmatter block): it holds the ecosystem together by converting divergence into documented local dialects. -- If v0.2 ships nothing on types: tooling defaults (starting with the reference visualizer's palette) harden into an unwritten registry, and the gap resurfaces as interop bugs rather than as a design discussion. +**Scored 2026-07-26 — v0.2 shipped 2026-07-24.** Outcome: the third branch, with one asterisk. §4.1 keeps `type` a free string ("Type values are **not** registered centrally") and no glossary or self-description convention landed, so the "nothing on types" prediction is the live one: watch whether tooling defaults harden into an unwritten registry. The asterisk: v0.2 defines contract semantics for exactly one type value, `Attested Computation` (§10, with required `runtime`/`executor`/`attester` fields) — the first spec-blessed type, a per-type contract rather than a vocabulary. That is a fourth path this note did not predict: the spec assigning meaning to individual type values one at a time, as capabilities need it. + +- If OKF v0.2 ships a controlled type vocabulary: adoption bifurcates within roughly a year. Asset-catalog bundles conform; wiki-style bundles ignore it; "OKF-conformant" stops naming one thing. **(Did not happen.)** +- If v0.2 ships a self-description convention (glossary file or frontmatter block): it holds the ecosystem together by converting divergence into documented local dialects. **(Did not happen; Scriptorium's `## Page types` glossary remains a local dialect.)** +- If v0.2 ships nothing on types: tooling defaults (starting with the reference visualizer's palette) harden into an unwritten registry, and the gap resurfaces as interop bugs rather than as a design discussion. **(This is the branch we are on, modulo the `Attested Computation` asterisk above; still open, re-score as tooling accumulates.)** ## Possible Scriptorium moves (recorded, not committed to) diff --git a/docs/research/2026-07-26-okf-v0.2-migration.md b/docs/research/2026-07-26-okf-v0.2-migration.md new file mode 100644 index 0000000..9629fa0 --- /dev/null +++ b/docs/research/2026-07-26-okf-v0.2-migration.md @@ -0,0 +1,170 @@ +# OKF v0.2 migration assessment + +Research note, 2026-07-26. OKF bumped from v0.1 to v0.2 on 2026-07-24 +(upstream PR #227, "okf: migrate format and tooling to Open Knowledge +Format v0.2", commits `780fe9d` + `3fcbb9f`). This note diffs the new spec +against Scriptorium's implementation and lays out the migration: what is +mechanical, what is a decision, and what we deliberately skip. + +Spec text cited below is from `okf/SPEC.md` at `main` as of 2026-07-26. + +## 1. Verdict up front + +Scriptorium bundles **remain conformant**. The three conformance rules +(§11) are carried forward verbatim from v0.1: parseable frontmatter on +every non-reserved `.md`, a non-empty `type`, reserved-file structure. +v0.2 calls itself a minor bump with "two deliberate breaking changes" +(§13), neither of which breaks us. The migration is one version-string +bump, one README rewrite, one ADR addendum — plus **one real decision**: +what to do about the `sources` key (§4 below). + +## 2. What broke our way + +Three open questions from the v0.1 era resolved in Scriptorium's favor: + +1. **Root `index.md` frontmatter is now blessed.** v0.1 said index files + have no frontmatter, while our `rebuild-index` stamps + `okf_version: "0.1"` onto the root index — a documented tension. v0.2 + §8/§12 makes our behavior the spec's own mechanism: "a bundle-root + `index.md` MAY carry an `okf_version` key," described as "the only + place frontmatter is permitted in an `index.md`." +2. **Bundle-absolute links are now the *recommended* form.** §6.1: + "Absolute (bundle-relative): begins with `/` ... This is the + **recommended** form because it is stable when documents are moved." + Upstream PR #165 (swap the recommendation to file-relative) lost. + ADR 0001's bet is now spec-endorsed; the ADR gets an addendum, not a + revision. +3. **`type` remains a free string.** §4.1: "Type values are **not** + registered centrally." No controlled vocabulary shipped. Scorecard for + the predictions in + `2026-07-22-okf-type-vocabulary-position.md`: the outcome is + prediction 3's branch ("nothing on types"), with one asterisk — v0.2 + defines contract semantics for exactly one type value, + `Attested Computation` (§10), the first spec-blessed type. No glossary + convention landed, so our `## Page types` prototype remains a local + dialect, and the prediction that tooling defaults harden into an + unwritten registry is still live. + +## 3. The two breaking changes, against our usage + +- **`timestamp` → `generated: { by, at }`** (§13.1, §5.2). We never used + `timestamp`; our README maps it to our `updated` field. That mapping + line is now dead prose. `updated` itself is fine as an extension key + (§4.1 permits any producer keys), so the only change is the README + sentence — the deviations paragraph should now say `generated.at` is + the spec's analog of our `updated`. Whether to *adopt* `generated` + (with its actor convention) is deferred, see §6. +- **Body `# Citations` list → frontmatter `sources`** (§13.1, §5.1). We + never used `# Citations`; provenance has lived in our `sources` + frontmatter since the first template. Right instinct, but v0.2 defines + that exact key with a different shape — which is the decision below. + +## 4. The decision: the `sources` key + +**The conflict.** v0.2 §5.1 defines `sources` as a list of mappings: + +```yaml +sources: + - id: rev-policy # optional; footnote join key + resource: /sources/foo.md # REQUIRED per entry + title: Revenue policy # optional + # optional credibility signals: author, usage_count, last_modified +``` + +Scriptorium uses a plain string list: `sources: [sources/foo.md]`. This +is SHOULD-level, not MUST ("when the ... provenance ... families are +present, producers SHOULD follow §5"), and consumers MUST NOT reject us +either way. But once we stamp `okf_version: "0.2"`, a v0.2 consumer +reading our `sources` expects mappings and finds strings. + +**A second, subtler problem with our current spelling.** §6.2 defines the +path grammar for `sources[].resource`: an absolute URL, a `/`-prefixed +bundle path, or a relative path. Our bare `sources/foo.md` parses as a +*relative* path — which, from a page in `concepts/`, resolves to +`concepts/sources/foo.md`. Our lint resolves it leniently by stem/rel, +but under spec semantics every one of our sources references is broken. +Whatever we decide on shape, the paths should become bundle-absolute +(`/sources/foo.md`) — consistent with ADR 0001's link style anyway. + +**Options:** + +- **A. Migrate to the mapping form** — `- resource: /sources/foo.md`. + - For: "natively conformant" is the project's entire pitch; single-key + entries are one line each; the path string still appears verbatim, so + grep ergonomics survive; opens per-claim footnote attribution + (`sources[].id`, §5.1) later without another migration. + - Against: heavier YAML for agents to write correctly; every existing + page in every install (including okf-wiki's 9 pages) needs a + mechanical rewrite; `check_links_and_orphans` and the frontmatter + docs in all four CLAUDE.md files must learn the new shape. +- **B. Keep string lists, document the divergence.** Cheapest; stays + within "unknown keys MUST NOT be rejected" territory in practice. But + we would be shadowing a spec-defined key with a nonconforming shape — + the worst kind of divergence, because consumers can't tell our + `sources` from the spec's. +- **C. Rename our key** (e.g. `cites`) and leave `sources` to the spec. + Avoids the shadowing honestly, but abandons a spec-aligned field we + already hold, and still touches every page and doc — all of A's churn + for less conformance. + +**Recommendation: A**, with bundle-absolute `resource` paths, entries +kept to the single required key until a concrete need for `id`/signals +appears. Lint's `path_fields` handling extends to read `resource` out of +mapping entries (accepting both shapes during the transition, warning on +strings once installs migrate). + +**Decision (2026-07-26): Option A.** Implemented in the same change as +the version bump: the frontmatter parser accepts one-level mapping +entries in block lists, `check_sources` enforces the shape (string +entries warn, mappings without `resource` error, relative bundle paths +warn toward bundle-absolute), and external URLs are skipped by the +dangling-reference check. + +## 5. Mechanical migration checklist (independent of the §4 decision) + +All four variants, engine byte-identical as always: + +1. `wikilint/model.py`: `OKF_VERSION = "0.1"` → `"0.2"`. +2. `tests/test_checks.py`: the `"0.1"` literals in TestOkfConformance. +3. `README.md`: two "OKF v0.1" links → v0.2; rewrite the deviations + sentence (drop the `timestamp` mapping, reference `generated.at`; + note bundle-absolute is now the spec-recommended link form). +4. `docs/adr/0001-bundle-absolute-links.md`: addendum — v0.2 §6.1 + adopted bundle-absolute as recommended; decision unchanged, now + spec-endorsed. +5. `docs/research/2026-07-22-okf-type-vocabulary-position.md`: date-mark + the predictions scorecard with the §2.3 outcome. +6. Re-stamp indexes (`rebuild-index`) in each variant's example tree and + in okf-wiki after its engine sync. +7. okf-wiki: sync engine, then a fresh-context subagent ingests the v0.2 + news (its `synthesis/okf-type-semantics-gap.md` explicitly says + revisit when v0.2 lands; `concepts/okf-link-styles.md` and both + entity pages need updates). + +## 6. New optional families: adopt nothing yet + +v0.2 adds `generated`/`verified` (+ actor convention + trust tiers), +`status`, `stale_after`, credibility signals, and Attested Computations. +Per the project's scope rule, adoption needs a concrete driver, not spec +novelty. Current read: + +- **`stale_after`** — the strongest candidate. It is a plain date + comparison ("a check that can be a script"), and our lint workflow's + "stale claims" item is currently LLM-judgment-only. Worth its own + design pass; not part of this migration. +- **`verified` / trust tiers** — conceptually a fit for the unattended- + maintenance loop (merging the maintenance branch is a human + verification event), but that design session hasn't happened. Defer to + the unattended-ingest doctrine thread. +- **`status`** — overlaps our `supersedes` + banner mechanism and + `confidence`; adopting it now would give us two lifecycle vocabularies. + Skip. +- **`generated`, credibility signals, Attested Computation, `references/` + convention** — no current need. Skip. + +## Sources + +- OKF spec v0.2: https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md +- Changes from v0.1: SPEC.md §13; migration PR #227 (merged 2026-07-24) +- ADR 0001 (bundle-absolute links), `docs/adr/0001-bundle-absolute-links.md` +- Position note with predictions: `docs/research/2026-07-22-okf-type-vocabulary-position.md` diff --git a/generic/CLAUDE.md b/generic/CLAUDE.md index b2505d7..83be57a 100644 --- a/generic/CLAUDE.md +++ b/generic/CLAUDE.md @@ -54,7 +54,8 @@ created: 2026-04-10 updated: 2026-04-10 description: One line saying what this page covers, used for scanning. tags: [tag1, tag2] -sources: [sources/foo.md, sources/bar.md] +sources: + - resource: /sources/foo.md supersedes: [] confidence: low | contested # optional; absent means normal --- @@ -63,7 +64,7 @@ confidence: low | contested # optional; absent means normal Rules: - `created` is set once and never changed. `updated` changes on every edit. - `description` is one sentence. It is how you and the index find this page without opening it. Keep it accurate on every edit. -- `sources` lists every source page that supports claims on this page. +- `sources` lists every source page that supports claims on this page; each entry is a mapping whose `resource` is the bundle-absolute page path (OKF v0.2 shape). External URLs are allowed as `resource` for material with no source page. - `confidence` is absent on normal pages. `low` means claims lack citations. `contested` means two or more sources disagree, and the page body must explain the disagreement. Contested is a state to exit, not a resting place: reconcile it. - Every tag must appear in `taxonomy.md`. Introducing a tag means adding it there, with a one-line meaning, in the same commit. The allowed page types are described there too, under '## Page types'. - Mark claims you inferred rather than read with `(inferred)` inline; a page containing any carries `confidence: low`. diff --git a/generic/lint.py b/generic/lint.py index f2081a4..98d2e20 100644 --- a/generic/lint.py +++ b/generic/lint.py @@ -23,7 +23,7 @@ "inbox_warn_count": 10, "inbox_warn_age_days": 14, # Hot-core size guard: warn when CLAUDE.md exceeds this many lines. - "claude_md_max_lines": 110, + "claude_md_max_lines": 200, # Every page tag must appear in this file (one `- tag — meaning` line each). "taxonomy_file": "taxonomy.md", # Contested pages untouched for this many days get flagged for reconcile. diff --git a/generic/wikilint/checks.py b/generic/wikilint/checks.py index cfdec49..bc9884d 100644 --- a/generic/wikilint/checks.py +++ b/generic/wikilint/checks.py @@ -136,7 +136,10 @@ def check_links_and_orphans(pages, report, root): if target.rel != p.rel: inbound[target.rel] += 1 for field in CONFIG["path_fields"] + CONFIG["edge_fields"]: - for value in p.field_list(field): + for value in p.path_list(field): + # External resources (URLs) are provenance, not wiki edges. + if URI_SCHEME_RE.match(value): + continue target = resolve_link(value, by_stem, by_rel) if target is None: report.error( @@ -606,8 +609,39 @@ def check_log(root, report): ) +def check_sources(pages, report): + """OKF v0.2 sources shape (SPEC §5.1): each entry is a mapping whose + required resource names either an external URL or a bundle-absolute page + path. Plain-string entries (the pre-0.2 shape) still resolve as paths in + check_links_and_orphans, but warn until installs migrate.""" + if not CONFIG["okf_conformance"]: + return + for p in pages: + for i, entry in enumerate(p.field_list("sources"), 1): + if isinstance(entry, dict): + resource = entry.get("resource", "") + if not resource: + report.error( + "sources", p.rel, + f"sources entry {i} has no resource " + "(required per entry, OKF v0.2 §5.1)", + ) + elif not URI_SCHEME_RE.match(resource) and not resource.startswith("/"): + report.warning( + "sources", p.rel, + f"sources entry {i}: bundle paths should be " + f"bundle-absolute (/dir/page.md), got {resource!r}", + ) + else: + report.warning( + "sources", p.rel, + f"sources entry {i} is a plain string; the OKF v0.2 shape " + "is '- resource: /dir/page.md'", + ) + + def check_okf(pages, report, root): - """OKF v0.1 conformance: (1) every non-reserved .md parses as frontmatter, + """OKF v0.2 conformance: (1) every non-reserved .md parses as frontmatter, (2) with a non-empty type, (3) reserved files follow their structure. Pages are skipped here: check_frontmatter is strictly stricter. log.md's structure is owned by check_log, not re-validated here. raw/ is excluded diff --git a/generic/wikilint/cli.py b/generic/wikilint/cli.py index 48a18bf..84383ee 100644 --- a/generic/wikilint/cli.py +++ b/generic/wikilint/cli.py @@ -40,6 +40,7 @@ def gather_report(root): checks.check_mermaid(pages, report) checks.check_adrs(pages, report, root) checks.check_log(root, report) + checks.check_sources(pages, report) checks.check_okf(pages, report, root) check_index_drift(pages, report, root) for extra in CONFIG["extra_checks"]: diff --git a/generic/wikilint/model.py b/generic/wikilint/model.py index 6fc8484..4c2cbae 100644 --- a/generic/wikilint/model.py +++ b/generic/wikilint/model.py @@ -10,7 +10,7 @@ GENERATED_MARKER = "" # OKF (Open Knowledge Format) version this engine produces and validates. -OKF_VERSION = "0.1" +OKF_VERSION = "0.2" # Pages allowed to break kebab-case and to have no inbound links. FILENAME_EXEMPT = ("README.md", "OWNERS.md") @@ -43,6 +43,10 @@ ADR_FILE_RE = re.compile(r"^(\d{4})-[a-z0-9-]+\.md$") YAML_FENCE_RE = re.compile(r"```yaml\n(.*?)```", re.DOTALL) KEY_RE = re.compile(r"^([A-Za-z_][A-Za-z0-9_-]*):\s*(.*)$") +# A mapping line inside a block list ("resource: /sources/foo.md"). The +# required whitespace after the colon keeps bare URLs (scheme:// has none) +# parsing as plain strings. +LIST_MAP_RE = re.compile(r"^([A-Za-z_][A-Za-z0-9_-]*):\s+(\S.*)$") class Report: @@ -100,8 +104,11 @@ def as_list(value): def parse_frontmatter(text): """Minimal YAML frontmatter parser: flat keys, inline and block lists. - Returns (fields, error). Nested mappings beyond one list level are not - supported by design; the page schemas only use flat fields. + Block-list items may be one-level mappings ("- resource: /x.md" plus + same-item continuation lines), the OKF v0.2 sources shape. Deeper + nesting is not supported by design; the page schemas stay flat. + + Returns (fields, error). """ if not text.startswith("---"): return None, "missing frontmatter" @@ -115,6 +122,7 @@ def parse_frontmatter(text): return None, "unterminated frontmatter" fields = {} current_list_key = None + current_item = None # dict being extended by continuation lines, or None for line in lines[1:end]: stripped = line.strip() if not stripped or stripped.startswith("#"): @@ -125,6 +133,7 @@ def parse_frontmatter(text): return None, f"unparseable frontmatter line: {stripped!r}" key, raw_val = m.group(1), m.group(2).strip() current_list_key = None + current_item = None if raw_val == "": fields[key] = [] current_list_key = key @@ -137,8 +146,18 @@ def parse_frontmatter(text): else: fields[key] = unquote(raw_val) elif stripped.startswith("- ") and current_list_key is not None: - fields[current_list_key].append(unquote(stripped[2:])) - # Indented non-list lines (nested mappings) are tolerated but ignored. + item_text = stripped[2:] + m = LIST_MAP_RE.match(item_text) + if m: + current_item = {m.group(1): unquote(m.group(2))} + fields[current_list_key].append(current_item) + else: + current_item = None + fields[current_list_key].append(unquote(item_text)) + elif current_item is not None and LIST_MAP_RE.match(stripped): + m = LIST_MAP_RE.match(stripped) + current_item[m.group(1)] = unquote(m.group(2)) + # Other indented non-list lines are tolerated but ignored. return fields, None @@ -234,6 +253,20 @@ def type(self): def field_list(self, name): return as_list((self.fields or {}).get(name)) + def path_list(self, name): + """Path values of a field: OKF v0.2 mapping entries contribute their + resource, plain strings pass through (the legacy shape). Mappings + without a resource contribute nothing; check_sources reports them.""" + values = [] + for entry in self.field_list(name): + if isinstance(entry, dict): + resource = entry.get("resource", "") + if resource: + values.append(resource) + else: + values.append(entry) + return values + def discover_pages(root, report): # Normalize through PurePosixPath so "./x" and "x" compare equal to the @@ -304,6 +337,10 @@ def build_link_index(pages): def resolve_link(target, by_stem, by_rel): target = target.split("|")[0].split("#")[0].strip() + # Bundle-absolute (leading /) targets resolve against the root, where + # page rel paths have no leading slash. + if target.startswith("/"): + target = target[1:] if not target: return None if target in by_rel: diff --git a/generic/wikilint/settings.py b/generic/wikilint/settings.py index 98b8009..52b1143 100644 --- a/generic/wikilint/settings.py +++ b/generic/wikilint/settings.py @@ -17,7 +17,7 @@ # these bare as CONFIG[key] is safe because configure() merges DEFAULTS under # every variant config. DEFAULTS = { - # Enforce OKF v0.1 conformance (check_okf) and stamp okf_version + # Enforce OKF v0.2 conformance (check_okf) and stamp okf_version # frontmatter into the rebuilt index. Off for non-OKF markdown trees. "okf_conformance": True, # Report pages with no inbound links (plus unlinked-mention hints). diff --git a/homelab/CLAUDE.md b/homelab/CLAUDE.md index 767acd9..04f2be8 100644 --- a/homelab/CLAUDE.md +++ b/homelab/CLAUDE.md @@ -60,7 +60,8 @@ created: 2026-04-10 updated: 2026-04-10 description: One line saying what this page covers, used for scanning. tags: [networking, vlan] -sources: [sources/decisions/2026-04-10-vlan-split.md] +sources: + - resource: /sources/decisions/2026-04-10-vlan-split.md confidence: low | contested # optional; absent means normal --- ``` @@ -84,6 +85,7 @@ includes: [components/proxmox-01.md, components/opnsense.md] ``` Rules: +- `sources` entries are mappings whose `resource` is the bundle-absolute page path (OKF v0.2 shape); external URLs are allowed for material with no source page. - `last_verified` on components is the most important field in the schema. Older than 60 days gets flagged by lint. - `depends_on` is the only stored dependency edge. Reverse edges (what consumes a component) are derived, never stored: `python3 lint.py reverse-deps` prints the map. - `confidence` is absent on normal pages. `low` means claims lack citations. `contested` means sources or the human disagree, and the page body must explain the disagreement. Contested is a state to exit, not a resting place: reconcile it. diff --git a/homelab/lint.py b/homelab/lint.py index 0b11c99..6321e06 100644 --- a/homelab/lint.py +++ b/homelab/lint.py @@ -26,7 +26,7 @@ "inbox_warn_count": 10, "inbox_warn_age_days": 14, # Hot-core size guard: warn when CLAUDE.md exceeds this many lines. - "claude_md_max_lines": 135, + "claude_md_max_lines": 200, # Every page tag must appear in this file (one `- tag — meaning` line each). "taxonomy_file": "taxonomy.md", # Contested pages untouched for this many days get flagged for reconcile. diff --git a/homelab/wikilint/checks.py b/homelab/wikilint/checks.py index cfdec49..bc9884d 100644 --- a/homelab/wikilint/checks.py +++ b/homelab/wikilint/checks.py @@ -136,7 +136,10 @@ def check_links_and_orphans(pages, report, root): if target.rel != p.rel: inbound[target.rel] += 1 for field in CONFIG["path_fields"] + CONFIG["edge_fields"]: - for value in p.field_list(field): + for value in p.path_list(field): + # External resources (URLs) are provenance, not wiki edges. + if URI_SCHEME_RE.match(value): + continue target = resolve_link(value, by_stem, by_rel) if target is None: report.error( @@ -606,8 +609,39 @@ def check_log(root, report): ) +def check_sources(pages, report): + """OKF v0.2 sources shape (SPEC §5.1): each entry is a mapping whose + required resource names either an external URL or a bundle-absolute page + path. Plain-string entries (the pre-0.2 shape) still resolve as paths in + check_links_and_orphans, but warn until installs migrate.""" + if not CONFIG["okf_conformance"]: + return + for p in pages: + for i, entry in enumerate(p.field_list("sources"), 1): + if isinstance(entry, dict): + resource = entry.get("resource", "") + if not resource: + report.error( + "sources", p.rel, + f"sources entry {i} has no resource " + "(required per entry, OKF v0.2 §5.1)", + ) + elif not URI_SCHEME_RE.match(resource) and not resource.startswith("/"): + report.warning( + "sources", p.rel, + f"sources entry {i}: bundle paths should be " + f"bundle-absolute (/dir/page.md), got {resource!r}", + ) + else: + report.warning( + "sources", p.rel, + f"sources entry {i} is a plain string; the OKF v0.2 shape " + "is '- resource: /dir/page.md'", + ) + + def check_okf(pages, report, root): - """OKF v0.1 conformance: (1) every non-reserved .md parses as frontmatter, + """OKF v0.2 conformance: (1) every non-reserved .md parses as frontmatter, (2) with a non-empty type, (3) reserved files follow their structure. Pages are skipped here: check_frontmatter is strictly stricter. log.md's structure is owned by check_log, not re-validated here. raw/ is excluded diff --git a/homelab/wikilint/cli.py b/homelab/wikilint/cli.py index 48a18bf..84383ee 100644 --- a/homelab/wikilint/cli.py +++ b/homelab/wikilint/cli.py @@ -40,6 +40,7 @@ def gather_report(root): checks.check_mermaid(pages, report) checks.check_adrs(pages, report, root) checks.check_log(root, report) + checks.check_sources(pages, report) checks.check_okf(pages, report, root) check_index_drift(pages, report, root) for extra in CONFIG["extra_checks"]: diff --git a/homelab/wikilint/model.py b/homelab/wikilint/model.py index 6fc8484..4c2cbae 100644 --- a/homelab/wikilint/model.py +++ b/homelab/wikilint/model.py @@ -10,7 +10,7 @@ GENERATED_MARKER = "" # OKF (Open Knowledge Format) version this engine produces and validates. -OKF_VERSION = "0.1" +OKF_VERSION = "0.2" # Pages allowed to break kebab-case and to have no inbound links. FILENAME_EXEMPT = ("README.md", "OWNERS.md") @@ -43,6 +43,10 @@ ADR_FILE_RE = re.compile(r"^(\d{4})-[a-z0-9-]+\.md$") YAML_FENCE_RE = re.compile(r"```yaml\n(.*?)```", re.DOTALL) KEY_RE = re.compile(r"^([A-Za-z_][A-Za-z0-9_-]*):\s*(.*)$") +# A mapping line inside a block list ("resource: /sources/foo.md"). The +# required whitespace after the colon keeps bare URLs (scheme:// has none) +# parsing as plain strings. +LIST_MAP_RE = re.compile(r"^([A-Za-z_][A-Za-z0-9_-]*):\s+(\S.*)$") class Report: @@ -100,8 +104,11 @@ def as_list(value): def parse_frontmatter(text): """Minimal YAML frontmatter parser: flat keys, inline and block lists. - Returns (fields, error). Nested mappings beyond one list level are not - supported by design; the page schemas only use flat fields. + Block-list items may be one-level mappings ("- resource: /x.md" plus + same-item continuation lines), the OKF v0.2 sources shape. Deeper + nesting is not supported by design; the page schemas stay flat. + + Returns (fields, error). """ if not text.startswith("---"): return None, "missing frontmatter" @@ -115,6 +122,7 @@ def parse_frontmatter(text): return None, "unterminated frontmatter" fields = {} current_list_key = None + current_item = None # dict being extended by continuation lines, or None for line in lines[1:end]: stripped = line.strip() if not stripped or stripped.startswith("#"): @@ -125,6 +133,7 @@ def parse_frontmatter(text): return None, f"unparseable frontmatter line: {stripped!r}" key, raw_val = m.group(1), m.group(2).strip() current_list_key = None + current_item = None if raw_val == "": fields[key] = [] current_list_key = key @@ -137,8 +146,18 @@ def parse_frontmatter(text): else: fields[key] = unquote(raw_val) elif stripped.startswith("- ") and current_list_key is not None: - fields[current_list_key].append(unquote(stripped[2:])) - # Indented non-list lines (nested mappings) are tolerated but ignored. + item_text = stripped[2:] + m = LIST_MAP_RE.match(item_text) + if m: + current_item = {m.group(1): unquote(m.group(2))} + fields[current_list_key].append(current_item) + else: + current_item = None + fields[current_list_key].append(unquote(item_text)) + elif current_item is not None and LIST_MAP_RE.match(stripped): + m = LIST_MAP_RE.match(stripped) + current_item[m.group(1)] = unquote(m.group(2)) + # Other indented non-list lines are tolerated but ignored. return fields, None @@ -234,6 +253,20 @@ def type(self): def field_list(self, name): return as_list((self.fields or {}).get(name)) + def path_list(self, name): + """Path values of a field: OKF v0.2 mapping entries contribute their + resource, plain strings pass through (the legacy shape). Mappings + without a resource contribute nothing; check_sources reports them.""" + values = [] + for entry in self.field_list(name): + if isinstance(entry, dict): + resource = entry.get("resource", "") + if resource: + values.append(resource) + else: + values.append(entry) + return values + def discover_pages(root, report): # Normalize through PurePosixPath so "./x" and "x" compare equal to the @@ -304,6 +337,10 @@ def build_link_index(pages): def resolve_link(target, by_stem, by_rel): target = target.split("|")[0].split("#")[0].strip() + # Bundle-absolute (leading /) targets resolve against the root, where + # page rel paths have no leading slash. + if target.startswith("/"): + target = target[1:] if not target: return None if target in by_rel: diff --git a/homelab/wikilint/settings.py b/homelab/wikilint/settings.py index 98b8009..52b1143 100644 --- a/homelab/wikilint/settings.py +++ b/homelab/wikilint/settings.py @@ -17,7 +17,7 @@ # these bare as CONFIG[key] is safe because configure() merges DEFAULTS under # every variant config. DEFAULTS = { - # Enforce OKF v0.1 conformance (check_okf) and stamp okf_version + # Enforce OKF v0.2 conformance (check_okf) and stamp okf_version # frontmatter into the rebuilt index. Off for non-OKF markdown trees. "okf_conformance": True, # Report pages with no inbound links (plus unlinked-mention hints). diff --git a/tests/test_checks.py b/tests/test_checks.py index 596ed81..0a78ae0 100644 --- a/tests/test_checks.py +++ b/tests/test_checks.py @@ -293,7 +293,7 @@ def test_rebuild_clears_drift(self): def test_hot_core_cap(self): root = make_wiki(self.tmp) - (root / "CLAUDE.md").write_text("x\n" * 200) + (root / "CLAUDE.md").write_text("x\n" * 201) report = gather(root) self.assertEqual(len(findings(report, "hot-core", "WARNING")), 1) @@ -833,7 +833,7 @@ def test_index_requires_okf_version(self): rebuild_index(root) self.assertEqual(findings(gather(root), "okf", "ERROR"), []) text = (root / "index.md").read_text() - self.assertTrue(text.startswith('---\nokf_version: "0.1"\n---\n'), text[:60]) + self.assertTrue(text.startswith('---\nokf_version: "0.2"\n---\n'), text[:60]) self.assertIn("* [A Note](/concepts/a-note.md) - A.", text) def test_rebuild_index_does_not_accumulate_frontmatter(self): @@ -853,5 +853,102 @@ def test_okf_conformance_gate_off(self): self.assertEqual(findings(gather(root), "okf"), []) +class TestSourcesShape(WikiTest): + """OKF v0.2 §5.1: sources entries are mappings with a required resource; + bundle paths use the bundle-absolute form. The pre-0.2 plain-string shape + still resolves but warns.""" + + SRC = "---\ntype: source\ncreated: 2026-07-01\nupdated: 2026-07-01\n" \ + "description: A source.\ntags: [alpha]\nsources: []\n" \ + "source_path: raw/articles/foo.md\n---\n\nBody.\n" + + def _wiki(self, citing_fm): + return make_wiki(self.tmp, files={ + "sources/foo-article.md": self.SRC, + "concepts/cites.md": ( + "---\ntype: concept\ncreated: 2026-07-01\nupdated: 2026-07-01\n" + f"description: Cites foo.\ntags: [alpha]\n{citing_fm}---\n\n" + "Body cites [foo](/sources/foo-article.md).\n" + ), + }) + + def test_mapping_entry_resolves_and_passes(self): + root = self._wiki("sources:\n - resource: /sources/foo-article.md\n") + report = gather(root) + self.assertEqual(findings(report, "sources"), []) + self.assertEqual(findings(report, "reference", "ERROR"), []) + + def test_multi_key_mapping_entry_parses(self): + from wikilint.model import parse_frontmatter + fields, err = parse_frontmatter( + "---\ntype: concept\nsources:\n" + " - resource: /sources/foo.md\n" + " id: foo\n" + " title: Foo article\n---\n" + ) + self.assertIsNone(err) + self.assertEqual(fields["sources"], [{ + "resource": "/sources/foo.md", "id": "foo", "title": "Foo article", + }]) + + def test_url_list_entry_stays_a_string(self): + # A bare URL contains a colon but is not a mapping entry. + from wikilint.model import parse_frontmatter + fields, err = parse_frontmatter( + "---\ntype: concept\nsources:\n - https://example.com/schema\n---\n" + ) + self.assertIsNone(err) + self.assertEqual(fields["sources"], ["https://example.com/schema"]) + + def test_string_entry_warns_but_still_resolves(self): + root = self._wiki("sources: [sources/foo-article.md]\n") + report = gather(root) + warnings = findings(report, "sources", "WARNING") + self.assertEqual(len(warnings), 1) + self.assertIn("plain string", warnings[0][3]) + self.assertEqual(findings(report, "reference", "ERROR"), []) + + def test_mapping_without_resource_errors(self): + root = self._wiki("sources:\n - id: foo\n") + report = gather(root) + errors = findings(report, "sources", "ERROR") + self.assertEqual(len(errors), 1) + self.assertIn("resource", errors[0][3]) + + def test_relative_resource_warns_bundle_absolute(self): + root = self._wiki("sources:\n - resource: sources/foo-article.md\n") + report = gather(root) + warnings = findings(report, "sources", "WARNING") + self.assertEqual(len(warnings), 1) + self.assertIn("bundle-absolute", warnings[0][3]) + + def test_url_resource_is_skipped_by_reference_check(self): + root = self._wiki("sources:\n - resource: https://example.com/schema\n") + report = gather(root) + self.assertEqual(findings(report, "sources"), []) + self.assertEqual(findings(report, "reference", "ERROR"), []) + + def test_bundle_absolute_path_field_resolves(self): + # Leading-slash targets in any path field resolve against the root. + root = self._wiki( + "sources:\n - resource: /sources/foo-article.md\n" + "supersedes: [/sources/foo-article.md]\n" + ) + report = gather(root) + self.assertEqual(findings(report, "reference", "ERROR"), []) + + def test_dangling_mapping_resource_still_reported(self): + root = self._wiki("sources:\n - resource: /sources/never-written.md\n") + report = gather(root) + errors = findings(report, "reference", "ERROR") + self.assertEqual(len(errors), 1) + self.assertIn("nonexistent", errors[0][3]) + + def test_gate_off_disables_shape_check(self): + root = self._wiki("sources: [sources/foo-article.md]\n") + use_variant_with("generic", okf_conformance=False) + self.assertEqual(findings(gather(root), "sources"), []) + + if __name__ == "__main__": unittest.main()