claude: documented-recipe verifier (step 1 of 2)

[TomHennen]

claude: Draft — step 1 of 2. Builds the wrangle-side core of a "documented-recipe verifier". Step 2 (wiring it into wrangle-test's showcase) is the owner's, after review.

What this does

docs/verifying_artifacts.md documents the exact raw ampel/cosign/gh attestation commands an adopter runs to verify wrangle-built artifacts. Nothing executes those recipes against live artifacts, so they can silently drift (today only the manual cut-release gate catches it). This adds a runner a showcase can call every heartbeat to prove the documented commands still verify.

Anti-drift approach: extract-and-run (the brief's preferred option)

tools/verify_documented_recipes.sh does not reimplement the recipes. At run time it extracts each fenced ```bash block from the doc, substitutes the <placeholders> from flags, and executes the block verbatim. The checker therefore cannot diverge from the docs — drift is impossible, not merely detected.

I prototyped extraction first (as the brief asked). Of the three brittleness triggers, only "multiple commands per block" is hit (the two cosign blocks), and that's benign — the whole block is run. Prose sits between blocks, not inside; the one prose-y placeholder (<resourceUri from the table above>) is a single <[^>]+> match. So extract-and-run held; I did not fall back to the equivalence-guard.

Guards that give it the anti-drift property:

• Blocks are selected by a literal content signature matching exactly one template block — zero or many is a hard error (exit 2). The concrete worked examples (no <...>) are never selected.
• Any <...> surviving substitution fails the run — a new doc placeholder breaks the checker instead of running a half-filled command.
• Each block runs in a private temp CWD with set -euo pipefail prepended (the doc blocks carry none; without it the mid-block jq -e asserts and the cosign download | jq pipeline would fail open).

Recipes covered

Recipe	Doc section	Covered
(a) file VSA — ampel verify … jsonl:	Recommended: ampel	✅
(b) image VSA — ampel verify … oci:	Recommended: ampel	✅
by-digest — ampel … github: collector	By-digest fetch	✅ (--attestation-store)
verifiedLevels read (jq)	What verifiedLevels carries	✅
(c) file — cosign + jq fallback	Without ampel	✅
(d) image — cosign download attestation	Without ampel	✅
(e) raw provenance — gh attestation verify	Verifying raw provenance	✅ (--provenance)

Deferred to the showcase (need live network/registry, can't be hermetic): the ecosystem-native npm audit signatures / PyPI PEP-740 checks are out of scope (not VSA/provenance recipes). Private-image pull auth is out of scope (doc calls it out).

Two deviations from the literal brief — flagged per the draft's purpose

1. Tool install uses tools/go.mod (go install), not download_verify.sh with a hardcoded checksum. ampel v1.3.1 and cosign v3.0.6 are already pinned in tools/go.mod. A second download_verify pin of the same tools is exactly the duplicate-pin drift DEP_MGMT.md §Drift forbids ("a pin literal in more than one file must be single-sourced"); branch-1 / go install is the documented default for canonical Go tools, and #247 says ampel/cosign install via the go.mod manifest. So the composite installs them from go.mod (mirroring test/setup_integration.sh) — single source, Dependabot-covered, no checksum to drift. I did not add a download_verify install.

2. The "passes on good / fails on tampered" fixture check is an integration test, not the hermetic unit test. A keyless VSA verify needs Sigstore/Rekor/TUF; CLAUDE.md ("a test needing a real binary or network is an integration test") puts that out of the deterministic unit suite — which is why the existing test/consumer/verify_consumer_vsa.bats gates on require_sigstore. So:

• Hermetic unit suite (actions/verify-showcase-recipes/*.bats, runs in the PR bats job): the drift guard (every recipe extracts + fully substitutes against the real doc; duplicate/missing/new-placeholder fail closed) + stub-driven orchestration (exit-code plumbing, bounded retry, input validation). No network.
• Integration (test/consumer/verify_documented_recipes.bats, in INTEGRATION_BATS + the dogfood job): drives the script end-to-end against the captured npm fixture — good → 3 passed, tampered byte → fail closed.

(Note: tools/verify_documented_recipes.bats would not be picked up by make bats — its glob is tools/*/test.bats — so the hermetic tests live under the action, which the brief allowed.)

Retry note

Because a documented recipe is one fenced block, retry is at block granularity (coarser than the brief's "wrap only the Sigstore read"): the idempotent fetch is retried alongside the verify (harmless), and a genuine tamper burns the retries before failing (slower, still fail-closed). Direct consequence of extract-and-run.

Interface

--file/--bundle (file class) · --image/--digest (image class) · --provenance · --attestation-store · --resource-uri --repo --type · --non-strict (or WRANGLE_VSA_NON_STRICT=1, mirroring the verify-vsa gate). Exit 0 = all verified, 1 = a recipe failed, 2 = usage/config/doc-drift. All inputs are allowlist-validated before substitution (they reach an executed block).

Verification

• shellcheck -x clean on all new .sh and .bats (runner's exact flags). WSL001-007 conform by inspection (ast-grep unavailable in my sandbox; scripts match verify_vsa.sh preamble/[[ ]]/printf shape).
• New hermetic bats: 22/22 green.
• check_pin_ancestry / check_pin_freshness / check_catalog all exit 0 — the tools/ (non-catalog) + actions/ additions do not trip freshness. No wrangle workflow references the new action, so no self-ref converge is needed.
• No live smoke run (no showcase artifact/registry access from here); the integration test exercises the real-tool path in CI's dogfood job.

Unsure / for the owner

• The integration test uses the doc's remote policy locator (git+…@v0.3.1#…) against the v0.2.2-signed npm fixture; both resolve the same -v1 contract, so it should PASS in the dogfood job exactly as verify_consumer_vsa does — worth confirming on first CI run.
• Non-strict applies two narrow post-extraction relaxations (strict policy → nonstrict policy; @refs/tags/v[0-9.]+$ anchor → any ref), mirroring the verify-vsa gate. The drift guard asserts the strict verbatim form; non-strict is a tested transform on top.

🤖 Generated with Claude Code

[TomHennen]

claude: Smoke run done — the mechanism is empirically proven, not just stub-tested. Installed real ampel v1.3.1 + cosign v3.0.6 from `tools/go.mod` (the exact path the composite uses) and ran the integration test against live Sigstore:

```
ok 1 runner verifies a real release-tag VSA through the documented file recipes
ok 2 runner FAILS closed on tampered artifact bytes (valid VSA, wrong content)
```

This executes the extracted-and-substituted blocks 3/6/7 end-to-end (temp script with prepended `set -euo pipefail`, staging, the `jq -e` asserts, base64 decode) through real tools — so "the substituted blocks are valid commands that actually verify a real VSA" is now verified, not asserted. It also resolves my #1 flagged uncertainty: the doc's remote `git+https://github.com/TomHennen/[email protected]#…\` policy locator resolves and verifies the v0.2.2-signed npm fixture (nothing else tests the remote git+ fetch path — `verify_consumer_vsa` uses the local policy file).

Also eyeballed the substituted image blocks 4/8 (no live registry here to run them): `imagename=…; digest=…` first line and the `--digest "$digest" --digestAlg sha256` tail are correct; they're exercised for real in step 2's showcase.

Also failed-fast the tamper test (`WRANGLE_RECIPE_RETRIES=1`) so its deterministic failure doesn't burn 3 Sigstore round-trips.