docs(run-evaluation): retro evidence, eval-agent spec, rubric, ADR 0031, actor sketch#332
Open
norrietaylor wants to merge 9 commits into
Open
docs(run-evaluation): retro evidence, eval-agent spec, rubric, ADR 0031, actor sketch#332norrietaylor wants to merge 9 commits into
norrietaylor wants to merge 9 commits into
Conversation
Grounding questions for two deliverables: a retroactive evaluation of a completed consumer pipeline run, and a spec for an evaluation agent that measures future runs deterministically and by inference. Covers evidence access, run ground truth, rubric scope, eval-agent posture, and logistics. Co-Authored-By: Claude Fable 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01JsKK8YPAegjX9xDkjpPSgS
…harvest Structured harvest of the 39 spectacles issues surrounding the June pipeline run window: failure timeline, taxonomy, fix-cost accounting, safety-critical incidents, detection-gap matrix, and pre-run baseline yardsticks. Consumer identities scrubbed to placeholders; source data is this repository's own public issue tracker. Co-Authored-By: Claude Fable 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01JsKK8YPAegjX9xDkjpPSgS
…run evidence One-command gh-based export of a pipeline run's full forensic surface: tracking issue + sub-issue tree, timelines with label actors, comments, minimized flags and review threads via GraphQL, spec/arch/sdd PR data and diffs, workflow-run index, optional per-run jobs and agent artifacts. Fallback path for evaluating a run on a repository this session cannot read directly. Co-Authored-By: Claude Fable 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01JsKK8YPAegjX9xDkjpPSgS
Self-contained handoff for continuing the engagement in another session: mission, binding interview decisions, inherited state and verified facts, Phase 1 retro plan with the full metric checklist, Phase 2 eval-agent spec requirements, and repo constraints. Interview doc now carries the operator's answers (identities as placeholders). Co-Authored-By: Claude Fable 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01JsKK8YPAegjX9xDkjpPSgS
…r-agent sketch Two-part run-evaluation capability per the operator interview: deterministic collector (no engine, OTLP-exempt) + rubric-ratified inference judge, all four trigger tiers, roll-up-issue delivery, App identity default-on with an unset-safe enablement predicate, judge budget anchored to the evaluated feature's own usage, and a durable /retro-eval backfill entry point. The actor agent (bounded actions on findings) is sketched as spec 04. Co-Authored-By: Claude Fable 5 <[email protected]>
Phase-1 retro delivered to the consumer tracker; this commits the leak-scrubbed copy (no consumer identity, no stack names, no spend figures) and records both phases' completion state in the handoff brief. Co-Authored-By: Claude Fable 5 <[email protected]>
…y rubric item New post-mortem evidence: the generated spec inverted the source document's topology in three normative requirements; every downstream stage validates against the generated spec and is blind to the class by construction, so it was corrected only by amending the spec inside a remediation PR. Adds the addendum to the genericized retro and a judge-only source-fidelity item to the rubric (draft, unratified) and to spec R2.4's minimum coverage. Co-Authored-By: Claude Fable 5 <[email protected]>
…ec-edit tripwire Root-cause investigation of the evaluated run's spec drift: the generated spec was internally contradictory from birth (dual-model source, no reconciliation pass), no stage checks spec semantics, and a later spec-editing implementation PR harmonized toward the wrong pole. RB-input-3 gains the internal-consistency half (needs no source) and a re-run-on-spec- edit clause; the collector gains a deterministic spec-edit tripwire. Co-Authored-By: Claude Fable 5 <[email protected]>
…on-gap matrix Appends the four-stage root-cause chain (dual-model source x single-pass generator; no semantic spec gates; drift spread by a spec-editing implementation PR; corrected only when routing became load-bearing) to the genericized retro, and adds the spec-transcription-drift row to the detection-gap matrix with its deterministic (spec-edit tripwire) and inference (consistency + source-fidelity judge) signals. Co-Authored-By: Claude Fable 5 <[email protected]>
📝 WalkthroughSummary by CodeRabbit
WalkthroughThis PR adds documentation only: ADR 0031 describing a deterministic collector plus judge design for run evaluation, two new specs (run-evaluation and eval-actor), a scoring rubric, interview/handoff working documents, an evidence-export shell script, and retro report/evidence artifacts. ChangesRun Evaluation Design and Evidence
Estimated code review effort: 2 (Simple) | ~15 minutes Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 7
🧹 Nitpick comments (1)
docs/specs/04-spec-eval-actor/04-spec-eval-actor.md (1)
97-103: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winExpand verification to cover the remaining action rows.
The current proof artifacts only exercise the dispatch and fingerprint paths.
R1.2also defines stale-needs-human, integrity-tripwire, and merged-unnoticed incident actions, so those behaviors still lack explicit proof.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/specs/04-spec-eval-actor/04-spec-eval-actor.md` around lines 97 - 103, The proof artifacts for the actor spec only cover the dispatch and recurrence fingerprint paths, leaving the stale-`needs-human`, integrity-tripwire, and merged-unnoticed action rows unverified. Update the proof artifacts section in the spec to add explicit tests for those remaining `R1.2` behaviors, using the same style as the existing dispatch/fingerprint examples so the coverage maps cleanly to the corresponding action handlers and ledger outcomes.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/specs/03-spec-run-evaluation/03-spec-run-evaluation.md`:
- Around line 474-479: Clarify the collector’s App permissions to match R1.8 and
avoid the current “write access only to issues” ambiguity. Update the collector
permission description in the run-evaluation spec to explicitly list the full
least-privilege App set: issues: write, pull-requests: read, actions: read,
checks: read, and contents: read. Keep the rest of the collector behavior
description unchanged, including the sentinel-only edits and the `/eval` gate
references.
In `@docs/specs/03-spec-run-evaluation/evidence/_export-run-evidence.sh`:
- Around line 100-115: The GraphQL query in GQL only fetches the first page of
issue, pull request, and review-thread comments, so later items are missed from
the evidence export. Update the comments and reviewThreads connections to
paginate through all pages, and keep using the GraphQL fields that include
isMinimized/minimizedReason so the evidence bundle contains the full set of
comment metadata. Use the existing GQL query structure in
_export-run-evidence.sh to locate and adjust the affected connections.
- Around line 46-62: The issue is that _export-run-evidence.sh only walks two
levels of sub-issues, so deeper descendants never get added to .issuelist and
their evidence is missed. Update the issue traversal around numbers_in,
fetch_json, and the TRACKER/unit/task loop to use a recursive walk or queue that
continues fetching sub_issues for every discovered issue until no children
remain. Keep the export flow the same, but ensure every descendant ID is
appended to .issuelist before the final sort.
- Around line 74-82: The PR export logic is capped by the fixed `gh pr list
--limit 300` call, so `jq` filters in `_export-run-evidence.sh` may miss
matching `sdd/spec/arch` PRs in larger repos. Update the `gh pr list` invocation
in the PR index export block to page through all available PRs instead of
truncating at 300, then keep the existing `jq`, `EXTRA_PRS`, and `sort` steps
unchanged so the full set is filtered correctly.
- Around line 40-42: The fetch_json helper is masking transport/auth/rate-limit
failures by writing [] and allowing the export to continue, which can hide
missing evidence in the run export. Update fetch_json and the related PR
diff/workflow-run fetch paths in _export-run-evidence.sh to fail the script on
required fetch errors instead of substituting empty placeholders, so the export
only succeeds when all evidence sources are actually retrieved.
In `@docs/specs/03-spec-run-evaluation/evidence/_run-window-evidence.md`:
- Around line 1-3: The document’s provenance stamp is stale because the header
still says the state is as of 2026-07-01 while the evidence now includes a
2026-07-02 addendum/root-cause section. Update the “Data reflects state as of …”
timestamp in the document header to match the newest content, keeping the change
aligned with the existing run title and evidence narrative in
_run-window-evidence.md.
In `@docs/specs/03-spec-run-evaluation/rubric.md`:
- Around line 1-4: The rubric frontmatter is missing the recorded content hash
required by the contract, so item or scale edits won’t be detected by lint.
Update the frontmatter in the rubric document to include the recorded hash field
alongside the existing version/status metadata, using the same naming expected
by the rubric spec, so changes to in-file rubric content trigger a version bump.
Refer to the frontmatter block at the top of the rubric file and keep the
metadata consistent with the existing versioning scheme.
---
Nitpick comments:
In `@docs/specs/04-spec-eval-actor/04-spec-eval-actor.md`:
- Around line 97-103: The proof artifacts for the actor spec only cover the
dispatch and recurrence fingerprint paths, leaving the stale-`needs-human`,
integrity-tripwire, and merged-unnoticed action rows unverified. Update the
proof artifacts section in the spec to add explicit tests for those remaining
`R1.2` behaviors, using the same style as the existing dispatch/fingerprint
examples so the coverage maps cleanly to the corresponding action handlers and
ledger outcomes.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 2ef3dec8-e116-4f09-96cc-aac6bb98ceb0
📒 Files selected for processing (10)
decisions/0031-run-evaluation-split.mddocs/specs/03-spec-run-evaluation/03-spec-run-evaluation.mddocs/specs/03-spec-run-evaluation/_handoff.mddocs/specs/03-spec-run-evaluation/_interview.mddocs/specs/03-spec-run-evaluation/evidence/_export-run-evidence.shdocs/specs/03-spec-run-evaluation/evidence/_retro-report.mddocs/specs/03-spec-run-evaluation/evidence/_run-window-evidence.mddocs/specs/03-spec-run-evaluation/evidence/_run-window-issue-records.jsondocs/specs/03-spec-run-evaluation/rubric.mddocs/specs/04-spec-eval-actor/04-spec-eval-actor.md
Comment on lines
+474
to
+479
| - The collector runs on an App token with write access only to issues; it | ||
| edits only content carrying its own sentinel. Prompt-injection surface: | ||
| issue/PR text flows into scorecards — the collector treats all body text | ||
| as data (no command interpretation beyond the token-strict `/eval` gate, | ||
| which checks actor write access via the repository-permission API, not | ||
| `author_association`). |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '\n== Target section ==\n'
sed -n '450,500p' docs/specs/03-spec-run-evaluation/03-spec-run-evaluation.md
printf '\n== Search for R1.8 and permission/scope references ==\n'
rg -n "R1\.8|scope|scopes|permission|permissions|issues; it edits only content carrying its own sentinel|App token" docs/specs -SRepository: norrietaylor/spectacles
Length of output: 41724
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '\n== R1.8 and nearby requirements ==\n'
sed -n '206,232p' docs/specs/03-spec-run-evaluation/03-spec-run-evaluation.md
printf '\n== Collector/App permission references in run-evaluation docs ==\n'
rg -n "collector|App identity|permissions|pull-requests|issues|checks|actions|contents" docs/specs/03-spec-run-evaluation -S
printf '\n== Install docs permission set, if present ==\n'
rg -n "permissions|issues|pull-requests|checks|actions|contents" docs/sdd/install.md -SRepository: norrietaylor/spectacles
Length of output: 50379
Clarify the collector permissions
write access only to issues is too narrow here and conflicts with R1.8; spell out the full App set (issues: write, pull-requests: read, actions: read, checks: read, contents: read) so the install can be provisioned least-privilege without ambiguity.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/specs/03-spec-run-evaluation/03-spec-run-evaluation.md` around lines 474
- 479, Clarify the collector’s App permissions to match R1.8 and avoid the
current “write access only to issues” ambiguity. Update the collector permission
description in the run-evaluation spec to explicitly list the full
least-privilege App set: issues: write, pull-requests: read, actions: read,
checks: read, and contents: read. Keep the rest of the collector behavior
description unchanged, including the sentinel-only edits and the `/eval` gate
references.
Comment on lines
+40
to
+42
| fetch_json() { # fetch_json <path> <outfile> ; writes [] on failure | ||
| if ! api --paginate "$1" > "$2" 2>/dev/null; then echo '[]' > "$2"; fi | ||
| } |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
set -e
file='docs/specs/03-spec-run-evaluation/evidence/_export-run-evidence.sh'
wc -l "$file"
cat -n "$file" | sed -n '1,220p'Repository: norrietaylor/spectacles
Length of output: 7993
Fail fast on required evidence fetches. fetch_json() turns transport/auth/rate-limit failures into [], so the export can succeed while silently omitting issue, PR, timeline, review, and GraphQL data. The same applies to the PR diff fetches and the workflow-run index; those should fail the script instead of writing empty placeholders.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/specs/03-spec-run-evaluation/evidence/_export-run-evidence.sh` around
lines 40 - 42, The fetch_json helper is masking transport/auth/rate-limit
failures by writing [] and allowing the export to continue, which can hide
missing evidence in the run export. Update fetch_json and the related PR
diff/workflow-run fetch paths in _export-run-evidence.sh to fail the script on
required fetch errors instead of substituting empty placeholders, so the export
only succeeds when all evidence sources are actually retrieved.
Comment on lines
+46
to
+62
| # ---- 1. tracking issue + two-level sub-issue tree ------------------------- | ||
| note "tracking issue #$TRACKER and sub-issue tree" | ||
| api "repos/$REPO/issues/$TRACKER" > "$OUT/issues/$TRACKER.json" | ||
| fetch_json "repos/$REPO/issues/$TRACKER/sub_issues" "$OUT/issues/$TRACKER.sub_issues.json" | ||
|
|
||
| : > "$OUT/.issuelist" | ||
| echo "$TRACKER" >> "$OUT/.issuelist" | ||
| while IFS= read -r unit; do | ||
| [ -n "$unit" ] || continue | ||
| echo "$unit" >> "$OUT/.issuelist" | ||
| fetch_json "repos/$REPO/issues/$unit/sub_issues" "$OUT/issues/$unit.sub_issues.json" | ||
| while IFS= read -r task; do | ||
| [ -n "$task" ] || continue | ||
| echo "$task" >> "$OUT/.issuelist" | ||
| done <<< "$(numbers_in "$OUT/issues/$unit.sub_issues.json")" | ||
| done <<< "$(numbers_in "$OUT/issues/$TRACKER.sub_issues.json")" | ||
| sort -un "$OUT/.issuelist" -o "$OUT/.issuelist" |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Locate the script and inspect the relevant section with line numbers.
git ls-files 'docs/specs/03-spec-run-evaluation/evidence/_export-run-evidence.sh'
wc -l docs/specs/03-spec-run-evaluation/evidence/_export-run-evidence.sh
sed -n '1,220p' docs/specs/03-spec-run-evaluation/evidence/_export-run-evidence.sh
# Search for the helper functions and any other traversal logic.
rg -n 'collect_subissues|numbers_in|sub_issues|issuelist|timeline|comments|issues/' docs/specs/03-spec-run-evaluation/evidence/_export-run-evidence.sh docs/specs/03-spec-run-evaluation -g '!**/*.json'
# If the script references any helper file, inspect it too.
fd -a '.*' docs/specs/03-spec-run-evaluation/evidenceRepository: norrietaylor/spectacles
Length of output: 24485
Walk the full sub-issue tree. In docs/specs/03-spec-run-evaluation/evidence/_export-run-evidence.sh:51-62, this only adds the tracker’s children and grandchildren to .issuelist; deeper descendants are never exported, so their bodies/comments/timelines are missing. Use a recursive walk (or queue) from TRACKER so every descendant is included.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/specs/03-spec-run-evaluation/evidence/_export-run-evidence.sh` around
lines 46 - 62, The issue is that _export-run-evidence.sh only walks two levels
of sub-issues, so deeper descendants never get added to .issuelist and their
evidence is missed. Update the issue traversal around numbers_in, fetch_json,
and the TRACKER/unit/task loop to use a recursive walk or queue that continues
fetching sub_issues for every discovered issue until no children remain. Keep
the export flow the same, but ensure every descendant ID is appended to
.issuelist before the final sort.
Comment on lines
+74
to
+82
| gh pr list --repo "$REPO" --state all --limit 300 \ | ||
| --json number,title,headRefName,baseRefName,state,createdAt,closedAt,mergedAt,additions,deletions,changedFiles,author,labels,url \ | ||
| > "$OUT/prs/index.json" | ||
|
|
||
| : > "$OUT/.prlist" | ||
| jq -r '.[] | select(.headRefName | test("^(sdd|spec|arch)/")) | .number' \ | ||
| "$OUT/prs/index.json" >> "$OUT/.prlist" | ||
| for x in ${EXTRA_PRS:-}; do echo "$x" >> "$OUT/.prlist"; done | ||
| sort -un "$OUT/.prlist" -o "$OUT/.prlist" |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
git ls-files docs/specs/03-spec-run-evaluation/evidence/_export-run-evidence.sh
wc -l docs/specs/03-spec-run-evaluation/evidence/_export-run-evidence.sh
sed -n '1,140p' docs/specs/03-spec-run-evaluation/evidence/_export-run-evidence.shRepository: norrietaylor/spectacles
Length of output: 6462
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Look for any other references to the PR export limit or filtering logic.
rg -n --hidden --glob '!**/.git/**' 'gh pr list|--limit 300|headRefName | test\("^(sdd|spec|arch)/"\)' docs/specs/03-spec-run-evaluation -SRepository: norrietaylor/spectacles
Length of output: 428
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Check gh pr list flag semantics from local help, if available.
gh pr list --help | sed -n '1,220p'Repository: norrietaylor/spectacles
Length of output: 3302
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# If gh help is unavailable, inspect the docs/spec script for context around the cap and fallback behavior.
sed -n '1,220p' docs/specs/03-spec-run-evaluation/evidence/_export-run-evidence.shRepository: norrietaylor/spectacles
Length of output: 6851
🏁 Script executed:
#!/bin/bash
set -euo pipefail
python3 - <<'PY'
print("waiting for tool output")
PYRepository: norrietaylor/spectacles
Length of output: 185
Remove the fixed 300-PR cap
gh pr list --limit 300 truncates the repo before the jq prefix filter runs, so repos with more than 300 PRs can silently miss matching spec/arch/sdd PRs. Page through the full result set before filtering.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/specs/03-spec-run-evaluation/evidence/_export-run-evidence.sh` around
lines 74 - 82, The PR export logic is capped by the fixed `gh pr list --limit
300` call, so `jq` filters in `_export-run-evidence.sh` may miss matching
`sdd/spec/arch` PRs in larger repos. Update the `gh pr list` invocation in the
PR index export block to page through all available PRs instead of truncating at
300, then keep the existing `jq`, `EXTRA_PRS`, and `sort` steps unchanged so the
full set is filtered correctly.
Comment on lines
+100
to
+115
| GQL='query($owner:String!,$name:String!,$num:Int!){ | ||
| repository(owner:$owner,name:$name){ | ||
| issueOrPullRequest(number:$num){ | ||
| __typename | ||
| ... on Issue { comments(first:100){nodes{ | ||
| databaseId isMinimized minimizedReason createdAt author{login}}}} | ||
| ... on PullRequest { | ||
| comments(first:100){nodes{ | ||
| databaseId isMinimized minimizedReason createdAt author{login}}} | ||
| reviewThreads(first:100){nodes{ | ||
| isResolved isOutdated path | ||
| comments(first:50){nodes{databaseId isMinimized author{login}}}}} | ||
| } | ||
| } | ||
| } | ||
| }' |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Inspect the target file around the cited lines
file='docs/specs/03-spec-run-evaluation/evidence/_export-run-evidence.sh'
wc -l "$file"
sed -n '1,220p' "$file"
# Look for any pagination logic or follow-up queries in this script
rg -n "first:|after:|pageInfo|hasNextPage|endCursor|paginate|cursor" "$file"
# Inspect nearby documentation/spec files for how this export is described
rg -n "export-run-evidence|GraphQL|minimized|reviewThreads|comments\(first" docs/specs/03-spec-run-evaluation -g '!**/*.png'Repository: norrietaylor/spectacles
Length of output: 13463
Page the GraphQL connections. first:100 / first:50 only fetch the initial page here, so issue/PR comments and review-thread comments beyond those limits never enter the evidence bundle. The REST comment export only recovers bodies, not isMinimized.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/specs/03-spec-run-evaluation/evidence/_export-run-evidence.sh` around
lines 100 - 115, The GraphQL query in GQL only fetches the first page of issue,
pull request, and review-thread comments, so later items are missed from the
evidence export. Update the comments and reviewThreads connections to paginate
through all pages, and keep using the GraphQL fields that include
isMinimized/minimizedReason so the evidence bundle contains the full set of
comment metadata. Use the existing GQL query structure in
_export-run-evidence.sh to locate and adjust the affected connections.
Comment on lines
+1
to
+3
| # Retro Evidence Backbone — SDD Agent-Factory Run (June 2026, consumer tracking issue #478, networking feature, full-pipeline path, v0.3.0-era) | ||
|
|
||
| All issue numbers refer to the spectacles repository unless noted. The pilot target is referred to only as "the consumer repo." Data reflects state as of 2026-07-01. |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Update the as-of stamp.
The header still says the document reflects state as of 2026-07-01, but the body now includes a 2026-07-02 addendum/root-cause section. That makes the provenance stamp stale.
Proposed fix
-All issue numbers refer to the spectacles repository unless noted. The pilot target is referred to only as "the consumer repo." Data reflects state as of 2026-07-01.
+All issue numbers refer to the spectacles repository unless noted. The pilot target is referred to only as "the consumer repo." Main body reflects state as of 2026-07-01; the addendum is dated 2026-07-02.📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| # Retro Evidence Backbone — SDD Agent-Factory Run (June 2026, consumer tracking issue #478, networking feature, full-pipeline path, v0.3.0-era) | |
| All issue numbers refer to the spectacles repository unless noted. The pilot target is referred to only as "the consumer repo." Data reflects state as of 2026-07-01. | |
| # Retro Evidence Backbone — SDD Agent-Factory Run (June 2026, consumer tracking issue `#478`, networking feature, full-pipeline path, v0.3.0-era) | |
| All issue numbers refer to the spectacles repository unless noted. The pilot target is referred to only as "the consumer repo." Main body reflects state as of 2026-07-01; the addendum is dated 2026-07-02. |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/specs/03-spec-run-evaluation/evidence/_run-window-evidence.md` around
lines 1 - 3, The document’s provenance stamp is stale because the header still
says the state is as of 2026-07-01 while the evidence now includes a 2026-07-02
addendum/root-cause section. Update the “Data reflects state as of …” timestamp
in the document header to match the newest content, keeping the change aligned
with the existing run title and evidence narrative in _run-window-evidence.md.
Comment on lines
+1
to
+4
| --- | ||
| version: v1-draft | ||
| status: unratified | ||
| --- |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Add the recorded content hash required by R2.2.
version/status alone don't let the lint detect in-file item changes. The rubric contract expects a recorded hash so item/scale edits force a version bump.
🛠️ Suggested frontmatter addition
---
version: v1-draft
status: unratified
+hash: <rubric-content-hash>
---📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| --- | |
| version: v1-draft | |
| status: unratified | |
| --- | |
| --- | |
| version: v1-draft | |
| status: unratified | |
| hash: <rubric-content-hash> | |
| --- |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/specs/03-spec-run-evaluation/rubric.md` around lines 1 - 4, The rubric
frontmatter is missing the recorded content hash required by the contract, so
item or scale edits won’t be detected by lint. Update the frontmatter in the
rubric document to include the recorded hash field alongside the existing
version/status metadata, using the same naming expected by the rubric spec, so
changes to in-file rubric content trigger a version bump. Refer to the
frontmatter block at the top of the rubric file and keep the metadata consistent
with the existing versioning scheme.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Deliverables of the run-evaluation engagement (retro of the evaluated pilot run + the eval capability it motivates):
docs/specs/03-spec-run-evaluation/— spec for the run-evaluation suite: a deterministic collector (sdd-eval-collect: no engine, OTLP-exempt per ADR 0020 §5, all four trigger tiers, roll-up-issue delivery, versionedeval_snapshot.json) plus an engine-bearing judge (sdd-eval-judge: pinned model, rubric-version-stamped scores, budget anchored to the evaluated feature's own usage), and a durable/retro-eval <issue>backfill entry point whose first manual execution was the retro itself.rubric.md— v1-draft scoring rubric (unratified): 17 items, each with anchored scales, evidence sources, and a machine-check class. Merging this PR is the natural moment for the first ratification decision (/ratify-rubric v1per spec R2.3).docs/specs/04-spec-eval-actor/— companion sketch for the opt-in actor agent (bounded, audited condition→action table; never clearsneeds-human).evidence/— genericized retro report (scorecard vs. the pre-run yardsticks, remediation-delta split, priors audit, spec-drift root cause) and the updated detection-gap matrix, which is the collector's and judge's metric backlog.Found during the retro, filed separately
SDD_STATUSis unset; the status surface has never run on any consumer.Verification
lint.ymlgates apply (leak-scan: all committed files carry no consumer identity, no internal URL, no cost figure — underscore-prefixed evidence files are additionally excluded from distillery ingestion).docs/specs/TEMPLATE.mdfrontmatter + demoable-unit format; ADR numbered next-free (0031); rubric shipsstatus: unratifiedand takes effect only through the spec's ratification gate.🤖 Generated with Claude Code
https://claude.ai/code/session_016pTpLspS373UqfiyxBsJwa