Skip to content

feat(eval): first-class artifact grading — measure an ACTING persona by her HANDS (workspace file), not her mouth#1942

Merged
joelteply merged 1 commit into
canaryfrom
feat/artifact-grading-workspace-file
Jul 8, 2026
Merged

feat(eval): first-class artifact grading — measure an ACTING persona by her HANDS (workspace file), not her mouth#1942
joelteply merged 1 commit into
canaryfrom
feat/artifact-grading-workspace-file

Conversation

@joelteply

Copy link
Copy Markdown
Contributor

The eval grades settled.spoken — her spoken answer. But when a persona ACTS (writes code to a file, compiles,
fixes), her code is in the FILE and her spoken text becomes a summary — so the act→verify loop scored near-zero
not because she failed but because her work was invisible to a speech-grader. (Confirmed: solo WITH a verify
directive scored 25% vs 75% spoken, purely because she filed the code.)

Fix (first-class, replaces the /tmp/grade.py hack): a solution_file field on EvalTask. When set alongside
test, the grade reads HER FILE via new gym_grader::test_grade_file and runs the SAME harness as a spoken
answer (grade_rust: strip her main, append the task's test, compile, run). The file resolves relative to the
core cwd — the workspace root code/write sandboxes to — so a relative sol_x.rs lands exactly where the grader
reads. Fails LOUD (never a silent pass) on a missing file ("never wrote it" = acts=0), an empty file, or a path
escaping the workspace; the file is removed after grading so a stale artifact can't false-pass a later task.

Committed reproducible gym docs/genome/hard-rs-acted.jsonl (8 tasks: verify-prompt + solution_file + hidden
test) — no /tmp fixtures, no external grader script; the eval grades her file natively. Unit test covers
pass-correct / clean-up / fail-loud-missing / refuse-traversal.

This is the seam that finally makes the act→verify loop — and the team's reviewer catching a real compile error —
show up in the score. Runs detached via #86; survives the client via #85.

Co-Authored-By: Claude Opus 4.8 [email protected]
Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo

…by her HANDS (workspace file), not her mouth

The eval grades `settled.spoken` — her spoken answer. But when a persona ACTS (writes code to a file, compiles,
fixes), her code is in the FILE and her spoken text becomes a summary — so the act→verify loop scored near-zero
not because she failed but because her work was invisible to a speech-grader. (Confirmed: solo WITH a verify
directive scored 25% vs 75% spoken, purely because she filed the code.)

Fix (first-class, replaces the /tmp/grade.py hack): a `solution_file` field on EvalTask. When set alongside
`test`, the grade reads HER FILE via new `gym_grader::test_grade_file` and runs the SAME harness as a spoken
answer (`grade_rust`: strip her `main`, append the task's test, compile, run). The file resolves relative to the
core cwd — the workspace root `code/write` sandboxes to — so a relative `sol_x.rs` lands exactly where the grader
reads. Fails LOUD (never a silent pass) on a missing file ("never wrote it" = acts=0), an empty file, or a path
escaping the workspace; the file is removed after grading so a stale artifact can't false-pass a later task.

Committed reproducible gym `docs/genome/hard-rs-acted.jsonl` (8 tasks: verify-prompt + solution_file + hidden
test) — no /tmp fixtures, no external grader script; the eval grades her file natively. Unit test covers
pass-correct / clean-up / fail-loud-missing / refuse-traversal.

This is the seam that finally makes the act→verify loop — and the team's reviewer catching a real compile error —
show up in the score. Runs detached via #86; survives the client via #85.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
@joelteply joelteply merged commit e63261f into canary Jul 8, 2026
@joelteply joelteply deleted the feat/artifact-grading-workspace-file branch July 8, 2026 06:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant