feat(eval): first-class artifact grading — measure an ACTING persona by her HANDS (workspace file), not her mouth #672
Workflow file for this run
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| name: auto-close-queue-cards | |
| # Auto-close airc-queue cards when their PR merges into canary. | |
| # | |
| # GitHub's native "Closes #N" only closes issues automatically when the PR | |
| # lands in the default branch. Continuum lands work in canary first, so queue | |
| # cards otherwise remain open until someone cleans them up manually. | |
| # | |
| # On PR merge into canary, this workflow parses the PR body for queue-card refs, | |
| # verifies each target has an airc-queue-card-v1 envelope, marks it merged with | |
| # a status-log entry, and closes it. The AIRC CLI is checked out from | |
| # CambrianTech/airc because Continuum intentionally does not vendor it. | |
| on: | |
| pull_request: | |
| types: [closed] | |
| branches: [canary] | |
| concurrency: | |
| group: auto-close-queue-cards | |
| cancel-in-progress: false | |
| jobs: | |
| close-cards: | |
| if: github.event.pull_request.merged == true | |
| runs-on: ubuntu-latest | |
| permissions: | |
| issues: write | |
| pull-requests: read | |
| contents: read | |
| steps: | |
| - name: Checkout Continuum | |
| uses: actions/checkout@v4 | |
| - name: Checkout AIRC CLI | |
| uses: actions/checkout@v4 | |
| with: | |
| repository: CambrianTech/airc | |
| ref: canary | |
| path: .airc-src | |
| - name: Verify environment | |
| run: | | |
| set -euo pipefail | |
| which gh python3 bash | |
| gh --version | head -1 | |
| python3 --version | |
| bash --version | head -1 | |
| test -x .airc-src/airc | |
| - name: Run airc queue close-merged | |
| env: | |
| GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} | |
| run: | | |
| set -euo pipefail | |
| .airc-src/airc queue close-merged \ | |
| "${{ github.event.pull_request.html_url }}" \ | |
| --merge-sha "${{ github.event.pull_request.merge_commit_sha }}" \ | |
| --actor "github-actions[continuum#1142]" | |
| # ─── Post-merge auto-nudge (continuum#1179) ───────────────────── | |
| # When a PR merges, fire 'airc queue next' for the PR author so | |
| # they see a tailored candidate list as a comment on their just- | |
| # merged PR. Closes the "I forgot to look for next work" gap that | |
| # leaves agents idle between events. | |
| # | |
| # Identity assumption (v1): PR author's GH login == airc work | |
| # identity. Most contributors today have matching identities; | |
| # an identity-mapping table is a future PR (continuum#?). | |
| # | |
| # Best-effort: never fails the workflow if the nudge step errors. | |
| # The auto-close above is the load-bearing primitive; the nudge | |
| # is a UX win on top. | |
| - name: Post-merge auto-nudge (queue next candidates) | |
| if: always() | |
| continue-on-error: true | |
| env: | |
| GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} | |
| PR_AUTHOR: ${{ github.event.pull_request.user.login }} | |
| PR_NUMBER: ${{ github.event.pull_request.number }} | |
| run: | | |
| set -uo pipefail | |
| # Get top-5 next candidates from the queue. We intentionally | |
| # do NOT pass --owner here — codex review on continuum#1181 | |
| # caught that the workflow's airc binary (checked out from | |
| # CambrianTech/airc:canary) may not yet support that flag in | |
| # all build envs, and the nudge silently soft-fails when an | |
| # unsupported flag is passed. Until that's stable, the | |
| # post-merge comment shows the top-5 unowned-or-stale cards | |
| # — useful as a "here's pickable work" surface even without | |
| # per-author personalization. Personalization comes back in | |
| # a follow-up PR once --owner is guaranteed across all | |
| # consumer airc builds. | |
| if ! .airc-src/airc queue next --help >/dev/null 2>&1; then | |
| echo "::notice::airc queue next not available in this airc build; skipping post-merge nudge" | |
| exit 0 | |
| fi | |
| NEXT_OUT=$(.airc-src/airc queue next CambrianTech/continuum --limit 5 2>&1) || { | |
| echo "::warning::queue next failed; skipping nudge" | |
| echo "$NEXT_OUT" | head -20 | |
| exit 0 | |
| } | |
| # If the candidate list is empty (queue clean), don't post a | |
| # comment — empty nudge is noise. | |
| if ! printf '%s' "$NEXT_OUT" | grep -qE '^## [0-9]+\.'; then | |
| echo "::notice::no candidates available — skipping nudge comment" | |
| exit 0 | |
| fi | |
| # Post as a PR comment with a clear header + the candidate list. | |
| # --body-file via a temp file so the markdown content (backticks, | |
| # code spans) doesn't get shell-interpreted (continuum#1142 lesson). | |
| BODY_FILE=$(mktemp) | |
| { | |
| printf '## 🎯 Next pickable from the queue\n\n' | |
| printf '@%s — your PR just merged. ' "$PR_AUTHOR" | |
| printf 'Auto-fired by [post-merge nudge](https://github.com/CambrianTech/continuum/issues/1179) — closes the "I forgot to look for next work" gap that leaves agents idle between events.\n\n' | |
| printf '<details>\n<summary>Top candidates from `airc queue next`</summary>\n\n```\n' | |
| printf '%s\n' "$NEXT_OUT" | |
| printf '```\n</details>\n\n' | |
| printf '_To claim, run `airc queue claim <issue-url>` from your scope._\n' | |
| } > "$BODY_FILE" | |
| gh pr comment "$PR_NUMBER" --repo CambrianTech/continuum \ | |
| --body-file "$BODY_FILE" || \ | |
| echo "::warning::posting nudge comment failed (non-fatal)" | |
| rm -f "$BODY_FILE" |