feat(e2e): /e2e staging-repo dispatcher (walking skeleton)#282
feat(e2e): /e2e staging-repo dispatcher (walking skeleton)#282norrietaylor wants to merge 3 commits into
Conversation
Walking skeleton for #89: drive a synthetic feature through the full SDD lifecycle on a dedicated staging repo, assert terminal state, report cost. - e2e-dispatch.yml: plain Actions orchestrator (no lock). /e2e [scenario] from write-access authors, nightly schedule, workflow_dispatch. Gate -> provision -> seed -> poll -> assert -> cost -> report. - Provision by ref: quick-setup.sh --ref <pr-head-sha> re-pins staging wrappers; cross-repo uses:@<sha> runs the PR's hosted lock. No lock push. - e2e-assert.py: walks staging state via gh api, checks expectations.yml (terminal label, artifact globs, PRs opened/merged). - e2e-setup-staging.sh: one-time staging provisioning, documented, not CI-run. - happy-path-feature fixture; docs/sdd/e2e.md + mkdocs nav entry. Cost is staging Actions minutes + a coarse per-scenario token constant; lock metadata carries no token cost. happy-path-bug, revise-loop-spec, and needs-human-handoff are follow-up fixture-only PRs. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
✅ Files skipped from review due to trivial changes (1)
🚧 Files skipped from review as they are similar to previous changes (1)
📝 WalkthroughSummary by CodeRabbit
WalkthroughAdds a complete E2E testing pipeline: a new ChangesE2E Staging Dispatcher and Assertion Pipeline
Sequence Diagram(s)sequenceDiagram
participant Trigger as Trigger (comment/cron/dispatch)
participant Workflow as e2e-dispatch.yml
participant StagingRepo as Staging Repo (GitHub)
participant AssertScript as e2e-assert.py
participant SpectaclesIssue as Spectacles Issue/PR
Trigger->>Workflow: fire event (scenario, ref)
Workflow->>Workflow: resolve inputs, gate (disable switch, fixture check, Layer-0 scan)
Workflow->>Workflow: mint app installation token
Workflow->>StagingRepo: quick-setup.sh (pin wrappers to ref)
Workflow->>StagingRepo: close stale seeded issues
Workflow->>StagingRepo: create tracking issue with seed labels
loop poll every ~60s until terminal or timeout
Workflow->>StagingRepo: read tracking issue labels
StagingRepo-->>Workflow: current labels
end
Workflow->>AssertScript: run with --repo, --issue, --expect, --since
AssertScript->>StagingRepo: gh api (issue labels, tree, PRs)
StagingRepo-->>AssertScript: JSON responses
AssertScript-->>Workflow: exit 0 (pass) or exit 1 + failure list
Workflow->>StagingRepo: query workflow runs → sum durations (cost)
Workflow->>SpectaclesIssue: post Markdown verdict comment (or create nightly issue)
alt assertion failed
Workflow->>StagingRepo: label tracking issue e2e:failed
Workflow-->>Trigger: exit failure
else assertion passed
Workflow-->>Trigger: exit success
end
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Possibly related issues
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
The compiled locks use engine: claude and read CLAUDE_CODE_OAUTH_TOKEN, not ANTHROPIC_API_KEY. Update the staging setup script and docs to match, and add GH_AW_OTEL_ENDPOINT to the optional telemetry secrets. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 9
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/e2e-dispatch.yml:
- Around line 176-188: The reset step only closes prior e2e:seed issues but
leaves old artifact files on the staging default branch, allowing
scripts/e2e-assert.py to find and validate artifacts from previous runs instead
of the current one, which masks regressions. Add logic to either reset the
staging repository default branch to a clean baseline by removing old artifact
files (particularly docs/specs/**/*.md) before the E2E run begins, or
alternatively modify the artifact assertion logic in scripts/e2e-assert.py to
scope checks only to artifacts produced by the current seeded run instead of
scanning the entire default branch history.
- Around line 86-88: The current validation for STAGING_REPO only checks if it
is empty, but does not validate its content before write/admin tokens are minted
in later steps. Add additional guard conditions after the empty check to ensure
STAGING_REPO is properly scoped: verify it targets a repo within the expected
owner and confirm it is not the source repo itself. These validation checks
should use the stop function to fail fast if the STAGING_REPO configuration is
unsafe or misconfigured.
- Around line 140-145: The jq filter in the gh api check-runs query only counts
check runs with explicit failure, cancelled, or timed_out conclusions, but
ignores pending or in-progress check runs that haven't yet completed. Modify the
select condition to also include check runs where conclusion is null or status
is not "completed", so that any queued or in-progress Layer-0 checks are treated
as not green and prevent the staging run from starting.
- Around line 59-60: The checkout action step named "Check out spectacles
(fixtures + scripts)" uses an unpinned version (actions/checkout@v4) and does
not disable credential persistence. To fix this, pin the actions/checkout action
to a specific commit SHA (format: actions/checkout@<full-sha>) consistent with
how other actions are pinned in the workflow, and add a with block containing
persist-credentials: false to prevent the GitHub token from being persisted in
the workspace.
- Around line 157-159: The staging_repo output from steps.gate is currently in
owner/name format, but the create-github-app-token action's repositories field
requires only the bare repository name. Extract just the repository name portion
(the part after the forward slash) from the ${{ steps.gate.outputs.staging_repo
}} output before passing it to the repositories field. This can be done by using
string manipulation to split on the "/" character and take only the second part,
or by modifying the gate step to output both the full owner/name format and a
separate bare repository name value that can be referenced here.
- Around line 54-58: In the workflow job permissions block (currently containing
contents, issues, and pull-requests read permissions), add a new permission line
for `checks: read`. This will grant the workflow job the necessary permission to
access the check-runs API endpoint that is called by the Layer-0 gate, which
prevents API call failures when the workflow runs on private or restricted
repositories.
In `@docs/sdd/e2e.md`:
- Line 17: The token budget limit for the issue.md file in the documentation is
stated as "≤ 50 tokens" but conflicts with the PR objectives which specify "~30
tokens per fixture". Update the documentation in the e2e.md file at the location
describing `tests/fixtures/e2e/<scenario>/issue.md` to align the token limit
with the actual constraint (either change it to "~30 tokens" or clarify the
correct intended boundary) to ensure consistency with the stated PR objectives.
In `@scripts/e2e-setup-staging.sh`:
- Around line 94-95: The current error handling on lines 94-95 suppresses all
failures from the gh api command indiscriminately, including auth, permission,
and API errors, which can hide real issues and leave branch protection in place.
Instead of using the broad || operator that ignores any gh api failure, capture
the actual error output from the gh api DELETE command and only suppress the
specific case where branch protection doesn't exist, while allowing auth
failures, permission errors, and other API errors to propagate and fail the
script appropriately. This will ensure that later operations like the --direct
install fail in the right place with clear error messages rather than silently
continuing with protected branches.
- Around line 110-120: The set_secret function currently silently skips missing
secrets with just a warning, but required secrets like APP_PRIVATE_KEY and
ANTHROPIC_API_KEY should cause the script to fail with an error instead.
Additionally, when the run command executes in dry-run mode, it echoes the full
gh secret set command including the secret value in the --body parameter,
exposing sensitive information. Modify the set_secret function to differentiate
between required and optional secrets, exiting with an error for required ones
that are missing, and refactor the gh secret set command execution to avoid
echoing the secret value in dry-run mode, either by passing the secret via stdin
instead of --body parameter, or by using a different approach that doesn't
expose the sensitive value in the command output.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 3a883194-9ee0-4eeb-8229-14e57e3af769
📒 Files selected for processing (7)
.github/workflows/e2e-dispatch.ymldocs/sdd/e2e.mdmkdocs.ymlscripts/e2e-assert.pyscripts/e2e-setup-staging.shtests/fixtures/e2e/happy-path-feature/expectations.ymltests/fixtures/e2e/happy-path-feature/issue.md
| permissions: | ||
| contents: read | ||
| issues: read | ||
| pull-requests: read | ||
| steps: |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
cat -n .github/workflows/e2e-dispatch.yml | head -100Repository: norrietaylor/spectacles
Length of output: 4689
🏁 Script executed:
cat -n .github/workflows/e2e-dispatch.yml | sed -n '100,200p'Repository: norrietaylor/spectacles
Length of output: 5431
🌐 Web query:
GitHub API check-runs endpoint permission scope checks read
💡 Result:
The permissions required to access the GitHub Check Runs API depend on the type of authentication used [1][2][3]. For GitHub Apps: GitHub Apps must be granted the Checks repository permission to interact with check runs [4]. - Read Access: To get check runs or annotations, the app must have the checks:read permission on a private repository or pull access to a public repository [2][3]. - Write Access: To create or update check runs, the app must have the checks:write permission [5][2][3]. For OAuth Apps and Authenticated Users (Personal Access Tokens): OAuth apps and users cannot create or manage check runs (such as creating them or re-requesting them); these actions are exclusive to GitHub Apps [1][5][6]. However, they can view check runs and check suites using the repo scope [1][2][3]. For private repositories, a token with the repo scope is required to access these endpoints [1]. In summary, if you are attempting to perform a read-only operation (like listing or getting details of a check run), a GitHub App requires checks:read permissions (or appropriate repository access), while an OAuth app or user requires the repo scope [2][3]. If you require write operations, you must use a GitHub App with checks:write permissions [1][5][7].
Citations:
- 1: https://docs.github.com/en/rest/checks/runs
- 2: https://help.github.com/en/[email protected]/rest/reference/checks
- 3: https://help.github.com/en/[email protected]/rest/checks/runs
- 4: https://github.com/github/docs/blob/main/content/apps/creating-github-apps/writing-code-for-a-github-app/building-ci-checks-with-a-github-app.md
- 5: https://docs.github.com/rest/guides/getting-started-with-the-checks-api
- 6: https://docs.github.com/en/rest/checks/suites
- 7: https://github.com/github/docs/blob/main/content/rest/guides/using-the-rest-api-to-interact-with-checks.md
🌐 Web query:
GitHub Actions github.token default permissions scopes job permissions
💡 Result:
The GITHUB_TOKEN is an automatically generated, short-lived secret used to authenticate API requests within GitHub Actions workflows [1][2]. Its permissions can be managed at the repository or organization level, as well as explicitly defined within workflow files to follow the principle of least privilege [1][3][4]. Default Permissions The default permissions for the GITHUB_TOKEN depend on repository and organization settings [5][4]. Newer repositories (created since early 2023) generally default to a restricted setting, typically providing read-only access to repository contents and packages [6][4]. Organizations and administrators can choose between: - Restricted: Read access to contents and packages [4]. - Permissive: Read and write access to all available scopes [1][4]. Explicit Job and Workflow Permissions You can explicitly define GITHUB_TOKEN permissions in your workflow YAML file to override default settings [3][2]. This can be done at the entire workflow level or for individual jobs [1][7]. - Syntax: Use the permissions key in the YAML file [1][3]. - Behavior: When the permissions key is explicitly defined, all unspecified permissions are set to none, with the exception of the metadata scope, which always retains read access [8][2]. - Scoping: Permissions defined at the job level override any permissions defined at the workflow level [9][7]. They do not merge; the job-level declaration replaces the workflow-level declaration entirely [9]. Special Cases - Forks: Pull requests from public forks are always granted a read-only token, regardless of the repository's configured default or explicitly defined workflow permissions, as a security measure to prevent unauthorized access [1][8]. - Metadata: The metadata scope is always granted read access and cannot be revoked [8][9]. Best Practices To improve security, it is recommended to follow the principle of least privilege by explicitly defining the minimum required permissions for each job, rather than relying on default settings [3][10]. If a task requires permissions beyond those available to the GITHUB_TOKEN (e.g., access to other repositories), consider using a GitHub App or a fine-grained personal access token [3].
Citations:
- 1: https://github.blog/changelog/2021-04-20-github-actions-control-permissions-for-github_token/
- 2: https://dev.to/github/the-githubtoken-in-github-actions-how-it-works-change-permissions-customizations-3cgp
- 3: https://help.github.com/en/actions/automating-your-workflow-with-github-actions/authenticating-with-the-github_token
- 4: https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/enabling-features-for-your-repository/managing-github-actions-settings-for-a-repository
- 5: https://github.com/github/docs/blob/main/content/repositories/managing-your-repositorys-settings-and-features/enabling-features-for-your-repository/managing-github-actions-settings-for-a-repository.md
- 6: https://github.blog/changelog/2023-02-02-github-actions-updating-the-default-github_token-permissions-to-read-only/
- 7: https://www.kenmuse.com/blog/github-actions-workflow-permissions/
- 8: https://docs.github.com/en/[email protected]/actions/security-guides/automatic-token-authentication
- 9: https://adaptive-enforcement-lab.com/secure/github-actions-security/token-permissions/job-scoping/
- 10: https://github.com/github/docs/blob/main/content/actions/reference/security/secure-use.md
🏁 Script executed:
cat -n .github/workflows/e2e-dispatch.yml | tail -50Repository: norrietaylor/spectacles
Length of output: 2682
🏁 Script executed:
grep -n "check-runs\|permissions:" .github/workflows/e2e-dispatch.ymlRepository: norrietaylor/spectacles
Length of output: 175
Add checks: read permission to the job.
The Layer-0 gate at line 140 calls repos/$REPO/commits/$SHA/check-runs with github.token, but the job permissions do not grant Checks read access. When permissions are explicitly defined in a workflow job, all unspecified permissions default to none (except metadata), causing this API call to fail on private or restricted repositories.
Proposed fix
permissions:
contents: read
+ checks: read
issues: read
pull-requests: read📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| permissions: | |
| contents: read | |
| issues: read | |
| pull-requests: read | |
| steps: | |
| permissions: | |
| contents: read | |
| checks: read | |
| issues: read | |
| pull-requests: read | |
| steps: |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.github/workflows/e2e-dispatch.yml around lines 54 - 58, In the workflow job
permissions block (currently containing contents, issues, and pull-requests read
permissions), add a new permission line for `checks: read`. This will grant the
workflow job the necessary permission to access the check-runs API endpoint that
is called by the Layer-0 gate, which prevents API call failures when the
workflow runs on private or restricted repositories.
| - name: Check out spectacles (fixtures + scripts) | ||
| uses: actions/checkout@v4 |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
# First, let's see the file and the specific lines in context
cat -n .github/workflows/e2e-dispatch.yml | head -80Repository: norrietaylor/spectacles
Length of output: 3705
🏁 Script executed:
# Also check if there are other action usages to understand the pattern
rg "uses:" .github/workflows/e2e-dispatch.yml -A 1Repository: norrietaylor/spectacles
Length of output: 369
🏁 Script executed:
# Verify if the SHA exists for actions/checkout
web_search
GitHub actions/checkout v4 commit SHA 34e114876b0b11c390a56381ad16ebd13914f8d5Repository: norrietaylor/spectacles
Length of output: 163
🏁 Script executed:
# Let's see more context around line 59-60 to check for any existing 'with' clauses
sed -n '58,65p' .github/workflows/e2e-dispatch.ymlRepository: norrietaylor/spectacles
Length of output: 277
🏁 Script executed:
# Check all action usages in the workflow to see the pinning pattern
rg "uses:" .github/workflows/e2e-dispatch.yml -B 1Repository: norrietaylor/spectacles
Length of output: 465
🌐 Web query:
actions/checkout v4 commit SHA 34e114876b0b11c390a56381ad16ebd13914f8d5
💡 Result:
The commit SHA 34e114876b0b11c390a56381ad16ebd13914f8d5 corresponds to version v4.3.1 of the official actions/checkout GitHub Action [1][2][3]. It is frequently used by organizations and developers to pin the actions/checkout dependency to a specific, immutable version [1][3][4]. By using the full 40-character commit SHA instead of a floating tag (like @v4), users can prevent potential security issues caused by unauthorized modifications or force-pushes to version tags [3][5]. It is standard practice to accompany this SHA with a comment indicating the version (e.g., # v4.3.1) to maintain clarity and facilitate automated updates via tools like Dependabot [3][5].
Citations:
- 1: obsproject/obs-studio@3fa6da6
- 2: Ultrathink-Solutions/looker-mcp-server@6adb6a7
- 3: 3ncr/tokencrypt-php@ccc0a92
- 4: KooshaPari/AgilePlus@7f61697
- 5: https://psmodule.io/docs/GitHub-Actions/Standards/
🌐 Web query:
GitHub actions/checkout@v4 latest releases SHA pinning security
💡 Result:
Pinning GitHub Actions, including actions/checkout@v4, to a full-length commit SHA is a recommended security best practice for ensuring the immutability and integrity of your CI/CD workflows [1][2]. Security Rationale Using a version tag (like @v4) is convenient but mutable; maintainers can move tags, or a compromised repository could be used to redirect a tag to malicious code [3][4]. Pinning to a full-length (40-character) commit SHA ensures that the exact code you audited and intended to run is the code executed, protecting against such supply chain attacks [1][3][2]. How to Pin Replace the tag with the full commit SHA, while optionally including the original version tag in a comment for readability and compatibility with dependency update tools [1][5]: - uses: actions/checkout@b4ffde6 # v4.1.1 Policy Enforcement GitHub now provides organization and repository-level policies that administrators can enable to enforce full commit SHA pinning [6][7]. When this policy is active, any workflow attempting to use a mutable reference (tags or branches) will fail [6][7]. Management and Updates To mitigate the maintenance burden of manually updating pinned SHAs, you should use automated dependency management tools like Dependabot or Renovate [1][8][4]. These tools recognize SHA-pinned actions and can open pull requests to update the SHA whenever a new version is released, ensuring your workflows remain secure while benefiting from the latest updates [6][8][5]. Top results: [1][6][2][7][4]
Citations:
- 1: https://adaptive-enforcement-lab.com/secure/github-actions-security/action-pinning/sha-pinning/
- 2: https://docs.github.com/en/actions/reference/security/secure-use
- 3: https://actsense.dev/vulnerabilities/no_hash_pinning/
- 4: https://github.com/kubernetes/community/blob/main/github-management/github-actions-policy.md
- 5: https://blog.rafaelgss.dev/why-you-should-pin-actions-by-commit-hash
- 6: https://www.romainlespinasse.dev/posts/github-actions-commit-sha-pinning/
- 7: https://github.blog/changelog/2025-08-15-github-actions-policy-now-supports-blocking-and-sha-pinning-actions/
- 8: https://emmer.dev/blog/pin-your-github-actions-to-protect-against-mutability/
Pin checkout action to SHA and disable persisted credentials.
The workflow uses an unpinned third-party action (actions/checkout@v4) while other actions in the workflow are pinned to commit SHAs (e.g., actions/create-github-app-token@bcd2ba49218906704ab6c1aa796996da409d3eb1). Additionally, the checkout step does not disable credential persistence, which can leak the GitHub token to the workspace. Both of these violate security best practices and the repository's action-pinning policy.
Proposed fix
- name: Check out spectacles (fixtures + scripts)
- uses: actions/checkout@v4
+ uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1
+ with:
+ persist-credentials: false📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| - name: Check out spectacles (fixtures + scripts) | |
| uses: actions/checkout@v4 | |
| - name: Check out spectacles (fixtures + scripts) | |
| uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1 | |
| with: | |
| persist-credentials: false |
🧰 Tools
🪛 zizmor (1.25.2)
[warning] 59-60: credential persistence through GitHub Actions artifacts (artipacked): does not set persist-credentials: false
(artipacked)
[error] 60-60: unpinned action reference (unpinned-uses): action is not pinned to a hash (required by blanket policy)
(unpinned-uses)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.github/workflows/e2e-dispatch.yml around lines 59 - 60, The checkout action
step named "Check out spectacles (fixtures + scripts)" uses an unpinned version
(actions/checkout@v4) and does not disable credential persistence. To fix this,
pin the actions/checkout action to a specific commit SHA (format:
actions/checkout@<full-sha>) consistent with how other actions are pinned in the
workflow, and add a with block containing persist-credentials: false to prevent
the GitHub token from being persisted in the workspace.
Sources: Learnings, Linters/SAST tools
| if [ -z "${STAGING_REPO:-}" ]; then | ||
| stop "SPECTACLES_E2E_STAGING_REPO is unset; nothing to target." | ||
| fi |
There was a problem hiding this comment.
Reject unsafe staging repo configuration before any write token is minted.
STAGING_REPO is only checked for non-empty, but later steps provision and mutate it with a write/admin App token. Add guards so a misconfigured variable cannot target the source repo or a repo outside the expected owner.
Proposed fix
if [ -z "${STAGING_REPO:-}" ]; then
stop "SPECTACLES_E2E_STAGING_REPO is unset; nothing to target."
fi
+ case "$STAGING_REPO" in
+ */*) : ;;
+ *) stop "SPECTACLES_E2E_STAGING_REPO must be owner/name, got '$STAGING_REPO'." ;;
+ esac
+ if [ "$STAGING_REPO" = "$REPO" ]; then
+ stop "SPECTACLES_E2E_STAGING_REPO must point at a dedicated staging repo, not $REPO."
+ fi
+ if [ "${STAGING_REPO%%/*}" != "${REPO%%/*}" ]; then
+ stop "staging repo must be owned by ${REPO%%/*}; got $STAGING_REPO."
+ fi📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| if [ -z "${STAGING_REPO:-}" ]; then | |
| stop "SPECTACLES_E2E_STAGING_REPO is unset; nothing to target." | |
| fi | |
| if [ -z "${STAGING_REPO:-}" ]; then | |
| stop "SPECTACLES_E2E_STAGING_REPO is unset; nothing to target." | |
| fi | |
| case "$STAGING_REPO" in | |
| */*) : ;; | |
| *) stop "SPECTACLES_E2E_STAGING_REPO must be owner/name, got '$STAGING_REPO'." ;; | |
| esac | |
| if [ "$STAGING_REPO" = "$REPO" ]; then | |
| stop "SPECTACLES_E2E_STAGING_REPO must point at a dedicated staging repo, not $REPO." | |
| fi | |
| if [ "${STAGING_REPO%%/*}" != "${REPO%%/*}" ]; then | |
| stop "staging repo must be owned by ${REPO%%/*}; got $STAGING_REPO." | |
| fi |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.github/workflows/e2e-dispatch.yml around lines 86 - 88, The current
validation for STAGING_REPO only checks if it is empty, but does not validate
its content before write/admin tokens are minted in later steps. Add additional
guard conditions after the empty check to ensure STAGING_REPO is properly
scoped: verify it targets a repo within the expected owner and confirm it is not
the source repo itself. These validation checks should use the stop function to
fail fast if the STAGING_REPO configuration is unsafe or misconfigured.
| failing="$(gh api "repos/$REPO/commits/$SHA/check-runs" \ | ||
| --jq '[.check_runs[] | select(.conclusion=="failure" or .conclusion=="cancelled" or .conclusion=="timed_out")] | length')" | ||
| if [ "$failing" -gt 0 ]; then | ||
| echo "e2e-dispatch: $failing Layer-0 check(s) not green on $SHA; refusing to run." >&2 | ||
| exit 1 | ||
| fi |
There was a problem hiding this comment.
Treat pending check runs as not green.
The comment says pending required checks fail fast, but the query only counts completed failures. A queued/in-progress Layer-0 check lets the expensive staging run start before the PR head is actually green.
Proposed fix
- failing="$(gh api "repos/$REPO/commits/$SHA/check-runs" \
- --jq '[.check_runs[] | select(.conclusion=="failure" or .conclusion=="cancelled" or .conclusion=="timed_out")] | length')"
- if [ "$failing" -gt 0 ]; then
- echo "e2e-dispatch: $failing Layer-0 check(s) not green on $SHA; refusing to run." >&2
+ not_green="$(gh api "repos/$REPO/commits/$SHA/check-runs" \
+ --jq '[.check_runs[] | select(.status!="completed" or (.conclusion!="success" and .conclusion!="neutral" and .conclusion!="skipped"))] | length')"
+ if [ "$not_green" -gt 0 ]; then
+ echo "e2e-dispatch: $not_green Layer-0 check(s) not green on $SHA; refusing to run." >&2
exit 1
fi🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.github/workflows/e2e-dispatch.yml around lines 140 - 145, The jq filter in
the gh api check-runs query only counts check runs with explicit failure,
cancelled, or timed_out conclusions, but ignores pending or in-progress check
runs that haven't yet completed. Modify the select condition to also include
check runs where conclusion is null or status is not "completed", so that any
queued or in-progress Layer-0 checks are treated as not green and prevent the
staging run from starting.
| repositories: | | ||
| ${{ github.event.repository.name }} | ||
| ${{ steps.gate.outputs.staging_repo }} |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
# First, locate and examine the e2e-dispatch.yml file
find . -name "e2e-dispatch.yml" -type fRepository: norrietaylor/spectacles
Length of output: 104
🏁 Script executed:
# Also check the structure of the workflows directory
ls -la .github/workflows/ 2>/dev/null || echo "No workflows found in standard location"Repository: norrietaylor/spectacles
Length of output: 2574
🏁 Script executed:
# Search for e2e-dispatch.yml in the repository
git ls-files | grep -i "e2e-dispatch"Repository: norrietaylor/spectacles
Length of output: 102
🏁 Script executed:
# Read the e2e-dispatch.yml file, focusing on the relevant sections
wc -l .github/workflows/e2e-dispatch.ymlRepository: norrietaylor/spectacles
Length of output: 106
🏁 Script executed:
# Read the file around lines 157-159 to see the current state
sed -n '150,170p' .github/workflows/e2e-dispatch.ymlRepository: norrietaylor/spectacles
Length of output: 1041
🏁 Script executed:
# Search for where staging_repo is emitted/generated
grep -n "staging_repo" .github/workflows/e2e-dispatch.ymlRepository: norrietaylor/spectacles
Length of output: 666
🏁 Script executed:
# Search for where STAGING_REPO is set
grep -n "STAGING_REPO" .github/workflows/e2e-dispatch.ymlRepository: norrietaylor/spectacles
Length of output: 406
🏁 Script executed:
# Look for the gate step to understand the context
grep -n "steps.gate" .github/workflows/e2e-dispatch.ymlRepository: norrietaylor/spectacles
Length of output: 1696
🏁 Script executed:
# Read the gate step to see what format STAGING_REPO has
sed -n '70,130p' .github/workflows/e2e-dispatch.ymlRepository: norrietaylor/spectacles
Length of output: 2777
🏁 Script executed:
# Check if there's any documentation or examples of what SPECTACLES_E2E_STAGING_REPO should be
grep -r "SPECTACLES_E2E_STAGING_REPO" . --include="*.md" --include="*.yml"Repository: norrietaylor/spectacles
Length of output: 379
🏁 Script executed:
# Look for the create-github-app-token action to understand requirements
grep -B5 -A15 "create-github-app-token" .github/workflows/e2e-dispatch.ymlRepository: norrietaylor/spectacles
Length of output: 932
🏁 Script executed:
# Check the e2e.md documentation to understand the format expectations
sed -n '1,50p' ./docs/sdd/e2e.md | grep -A5 -B5 "SPECTACLES_E2E_STAGING_REPO"Repository: norrietaylor/spectacles
Length of output: 49
🏁 Script executed:
# Search for more context about the STAGING_REPO usage in other parts of the workflow
grep -n "permission" .github/workflows/e2e-dispatch.yml | head -20Repository: norrietaylor/spectacles
Length of output: 465
🏁 Script executed:
# Check if there are any other workflows that use create-github-app-token to see patterns
grep -r "create-github-app-token" .github/workflows/ -A10 | grep -A10 "repositories:"Repository: norrietaylor/spectacles
Length of output: 41626
Extract the bare staging repository name for the App-token action.
The staging_repo output is currently emitted in owner/name format, but the create-github-app-token action's repositories field requires bare repo names (see the workflow comment on lines 153-154). Passing owner/name instead of just the repo name prevents the minted token from covering the staging repository.
Proposed fix
emit should_run true
emit scenario "$scenario"
emit spectacles_ref "$spectacles_ref"
emit origin_issue "$origin_issue"
emit staging_repo "$STAGING_REPO"
+ emit staging_repo_name "${STAGING_REPO#*/}"
echo "e2e-dispatch: scenario=$scenario ref=$spectacles_ref staging=$STAGING_REPO" repositories: |
${{ github.event.repository.name }}
- ${{ steps.gate.outputs.staging_repo }}
+ ${{ steps.gate.outputs.staging_repo_name }}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| repositories: | | |
| ${{ github.event.repository.name }} | |
| ${{ steps.gate.outputs.staging_repo }} | |
| repositories: | | |
| ${{ github.event.repository.name }} | |
| ${{ steps.gate.outputs.staging_repo_name }} |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.github/workflows/e2e-dispatch.yml around lines 157 - 159, The staging_repo
output from steps.gate is currently in owner/name format, but the
create-github-app-token action's repositories field requires only the bare
repository name. Extract just the repository name portion (the part after the
forward slash) from the ${{ steps.gate.outputs.staging_repo }} output before
passing it to the repositories field. This can be done by using string
manipulation to split on the "/" character and take only the second part, or by
modifying the gate step to output both the full owner/name format and a separate
bare repository name value that can be referenced here.
| - name: Reset staging (close stale E2E issues) | ||
| if: ${{ steps.gate.outputs.should_run == 'true' }} | ||
| env: | ||
| GH_TOKEN: ${{ steps.app-token.outputs.token }} | ||
| STAGING: ${{ steps.gate.outputs.staging_repo }} | ||
| run: | | ||
| set -euo pipefail | ||
| # Close any open tracking issue from a prior run so this run starts | ||
| # from a clean slate. Idempotent; a fresh staging repo closes nothing. | ||
| gh issue list --repo "$STAGING" --state open --label e2e:seed --json number \ | ||
| --jq '.[].number' | while read -r n; do | ||
| [ -n "$n" ] && gh issue close "$n" --repo "$STAGING" --comment "Superseded by a new E2E run." | ||
| done |
There was a problem hiding this comment.
Reset persisted staging artifacts, not just stale issues.
The reset step only closes prior e2e:seed issues. Since scripts/e2e-assert.py checks artifact globs against the entire default branch, old docs/specs/**/*.md files can satisfy the current run’s artifact assertions and mask a regression.
Reset the staging default branch to a known baseline, or make artifact assertions scoped to files produced by this seeded run. Based on the provided assertion contract, check_artifacts() scans the default branch without using --since.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.github/workflows/e2e-dispatch.yml around lines 176 - 188, The reset step
only closes prior e2e:seed issues but leaves old artifact files on the staging
default branch, allowing scripts/e2e-assert.py to find and validate artifacts
from previous runs instead of the current one, which masks regressions. Add
logic to either reset the staging repository default branch to a clean baseline
by removing old artifact files (particularly docs/specs/**/*.md) before the E2E
run begins, or alternatively modify the artifact assertion logic in
scripts/e2e-assert.py to scope checks only to artifacts produced by the current
seeded run instead of scanning the entire default branch history.
| spectacles ref under test. The staging wrappers call the hosted locks via | ||
| `uses: …@<ref>`, so the staging run exercises the PR's code — no lock push. | ||
| 3. It seeds a synthetic tracking issue from | ||
| `tests/fixtures/e2e/<scenario>/issue.md` (≤ 50 tokens to bound cost), with |
There was a problem hiding this comment.
Token bound inconsistency between documentation and PR objectives.
The documentation states issue.md should be "≤ 50 tokens", but the PR objectives describe fixtures as "bounded to ~30 tokens per fixture to control LLM costs." While this discrepancy is minor, clarifying the intended token budget would prevent future confusion when creating additional scenario fixtures.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/sdd/e2e.md` at line 17, The token budget limit for the issue.md file in
the documentation is stated as "≤ 50 tokens" but conflicts with the PR
objectives which specify "~30 tokens per fixture". Update the documentation in
the e2e.md file at the location describing
`tests/fixtures/e2e/<scenario>/issue.md` to align the token limit with the
actual constraint (either change it to "~30 tokens" or clarify the correct
intended boundary) to ensure consistency with the stated PR objectives.
| gh api -X DELETE "repos/$staging/branches/$default_branch/protection" >/dev/null 2>&1 \ | ||
| || echo "e2e-setup-staging: no branch protection to clear (ok)" |
There was a problem hiding this comment.
Only ignore the “no branch protection” case.
Line 94 currently swallows every gh api failure, including auth/permission/API errors, and continues as if protection was absent. That can leave staging protected and make the later --direct install fail in a less obvious place.
Proposed fix
- gh api -X DELETE "repos/$staging/branches/$default_branch/protection" >/dev/null 2>&1 \
- || echo "e2e-setup-staging: no branch protection to clear (ok)"
+ if ! out="$(gh api -X DELETE "repos/$staging/branches/$default_branch/protection" 2>&1)"; then
+ if printf '%s\n' "$out" | grep -q 'HTTP 404'; then
+ echo "e2e-setup-staging: no branch protection to clear (ok)"
+ else
+ printf '%s\n' "$out" >&2
+ echo "error: failed to clear branch protection on $staging@$default_branch" >&2
+ exit 1
+ fi
+ fi
fi📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| gh api -X DELETE "repos/$staging/branches/$default_branch/protection" >/dev/null 2>&1 \ | |
| || echo "e2e-setup-staging: no branch protection to clear (ok)" | |
| if ! out="$(gh api -X DELETE "repos/$staging/branches/$default_branch/protection" 2>&1)"; then | |
| if printf '%s\n' "$out" | grep -q 'HTTP 404'; then | |
| echo "e2e-setup-staging: no branch protection to clear (ok)" | |
| else | |
| printf '%s\n' "$out" >&2 | |
| echo "error: failed to clear branch protection on $staging@$default_branch" >&2 | |
| exit 1 | |
| fi | |
| fi |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@scripts/e2e-setup-staging.sh` around lines 94 - 95, The current error
handling on lines 94-95 suppresses all failures from the gh api command
indiscriminately, including auth, permission, and API errors, which can hide
real issues and leave branch protection in place. Instead of using the broad ||
operator that ignores any gh api failure, capture the actual error output from
the gh api DELETE command and only suppress the specific case where branch
protection doesn't exist, while allowing auth failures, permission errors, and
other API errors to propagate and fail the script appropriately. This will
ensure that later operations like the --direct install fail in the right place
with clear error messages rather than silently continuing with protected
branches.
| set_secret() { | ||
| local name="$1" value="$2" | ||
| if [ -z "$value" ]; then | ||
| echo "e2e-setup-staging: WARNING $name not in environment; set it manually" | ||
| return | ||
| fi | ||
| run gh secret set "$name" --repo "$staging" --body "$value" | ||
| } | ||
| echo "e2e-setup-staging: setting secrets from the environment" | ||
| set_secret APP_PRIVATE_KEY "${STAGING_APP_PRIVATE_KEY:-}" | ||
| set_secret ANTHROPIC_API_KEY "${ANTHROPIC_API_KEY:-}" |
There was a problem hiding this comment.
Don’t print or silently skip required secrets.
In dry-run mode, run gh secret set ... --body "$value" echoes the private key/API key. Also, the non-dry-run path reports success even when the required APP_PRIVATE_KEY or ANTHROPIC_API_KEY is missing.
Proposed fix
set_secret() {
local name="$1" value="$2"
if [ -z "$value" ]; then
- echo "e2e-setup-staging: WARNING $name not in environment; set it manually"
- return
+ echo "error: required secret source for $name is missing from the environment" >&2
+ return 1
+ fi
+ if [ "$dry_run" -eq 1 ]; then
+ echo "DRY-RUN: gh secret set $name --repo $staging --body <redacted>"
+ return 0
fi
- run gh secret set "$name" --repo "$staging" --body "$value"
+ gh secret set "$name" --repo "$staging" --body "$value"
}
echo "e2e-setup-staging: setting secrets from the environment"
set_secret APP_PRIVATE_KEY "${STAGING_APP_PRIVATE_KEY:-}"
set_secret ANTHROPIC_API_KEY "${ANTHROPIC_API_KEY:-}"🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@scripts/e2e-setup-staging.sh` around lines 110 - 120, The set_secret function
currently silently skips missing secrets with just a warning, but required
secrets like APP_PRIVATE_KEY and ANTHROPIC_API_KEY should cause the script to
fail with an error instead. Additionally, when the run command executes in
dry-run mode, it echoes the full gh secret set command including the secret
value in the --body parameter, exposing sensitive information. Modify the
set_secret function to differentiate between required and optional secrets,
exiting with an error for required ones that are missing, and refactor the gh
secret set command execution to avoid echoing the secret value in dry-run mode,
either by passing the secret via stdin instead of --body parameter, or by using
a different approach that doesn't expose the sensitive value in the command
output.
1 participant
Closes #89 (walking-skeleton layer; remaining scenarios are follow-up fixture-only PRs).
What
A non-agentic dispatcher that drives one synthetic feature through the full SDD lifecycle on a dedicated staging repo, asserts terminal state, and reports duration + a coarse cost estimate. Runs on
/e2e [scenario]from write-access authors, nightly, andworkflow_dispatch.Files
.github/workflows/e2e-dispatch.ymlscripts/e2e-assert.pygh api, checksexpectations.yml.scripts/e2e-setup-staging.shtests/fixtures/e2e/happy-path-feature/{issue.md,expectations.yml}docs/sdd/e2e.md(+mkdocs.ymlnav)Divergences from the issue text (intentional)
.yml, not.md+ lock. Non-agentic orchestrators in this repo are hand-written Actions YAML (lint.yml,sdd-unspecced-scan.yml). A gh-aw.mdsource always injects an agent step, so it cannot be "no gh-aw agent step."quick-setup.sh --ref <pr-head-sha> --directre-pins the staging wrappers; cross-repouses: …@<sha>then runs the PR's hosted lock. Nogh aw deploy, no literal lock copy.Verification
/e2eis deliberately invisible totest-command-table.py— it is a plain-workflow command, not an SDD slash command.Not in this PR
spectacles-stagingrepo, App install, and secrets. Runscripts/e2e-setup-staging.shonce, setSPECTACLES_E2E_STAGING_REPO, then/e2e happy-path-featureon a throwaway PR.happy-path-bug,revise-loop-spec,needs-human-handoffadd a fixture directory each, no dispatcher change.🤖 Generated with Claude Code