Skip to content

Commit 9fc686b

Browse files
claudeclaude
committed
feat: add factory-approve action for auto-approval of trivial PRs
Label-gated pipeline that auto-approves trivial PRs. Deterministic safety gates run first; only when every gate passes do two independent Claude reviewers (the second adversarial) judge the diff, and only unanimous approval makes the factory account post an approving review locked to the reviewed commit. Fails closed everywhere, never requests changes, never merges. Re-runs are cheap: a content fingerprint (insensitive only to hunk line numbers) skips re-reviewing unchanged diffs, and an active human review stands the pipeline down entirely. Includes a backtest CLI that replays the pipeline against recent PRs without posting anything. The built-in policy is a generic org-wide baseline; consuming repositories tune it through the optional `policy` input, a strictly validated JSON overrides document that can tighten anything but only loosen what is explicitly loosenable (hard numeric ceilings, immutable core deny globs and risky-content patterns, fail-closed on any invalid value). Design: factory-approve/docs/policy-overrides-spec.md. Co-Authored-By: Claude Fable 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01Qx1PzGEJfijvERu3LMBCMD
1 parent cc007ef commit 9fc686b

19 files changed

Lines changed: 3192 additions & 1 deletion

‎factory-approve/README.md‎

Lines changed: 117 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,117 @@
1+
# Factory approve — auto-approval for trivial PRs
2+
3+
Auto-approves very simple PRs (copy changes, styling, small self-contained tweaks) so they
4+
don't need another engineer's review. Deterministic safety gates run first; only if they all
5+
pass does Claude judge the diff, and only a unanimous `approve` makes the `apify-factory`
6+
account post an approving review. It never requests changes and never merges.
7+
8+
## How to use
9+
10+
Add the `factory-approve` label to a PR against `develop` (you can add it on a draft — it
11+
waits until the PR is ready). The label is a human opt-in flag the bot never touches: while
12+
it's on, every push is re-reviewed; remove it to opt out.
13+
14+
- **Approve** → `apify-factory` posts an approving review, locked to the reviewed commit.
15+
- **Reject / error** → `apify-factory` posts the report as a new PR comment and any stale
16+
factory approval is dismissed. The label stays on, so the next push re-reviews.
17+
18+
Two situations make a run stand down silently (no review, no comment, no LLM cost):
19+
20+
- **A human review is active** — an approval means the factory has nothing to add; a
21+
changes-requested means a human owns the review conversation now.
22+
- **The content is unchanged since the last factory verdict** — the diff (with hunk line
23+
numbers normalized away) and title are fingerprinted into each posted verdict, so
24+
develop-syncs, rebases, and empty pushes don't trigger a paid re-review. Any real content
25+
change (including whitespace) produces a new fingerprint and a full review, and policy
26+
changes invalidate all stored fingerprints.
27+
28+
Every outcome is rendered with the same fixed template (`scripts/report.mts`): status, the
29+
reviewer's one-sentence reason, a gates/reviewers result table with a link to the run, and —
30+
for non-approvals — a collapsed "Details and next steps" section with the rejecting
31+
reviewer's full explanation. Each run folds the previous report comment as outdated instead
32+
of editing or deleting it, so the timeline stays clean and the history stays honest. The bot
33+
never edits the PR description.
34+
35+
## How it works
36+
37+
1. **Static gates** (`scripts/prepare_review.mts`) — all must pass or the PR is rejected
38+
without calling Claude: trusted author + trigger, open & mergeable, targets `develop`,
39+
≤5 files / ≤100 lines, JS/TS only, no denied paths, no risky added lines, Conventional
40+
Commit title.
41+
2. **LLM verdict** (`anthropics/claude-code-action`, run twice — two independent reviewers,
42+
the second adversarial; both must approve). Each judges whether the change needs a human
43+
(databases, security, money, config, public contracts, infra, privacy) and is free of
44+
correctness bugs, then writes a strict `{"verdict","reason"}` file. The model can only read
45+
the code and write its verdict — it cannot touch the PR.
46+
3. **Post** (`scripts/post_verdict.mts`) — the only place GitHub is written to. Approves as
47+
`apify-factory` (with a separate token), or dismisses stale approvals and posts the report
48+
as a new comment (folding older report comments as outdated). Fails closed: any crash,
49+
missing/invalid verdict, or unknown state → no approval.
50+
51+
## Configure
52+
53+
The built-in policy (`scripts/policy.mts`) is the generic org-wide baseline: base `develop`,
54+
≤5 files / ≤100 lines, `.js`/`.ts` only (no `.json`), modifications plus added test files,
55+
Conventional Commit titles, authors and actors from `apify/product-engineering`, and two
56+
reviewers (`claude-sonnet-5` + `claude-opus-4-8`, the second adversarial).
57+
58+
A consuming repository tunes it through the optional `policy` input — a JSON document of
59+
overrides in the workflow file:
60+
61+
```yaml
62+
- uses: apify/actions/factory-approve@v1
63+
with:
64+
# ...tokens...
65+
policy: |
66+
{
67+
"baseBranch": "main",
68+
"denyGlobs": ["infra/**", "**/billing/**"],
69+
"authorGate": { "teamSlugs": ["tooling"] }
70+
}
71+
```
72+
73+
Overrides can tighten anything, but can only loosen what is explicitly loosenable:
74+
75+
- **Replaceable**: `label`, `factoryLogin`, `baseBranch`, `allowedExtensions`,
76+
`allowedAddedFileGlobs`, `prTitleRegex` (as a string), `authorGate.org`, `authorGate.teamSlugs`,
77+
`authorGate.extraUsers`, `llm.reviewerModels` (1–2 models; the last is adversarial), and
78+
`denyGlobs` — the repo tier only. A core tier of supply-chain paths (`.github/**`, dependency
79+
manifests, lockfiles, env files, Dockerfiles, migrations, secrets) is always kept.
80+
- **Clamped**: `maxChangedFiles` (≤10), `maxChangedLines` (≤300), `llm.maxTurns` (≤50),
81+
`llm.maxDiffChars` (≤120000), `llm.maxReasonChars` (≤600), `llm.maxDetailsChars` (≤4000).
82+
Values above a ceiling are config errors, not silent clamps.
83+
- **Append-only**: `denyGlobsAdd`, `riskyContentPatternsAdd` (`{ id, description, regex }`, regex
84+
as a string), `authorGate.deniedUsersAdd`. The built-in patterns and denied accounts can never
85+
be removed, and `factoryLogin` is always denied.
86+
87+
Everything else — the static check set, unanimity, fail-closed semantics, the report format, the
88+
comment lifecycle — is not configurable. Unknown keys, wrong types, or out-of-range values fail
89+
closed: the run reports "could not finish" and approves nothing, at zero LLM cost. Changing
90+
overrides also changes the review fingerprint, so previously memoized verdicts get a fresh
91+
review. Full design rationale: [docs/policy-overrides-spec.md](docs/policy-overrides-spec.md).
92+
93+
The reviewer's instructions (what needs a human vs. what's approvable) live in
94+
`scripts/prompt.mts` and are deliberately not overridable.
95+
96+
## Setup
97+
98+
1. **Secrets**: `APIFY_FACTORY_GITHUB_TOKEN` (the `apify-factory` account, `repo` + `read:org`)
99+
and `FACTORY_APPROVE_ANTHROPIC_API_KEY` (Anthropic key).
100+
2. **Label**: create `factory-approve` in the repo.
101+
3. **Branch protection**: confirm one `apify-factory` approval actually makes these PRs
102+
mergeable, and enable "dismiss stale approvals when new commits are pushed".
103+
104+
## Testing
105+
106+
Replay the whole pipeline (static gates + the dual-reviewer LLM step) over recent human PRs before
107+
rolling out — reads only, posts nothing. Requires the `claude` CLI installed and authenticated; run
108+
it from a `develop` checkout so the reviewer's Read/Grep context matches CI:
109+
110+
```bash
111+
GITHUB_TOKEN=… FACTORY_GITHUB_TOKEN=… \
112+
node backtest/backtest.mts --repo apify/apify-core --last 200
113+
```
114+
115+
Add `--output results.jsonl` to record a per-PR line for later inspection, and
116+
`--policy overrides.json` to replay the exact JSON document a repo would pass to the `policy`
117+
input before enabling it.

‎factory-approve/action.yaml‎

Lines changed: 126 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,126 @@
1+
name: Factory approve
2+
description: >-
3+
Label-gated auto-approval for trivial PRs. Deterministic safety gates run first; only when every
4+
gate passes does Claude judge the diff, and only a valid approve verdict makes the factory bot
5+
account post an approving review. Fails closed everywhere, never requests changes, never merges.
6+
7+
# Dependency-free Node scripts — nothing is installed at runtime. Generic defaults live in
8+
# scripts/policy.mts; per-repo tuning goes through the `policy` input.
9+
10+
inputs:
11+
pr-number:
12+
description: Number of the pull request under review.
13+
required: true
14+
actor:
15+
description: User whose action triggered the run (labeler or pusher); pass github.actor.
16+
required: true
17+
github-token:
18+
description: Token used for GitHub API reads (secrets.GITHUB_TOKEN).
19+
required: true
20+
factory-github-token:
21+
description: Token of the bot account that posts approvals (needs repo + read:org).
22+
required: true
23+
anthropic-api-key:
24+
description: Anthropic API key for the claude-code-action verdict step.
25+
required: true
26+
policy:
27+
description: >-
28+
Optional JSON document with per-repo policy overrides (see the README for the allowed keys).
29+
Empty means the built-in defaults. Invalid or out-of-range values fail closed: the run
30+
reports an error verdict and approves nothing.
31+
required: false
32+
default: ''
33+
34+
outputs:
35+
verdict:
36+
description: Final verdict — approve, reject, or error.
37+
value: ${{ steps.post.outputs.verdict }}
38+
39+
runs:
40+
using: composite
41+
steps:
42+
# Never fails the step: crashes are captured into gates.json and surface as an `error` verdict.
43+
- name: Run static safety gates
44+
id: prepare
45+
shell: bash
46+
env:
47+
GITHUB_TOKEN: ${{ inputs.github-token }}
48+
FACTORY_GITHUB_TOKEN: ${{ inputs.factory-github-token }}
49+
POLICY_OVERRIDES: ${{ inputs.policy }}
50+
run: |
51+
node "${{ github.action_path }}/scripts/prepare_review.mts" \
52+
--pr "${{ inputs.pr-number }}" \
53+
--repo "${{ github.repository }}" \
54+
--actor "${{ inputs.actor }}" \
55+
--out-dir "${{ runner.temp }}/factory-approve"
56+
57+
# The model only reads the checkout and writes its own verdict file; --disallowedTools blocks the
58+
# GitHub tools so it cannot touch the PR, and posting happens only in post_verdict.mts. The Edit()
59+
# rule (not Write()) governs the Write tool; the doubled slash marks an absolute path.
60+
# continue-on-error so an LLM outage still reaches the post step (a missing verdict = error).
61+
- name: Judge PR with Claude
62+
if: steps.prepare.outputs.gates_passed == 'true'
63+
continue-on-error: true
64+
uses: anthropics/claude-code-action@v1
65+
with:
66+
anthropic_api_key: ${{ inputs.anthropic-api-key }}
67+
github_token: ${{ inputs.github-token }}
68+
show_full_output: true
69+
prompt: ${{ steps.prepare.outputs.prompt }}
70+
claude_args: |
71+
--model ${{ steps.prepare.outputs.model }}
72+
--max-turns ${{ steps.prepare.outputs.max_turns }}
73+
--add-dir ${{ runner.temp }}/factory-approve
74+
--allowedTools "Read,Glob,Grep,Edit(/${{ runner.temp }}/factory-approve/verdict.json)"
75+
--disallowedTools "mcp__github,mcp__github_comment,mcp__github_inline_comment"
76+
77+
# Both reviewers must approve, so skip the second (more expensive) reviewer when the first did not
78+
# approve. A missing or invalid first verdict counts as not-approved.
79+
- name: Check first reviewer's verdict
80+
id: first
81+
if: steps.prepare.outputs.gates_passed == 'true' && steps.prepare.outputs.prompt2 != ''
82+
shell: bash
83+
env:
84+
VERDICT_FILE: ${{ runner.temp }}/factory-approve/verdict.json
85+
run: |
86+
node -e '
87+
const { readFileSync, appendFileSync } = require("node:fs");
88+
let approved = false;
89+
try { approved = JSON.parse(readFileSync(process.env.VERDICT_FILE, "utf-8")).verdict === "approve"; } catch {}
90+
appendFileSync(process.env.GITHUB_OUTPUT, `approved=${approved}\n`);
91+
console.log(`First reviewer approved: ${approved}`);
92+
'
93+
94+
# Second independent reviewer with an adversarial stance; runs only if the first approved.
95+
- name: Judge PR with Claude (second independent reviewer)
96+
if: steps.prepare.outputs.gates_passed == 'true' && steps.prepare.outputs.prompt2 != '' && steps.first.outputs.approved == 'true'
97+
continue-on-error: true
98+
uses: anthropics/claude-code-action@v1
99+
with:
100+
anthropic_api_key: ${{ inputs.anthropic-api-key }}
101+
github_token: ${{ inputs.github-token }}
102+
show_full_output: true
103+
prompt: ${{ steps.prepare.outputs.prompt2 }}
104+
claude_args: |
105+
--model ${{ steps.prepare.outputs.model2 }}
106+
--max-turns ${{ steps.prepare.outputs.max_turns }}
107+
--add-dir ${{ runner.temp }}/factory-approve
108+
--allowedTools "Read,Glob,Grep,Edit(/${{ runner.temp }}/factory-approve/verdict2.json)"
109+
--disallowedTools "mcp__github,mcp__github_comment,mcp__github_inline_comment"
110+
111+
# `!cancelled()` so the post step still runs when a reviewer step hard-fails (a missing verdict
112+
# aggregates to `error`, which never approves), but is skipped when a newer commit supersedes this
113+
# run (concurrency cancel) to avoid churning the PR-body comment.
114+
- name: Post verdict
115+
id: post
116+
if: ${{ !cancelled() }}
117+
shell: bash
118+
env:
119+
FACTORY_GITHUB_TOKEN: ${{ inputs.factory-github-token }}
120+
POLICY_OVERRIDES: ${{ inputs.policy }}
121+
WORKFLOW_RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
122+
run: |
123+
node "${{ github.action_path }}/scripts/post_verdict.mts" \
124+
--pr "${{ inputs.pr-number }}" \
125+
--repo "${{ github.repository }}" \
126+
--out-dir "${{ runner.temp }}/factory-approve"

0 commit comments

Comments
 (0)