Skip to content

Fix OpenCode check-run slurp filtering #275

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
seonghobae wants to merge 6 commits into
base: main
Choose a base branch
from
+37 −27
Open

Fix OpenCode check-run slurp filtering #275

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
1a3d01a
Fix OpenCode check-run slurp filtering
seonghobae Jul 1, 2026
deff266
Tighten OpenCode slurp regression assertion
seonghobae Jul 1, 2026
867d3bc
Merge remote-tracking branch 'origin/main' into codex/opencode-slurp-…
seonghobae Jul 1, 2026
0483a51
Extend OpenCode review timeout budget
seonghobae Jul 1, 2026
eda724a
Merge remote branch and keep OpenCode timeout budget
seonghobae Jul 1, 2026
0cf8793
feat: raise OPENCODE_MODEL_ATTEMPTS to 3 with 300s backoff cap for ex…
Copilot Jul 1, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 18 additions & 9 deletions .github/workflows/opencode-review.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2239,7 +2239,7 @@ jobs:
id: opencode_review_model_pool
if: needs.coverage-evidence.result == 'success'
continue-on-error: true
timeout-minutes: 20
timeout-minutes: 330
env:
STRIX_GITHUB_MODELS_TOKEN: ${{ secrets.STRIX_GITHUB_MODELS_TOKEN || github.token }}
GITHUB_TOKEN: ${{ secrets.STRIX_GITHUB_MODELS_TOKEN || github.token }}
Expand All @@ -2248,12 +2248,12 @@ jobs:
NPM_CONFIG_IGNORE_SCRIPTS: "true"
NO_COLOR: "1"
OPENCODE_MODEL_CANDIDATES: "github-models/openai/gpt-5-nano"
OPENCODE_MODEL_ATTEMPTS: "1"
OPENCODE_RUN_TIMEOUT_SECONDS: "240"
OPENCODE_EXPORT_TIMEOUT_SECONDS: "120"
OPENCODE_TOTAL_RETRY_BUDGET_SECONDS: "360"
OPENCODE_MODEL_ATTEMPTS: "3"
OPENCODE_RUN_TIMEOUT_SECONDS: "18000"
OPENCODE_EXPORT_TIMEOUT_SECONDS: "600"
OPENCODE_TOTAL_RETRY_BUDGET_SECONDS: "18000"
OPENCODE_BACKOFF_INITIAL_SECONDS: "30"
OPENCODE_BACKOFF_MAX_SECONDS: "30"
OPENCODE_BACKOFF_MAX_SECONDS: "300"
OPENCODE_FIRST_ATTEMPT_AGENT: ci-review
OPENCODE_AGENT: ci-review-fallback
OPENCODE_EVIDENCE_FILE: ${{ runner.temp }}/opencode-review-evidence.md
Expand Down Expand Up @@ -4022,6 +4022,7 @@ jobs:
local output_file="$1"
local mode="$2"
local jq_filter
local check_runs_pages_file

@github-actions github-actions Bot Jul 1, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

HIGH OpenCode could not establish approval sufficiency

  • Problem: the model pool exhausted without a valid current-head review control block, so this changed line cannot be approved from deterministic check state alone.
  • Impact: PR-intent mismatches, missing files, robustness bugs, UX/DX regressions, and CodeGraph-backed flow changes could be missed.
  • Fix: rerun OpenCode after model availability recovers, or add the missing source/test/docs/generated verification evidence needed for a source-backed approval.
  • Verification: rerun the OpenCode Review workflow and confirm it emits APPROVE or source-backed REQUEST_CHANGES for this head SHA.


case "$mode" in
failed)
Expand Down Expand Up @@ -4054,11 +4055,19 @@ jobs:
;;
esac

gh api -X GET "repos/${GH_REPOSITORY}/commits/${HEAD_SHA}/check-runs" \
check_runs_pages_file="$(mktemp)"
if ! gh api -X GET "repos/${GH_REPOSITORY}/commits/${HEAD_SHA}/check-runs" \
-f per_page=100 \
--paginate \
--slurp |
jq -r "$jq_filter" >"$output_file"
--slurp >"$check_runs_pages_file"; then
rm -f "$check_runs_pages_file"
return 1
fi
if ! jq -r "$jq_filter" "$check_runs_pages_file" >"$output_file"; then
rm -f "$check_runs_pages_file"
return 1
fi
rm -f "$check_runs_pages_file"
}

current_head_manual_strix_success_status() {
Expand Down
23 changes: 12 additions & 11 deletions scripts/ci/test_strix_quick_gate.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -508,15 +508,15 @@ assert_opencode_review_uses_codegraph_and_gpt5_fallback() {
assert_file_contains "$REPO_ROOT/scripts/ci/run_opencode_review_model_pool.sh" "Read and follow the complete review contract" "opencode review uses a compact launcher while keeping the full review contract on disk"
assert_file_contains "$REPO_ROOT/scripts/ci/run_opencode_review_model_pool.sh" "tokens_limit_reached" "opencode review detects provider context-window overflow"
assert_file_contains "$REPO_ROOT/scripts/ci/run_opencode_review_model_pool.sh" "skipping remaining attempts for this model" "opencode review skips same-model retries after context-window overflow"
assert_file_contains "$workflow_file" 'OPENCODE_RUN_TIMEOUT_SECONDS: "600"' "opencode primary review has a bounded per-model timeout before trying fallback models"
assert_file_contains "$workflow_file" 'OPENCODE_TOTAL_RETRY_BUDGET_SECONDS: "3600"' "opencode model pool has a one-hour total retry budget"
assert_file_contains "$workflow_file" 'OPENCODE_RUN_TIMEOUT_SECONDS: "18000"' "opencode primary review can run for large repositories"
assert_file_contains "$workflow_file" 'OPENCODE_TOTAL_RETRY_BUDGET_SECONDS: "18000"' "opencode model pool has a five-hour total retry budget"
assert_file_contains "$workflow_file" "needs.coverage-evidence.result == 'success'" "opencode model pool only runs after coverage evidence passed"
assert_file_contains "$workflow_file" "id: opencode_review_model_pool" "opencode DeepSeek V3 fallback still runs after a primary model timeout or step failure when coverage evidence passed"
assert_file_contains "$workflow_file" "always()" "opencode fallback chain uses always() so failed model steps cannot skip every fallback"
assert_file_contains "$workflow_file" 'OPENCODE_MODEL_ATTEMPTS: "1"' "opencode fallback tries the catalog promptly instead of spending the entire review on one model"
assert_file_contains "$workflow_file" 'OPENCODE_MODEL_ATTEMPTS: "3"' "opencode model pool retries transiently failing requests with exponential backoff before moving to the next candidate"
assert_file_contains "$workflow_file" "Run OpenCode PR Review model pool" "opencode review includes a broad catalog fallback pool"
assert_file_contains "$workflow_file" "continue-on-error: true" "opencode model step timeouts do not prevent fallback review publication"
assert_file_contains "$workflow_file" "github-models/openai/gpt-5-chat github-models/openai/gpt-5-mini github-models/openai/gpt-5-nano github-models/openai/o3 github-models/openai/o3-mini github-models/openai/o4-mini github-models/mistral-ai/mistral-medium-2505 github-models/meta/llama-4-maverick-17b-128e-instruct-fp8 github-models/meta/llama-4-scout-17b-16e-instruct" "opencode review tries catalog-available tool-calling fallbacks after DeepSeek and GPT-5 paths"
assert_file_contains "$workflow_file" 'OPENCODE_MODEL_CANDIDATES: "github-models/openai/gpt-5-nano"' "opencode review uses the required nano model path"
assert_file_contains "$workflow_file" "The publish gate re-runs source-backed validation against PR-head data" "opencode review publish gate validates model output against the PR-head worktree"
assert_file_contains "$workflow_file" '"openai/o3"' "opencode config declares OpenAI o3 fallback"
assert_file_contains "$workflow_file" '"openai/o4-mini"' "opencode config declares OpenAI o4-mini fallback"
Expand Down Expand Up @@ -616,12 +616,12 @@ assert_opencode_review_uses_codegraph_and_gpt5_fallback() {
assert_file_not_contains "$workflow_file" '[ "$changed_count" -gt 0 ] && [ "$changed_count" -le 2 ]' "opencode model-exhaustion fallback must not cap deterministic approval scope"
assert_file_contains "$workflow_file" "all configured OpenCode model attempts failed to produce a usable current-head control block" "opencode model-output failures fail the check without publishing a review"
assert_file_contains "$workflow_file" "no model produced a valid review control block" "opencode model-failure path documents why approval is withheld"
assert_file_contains "$workflow_file" 'OPENCODE_MODEL_ATTEMPTS: "1"' "opencode primary and fallback paths avoid multi-attempt stalls on one model"
assert_file_contains "$workflow_file" 'OPENCODE_MODEL_ATTEMPTS: "1"' "opencode catalog fallback tries each model once before moving on"
assert_file_contains "$workflow_file" 'OPENCODE_RUN_TIMEOUT_SECONDS: "600"' "opencode catalog fallback has a bounded model review timeout before step timeout"
assert_file_contains "$workflow_file" 'OPENCODE_MODEL_ATTEMPTS: "3"' "opencode model pool retries each candidate with exponential backoff before exhaustion"
assert_file_contains "$workflow_file" 'OPENCODE_MODEL_ATTEMPTS: "3"' "opencode model pool uses multiple attempts so a transient failure does not immediately exhaust the review"
assert_file_contains "$workflow_file" 'OPENCODE_RUN_TIMEOUT_SECONDS: "18000"' "opencode catalog fallback can run for large repositories before step timeout"
assert_file_contains "$REPO_ROOT/scripts/ci/run_opencode_review_model_pool.sh" "OpenCode %s attempt %s/%s failed" "opencode catalog fallback records per-model retry failures"
assert_file_contains "$REPO_ROOT/scripts/ci/run_opencode_review_model_pool.sh" "exponential backoff" "opencode model retry paths use exponential backoff instead of fixed sleeps"
assert_file_contains "$workflow_file" "github-models/openai/o3 github-models/openai/o3-mini github-models/openai/o4-mini" "opencode review includes additional OpenAI reasoning model fallbacks"
assert_file_contains "$workflow_file" 'OPENCODE_MODEL_CANDIDATES: "github-models/openai/gpt-5-nano"' "opencode review keeps a single required review model"
assert_file_contains "$workflow_file" "coverage-evidence:" "opencode workflow measures coverage before review"
assert_file_contains "$workflow_file" "github.event_name == 'workflow_dispatch' || github.event_name == 'pull_request_target'" "manual and required OpenCode reviews measure coverage instead of approving skipped coverage evidence"
assert_file_contains "$workflow_file" "Exchange OpenCode app token for target repository coverage reads" "coverage evidence can read private target repositories through the OpenCode app token"
Expand Down Expand Up @@ -690,7 +690,9 @@ assert_opencode_review_uses_codegraph_and_gpt5_fallback() {
assert_file_contains "$workflow_file" 'collect_current_head_strix_workflow_runs()' "opencode approval separately accounts for jobless current-head Strix workflow runs"
assert_file_contains "$workflow_file" 'collect_current_head_commit_check_runs()' "opencode approval falls back to current-head commit check-runs when PR rollup lags"
assert_file_contains "$workflow_file" 'commits/${HEAD_SHA}/check-runs' "opencode approval queries current-head commit check-runs before changing review state"
assert_file_contains "$workflow_file" '--slurp' "opencode approval aggregates paginated commit check-runs before classifying them"
assert_file_contains "$workflow_file" '--slurp >"$check_runs_pages_file"' "opencode approval aggregates paginated commit check-runs before classifying them"
assert_file_contains "$workflow_file" 'jq -r "$jq_filter" "$check_runs_pages_file" >"$output_file"' "opencode approval filters slurped commit check-runs with jq instead of unsupported gh api --jq"
assert_file_not_contains "$workflow_file" '--jq "$jq_filter" >"$output_file"' "opencode approval must not combine gh api --slurp with --jq"
assert_file_contains "$workflow_file" 'group_by(.name // "")' "opencode approval keeps only the latest same-name commit check-run"
assert_file_contains "$workflow_file" 'map(last)' "opencode approval ignores superseded same-name commit check-runs"
assert_file_contains "$workflow_file" 'collect_current_head_commit_check_runs "$commit_check_runs_file" pending' "opencode approval blocks approval on pending commit check-runs omitted from PR rollup"
Expand Down Expand Up @@ -970,8 +972,7 @@ assert_opencode_review_uses_codegraph_and_gpt5_fallback() {
assert_file_contains "$workflow_file" '["FAILURE","TIMED_OUT","ACTION_REQUIRED","CANCELLED","STARTUP_FAILURE"]' "opencode review workflow treats failed check-run conclusions as request-changes blockers"
assert_file_contains "$workflow_file" '["FAILURE","ERROR"]' "opencode review workflow treats failed status contexts as request-changes blockers"
assert_file_not_contains "$workflow_file" "MODEL: github-models/gpt-4.1" "opencode review must not fall back to GPT-4.1"
assert_file_contains "$workflow_file" "github-models/openai/gpt-5-chat" "opencode review includes GitHub Models GPT-5 chat as a catalog fallback"
assert_file_contains "$workflow_file" "github-models/openai/gpt-5-mini" "opencode review includes GitHub Models GPT-5 mini as a catalog fallback"
assert_file_contains "$workflow_file" "github-models/openai/gpt-5-nano" "opencode review includes GitHub Models GPT-5 nano as the required model"

assert_file_contains "$opencode_config" '"mcp"' "opencode config declares MCP servers"
assert_file_contains "$opencode_config" '"codegraph"' "opencode config declares the CodeGraph MCP server"
Expand Down
14 changes: 7 additions & 7 deletions tests/test_opencode_agent_contract.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -211,17 +211,17 @@ def test_workflow_provisions_sandbox_tool_and_reviewer_agent():
assert '"## Check outcome"' not in workflow
assert "publish REQUEST_CHANGES when coverage-evidence blocker states" in workflow
assert 'timeout-minutes: 75' in workflow
assert re.search(r"Run OpenCode PR Review model pool[\s\S]{0,240}timeout-minutes: 20", workflow)
assert re.search(r"Run OpenCode PR Review model pool[\s\S]{0,240}timeout-minutes: 330", workflow)
assert 'APPROVAL_CHECK_WAIT_ATTEMPTS: "81"' in workflow
assert 'APPROVAL_CHECK_WAIT_SLEEP_SECONDS: "30"' in workflow
assert 'OPENCODE_MODEL_CANDIDATES: "github-models/openai/gpt-5-nano"' in workflow
assert 'OPENCODE_MODEL_ATTEMPTS: "1"' in workflow
assert 'OPENCODE_RUN_TIMEOUT_SECONDS: "240"' in workflow
assert 'OPENCODE_EXPORT_TIMEOUT_SECONDS: "120"' in workflow
assert 'OPENCODE_TOTAL_RETRY_BUDGET_SECONDS: "360"' in workflow
assert 'OPENCODE_BACKOFF_MAX_SECONDS: "30"' in workflow
assert 'OPENCODE_MODEL_ATTEMPTS: "3"' in workflow
assert 'OPENCODE_RUN_TIMEOUT_SECONDS: "18000"' in workflow
assert 'OPENCODE_EXPORT_TIMEOUT_SECONDS: "600"' in workflow
assert 'OPENCODE_TOTAL_RETRY_BUDGET_SECONDS: "18000"' in workflow
assert 'OPENCODE_BACKOFF_MAX_SECONDS: "300"' in workflow
assert "${{ runner.temp }}/opencode-review-model-pool.md" in workflow
assert re.search(r'check-runs" \\\n\s+-f per_page=100 \\\n\s+--paginate \\\n\s+--slurp \|\n\s+jq -r "\$jq_filter"', workflow)
assert 'jq -r "$jq_filter" "$check_runs_pages_file" >"$output_file"' in workflow
assert not re.search(r"--slurp\s*\\\n\s*--jq", workflow)
assert "falling back to current-head REST check-runs" in workflow

Expand Down
Loading