Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,17 @@ Format: [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)

## [Unreleased]

### Added
- Added the local-only `audit pr-evidence <snapshot.json>` operator path with
strict `PRHeadEvidenceV1` input validation and deterministic
`PRHeadEvidenceVerdictV1` output. Reviews retain actor, state, dismissal, and
`commit_id`; checks and suites retain status, conclusion, and `head_sha`;
required approvals/checks come only from supplied rules; latest-push
approval, stale head evidence, non-success conclusions, missing permissions,
incomplete pagination, missing rules, and malformed input remain explicit.
The fixture-first evaluator does not consult credentials or GitHub, write
files, alter portfolio scoring, or regenerate portfolio truth.

### Changed
- Kept PortfolioTruth `0.11.0` and
`GitHubSecurityCoverageReceiptV1` read-compatible while adding normalized
Expand Down
13 changes: 13 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -126,6 +126,7 @@ Treat campaign/writeback, GitHub Projects, Notion sync, catalog overrides, score
- Project history: [docs/project-history.md](docs/project-history.md)
- Contributing: [CONTRIBUTING.md](CONTRIBUTING.md)
- Security policy: [SECURITY.md](SECURITY.md)
- PR evidence-to-head binding: [docs/pr-head-evidence.md](docs/pr-head-evidence.md)

## Features

Expand Down Expand Up @@ -272,6 +273,9 @@ audit run <github-username> --repos <repo-name> --html

# Action Sync — managed campaign preview / writeback
audit report <github-username> --campaign security-review --writeback-target github

# Local PR evidence binding — no token, network request, or output file
audit pr-evidence tests/fixtures/pr_head_evidence/current.json
```

Normal runs perform a lightweight automatic preflight before fetching repos. By default
Expand All @@ -288,6 +292,15 @@ groups work into `Needs Re-Approval`, `Ready For Review`, `Approved But Manual`,
`Blocked`, and writes `approval-center-<username>-<date>.json` plus `.md`. Local approval
capture stays separate from writeback apply.

`audit pr-evidence <snapshot.json>` is a separate local-only evidence check. It
validates a versioned `PRHeadEvidenceV1` snapshot and emits deterministic
`PRHeadEvidenceVerdictV1` JSON showing whether evidence required by the
supplied rules is current for the supplied PR head, while classifying every
supplied review and check. It does not read GitHub credentials, call GitHub,
write files, affect portfolio scoring, or regenerate portfolio truth. See
[docs/pr-head-evidence.md](docs/pr-head-evidence.md) for the input contract,
coverage requirements, exit codes, and claim ceiling.

Watch mode supports `--watch-strategy adaptive|incremental|full`. `adaptive` is the
default and uses the stored baseline contract plus the scheduled full-refresh interval to
decide whether each watch cycle should run full or incremental.
Expand Down
8 changes: 7 additions & 1 deletion docs/audit-cli-migration.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# CLI Migration: flat form → subcommand form

Arc F Sprint 4.3 introduced four subcommands — `run`, `triage`, `report`, and `serve` —
Arc F Sprint 4.3 introduced four workflow subcommands — `run`, `triage`, `report`, and `serve` —
to replace the flat `audit <username> --flag` invocation style. The flat form still
works and emits a deprecation warning; it will not be removed until a future major
version bump.
Expand All @@ -20,6 +20,12 @@ The mapping is:
| `audit report` | Portfolio truth, Excel workbooks, campaigns, writeback, exports |
| `audit serve` | Local FastAPI + HTMX web UI |

Specialized read-only gates can also have standalone subcommands. In
particular, `audit pr-evidence <snapshot.json>` evaluates a local
`PRHeadEvidenceV1` document without a username, credential lookup, network
request, output directory, or legacy flat-form equivalent. See
[pr-head-evidence.md](pr-head-evidence.md).

## Flag family mapping

### audit run
Expand Down
232 changes: 232 additions & 0 deletions docs/pr-head-evidence.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,232 @@
# Pull Request Evidence-to-Head Binding

`audit pr-evidence` is a local-only operator command that answers one narrow
question:

> Is the evidence required by the supplied rules current for the supplied pull
> request head?

It reads a `PRHeadEvidenceV1` JSON snapshot, writes a deterministic
`PRHeadEvidenceVerdictV1` JSON document to standard output, and performs no
GitHub requests or file writes.

```bash
audit pr-evidence tests/fixtures/pr_head_evidence/current.json
```

The command is intentionally separate from portfolio scoring and generated
portfolio truth. It does not regenerate or modify either contract.

## Exit codes

| Code | Meaning |
|---|---|
| `0` | All required evidence is current and successful for the supplied head. |
| `1` | The valid snapshot is stale, blocked, pending, missing, neutral, or skipped. |
| `2` | Coverage is unknown, the snapshot is inaccessible, or the input is malformed. |

JSON is emitted for every evaluable command result, including malformed input.

## `PRHeadEvidenceV1` input

All fields below are required unless they are explicitly nullable. Unknown
fields are rejected so misspellings cannot silently weaken the contract.
Unknown rule *types* are preserved as coverage gaps and force an `unknown`
verdict.

```json
{
"schema_version": "PRHeadEvidenceV1",
"pull_request": {
"repository": "owner/repository",
"number": 42,
"author": "pull-request-author",
"head_sha": "40-or-64-character-full-sha",
"base_sha": "40-or-64-character-full-sha"
},
"latest_reviewable_push": {
"pushed_at": "2026-07-28T10:00:00Z",
"actor": "pusher-login"
},
"reviews": [
{
"id": 1001,
"actor": {
"login": "reviewer-login",
"can_count": true
},
"state": "APPROVED",
"commit_id": "full-reviewed-commit-sha",
"submitted_at": "2026-07-28T10:05:00Z",
"dismissal": {
"dismissed": false,
"dismissed_at": null,
"prior_state": null
}
}
],
"checks": [
{
"id": 2001,
"kind": "check_run",
"name": "ci/test",
"head_sha": "full-checked-commit-sha",
"status": "completed",
"conclusion": "success",
"integration_id": 99
}
],
"rules": {
"source": "branch_protection",
"availability": "available",
"items": [
{
"type": "pull_request",
"required_approving_review_count": 1,
"dismiss_stale_reviews": true,
"require_last_push_approval": true,
"require_code_owner_reviews": false
},
{
"type": "required_status_checks",
"strict_required_status_checks_policy": false,
"checks": [
{
"context": "ci/test",
"integration_id": 99
}
]
}
]
},
"coverage": {
"reviews": {
"permission": "granted",
"complete": true,
"truncated": false,
"pages_fetched": 1
},
"checks": {
"permission": "granted",
"complete": true,
"truncated": false,
"pages_fetched": 1
},
"rules": {
"permission": "granted",
"complete": true,
"truncated": false,
"pages_fetched": 1
}
},
"provenance": {
"source": "synthetic_fixture",
"captured_at": "2026-07-28T10:10:00Z",
"freshness": "current"
}
}
```

`actor.can_count` is a normalized collector assertion that the reviewer is
eligible to count under the supplied repository policy. Use `null` when the
permission or actor eligibility could not be established; that keeps approval
coverage `unknown`. Actor logins are compared case-insensitively, matching
GitHub identity semantics. An approval from `pull_request.author` never counts,
even if a malformed collector assertion sets `can_count` to `true`.

`review.submitted_at` may be `null` only for a `PENDING` review, matching
GitHub's response shape for an unsubmitted review. Submitted reviews require a
timezone-qualified timestamp.

`latest_reviewable_push.pushed_at` and `.actor` are nullable only so a collector
can preserve missing data. If `require_last_push_approval` is true and either is
missing, the verdict is `unknown`.

`rules.source` is `branch_protection`, `ruleset`, or `combined`.
`rules.availability` is `available`, `missing`, or `inaccessible`. A complete,
available, empty rules array means that the supplied rules snapshot establishes
no approval or status-check requirements. Missing rules never receive invented
defaults.

The initial evaluator supports normalized pull-request approval counts,
stale-approval dismissal, latest-push approval, and required status-check
contexts. A code-owner requirement is accepted but yields `unknown` because a
head snapshot alone cannot prove every changed path's ownership groups.

## Binding and outcome are separate

Every review keeps its original `state`, actor, dismissal fields, and
`commit_id`. Every check keeps its kind, status, conclusion, and `head_sha`.
The verdict adds:

- `binding_state`: `current` when the evidence SHA matches the supplied head,
otherwise `stale`;
- a review `decision_state`, including `commented` and `dismissed`;
- a check `result_state`, including `success`, `pending`, `neutral`, `skipped`,
`cancelled`, `blocked`, `stale`, and `unknown`.

GitHub documents that `neutral` and `skipped` can satisfy a required status
check. The verdict exposes that narrow fact as
`github_requirement_satisfied=true`, but its top-level state remains `neutral`
or `skipped`; it is not collapsed into current successful evidence.

Check suites are SHA-bound and classified, but they do not masquerade as a
required status-check context. GitHub branch rules select checks or commit
statuses, not suite summaries.

When stale-review dismissal is disabled, GitHub can retain an approval after a
push. The verdict records that as a counted retained approval while keeping its
binding `stale`. If a requirement depends on retained stale approval, the
top-level evidence verdict is `stale`, not green.

## Coverage rules

The following conditions force `coverage.state=unknown`:

- review, check, or rule permission is `missing`, `inaccessible`, or `unknown`;
- pagination is incomplete or truncated;
- protection/ruleset data is missing or inaccessible;
- a rule type is unknown;
- reviewer counting eligibility is unknown and can change the result;
- a code-owner review requirement is present;
- provenance freshness is unknown.

A stale provenance assertion yields a `stale` verdict. No implicit freshness
window is invented.

## Supported sources

The first implementation is fixture-first. A snapshot may be produced by a
synthetic fixture, an operator export, or a separate read-only GitHub collector,
but the collector must normalize the same fields and state completeness
honestly.

The semantics track GitHub's primary documentation:

- [REST pull request reviews](https://docs.github.com/en/rest/pulls/reviews)
exposes review `state` and `commit_id`.
- [Protected branches](https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-protected-branches/about-protected-branches)
defines stale approval dismissal and approval of the most recent reviewable
push.
- [REST check runs](https://docs.github.com/en/rest/checks/runs) and
[REST check suites](https://docs.github.com/en/rest/checks/suites) bind check
evidence to `head_sha`.
- [Troubleshooting required status checks](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/collaborating-on-repositories-with-code-quality-features/troubleshooting-required-status-checks)
states that required checks apply to the latest commit SHA and describes
successful, neutral, and skipped conclusions.
- [REST repository rules](https://docs.github.com/en/rest/repos/rules) describes
normalized pull-request and required-status-check rule parameters.

## Claim ceiling

The command does **not** claim that GitHub would merge the pull request, that
the code is correct, that reviews were effective, that CI is trustworthy, or
that the branch is safe. Its top-level state evaluates required evidence under
the supplied rules. It still classifies every supplied review and check against
the supplied head so historical or non-required evidence cannot masquerade as
current evidence.

Synthetic fixture triplets live in
`tests/fixtures/pr_head_evidence/semantic_triplets.json`. Direct CLI smoke
snapshots for current, stale, comment-only, dismissed, incomplete, and malformed
cases live beside it.
54 changes: 54 additions & 0 deletions src/app/pr_head_evidence.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
"""CLI adapter for local PR head-evidence snapshots."""

from __future__ import annotations

import json
from pathlib import Path
from typing import Protocol

from src.pr_head_evidence import (
SnapshotValidationError,
evaluate_snapshot,
invalid_snapshot_verdict,
parse_snapshot,
)


class PRHeadEvidenceArgs(Protocol):
snapshot: str


def run_pr_head_evidence_mode(args: PRHeadEvidenceArgs) -> None:
"""Evaluate one local snapshot and emit deterministic JSON to stdout."""

path = Path(args.snapshot)
try:
raw_text = path.read_text(encoding="utf-8")
except OSError as exc:
Comment on lines +25 to +27

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Emit the versioned error verdict for invalid UTF-8

If the snapshot contains invalid UTF-8 bytes, Path.read_text() raises UnicodeDecodeError, which is not an OSError, so the command leaks a traceback and emits no JSON instead of the promised deterministic malformed-input verdict with exit code 2. Catch decoding failures here and route them through invalid_snapshot_verdict just like unreadable or syntactically invalid JSON.

AGENTS.md reference: src/AGENTS.md:L17-L19

Useful? React with 👍 / 👎.

verdict = invalid_snapshot_verdict(
"snapshot_unreadable",
[f"{exc.__class__.__name__}: {exc}"],
)
print(json.dumps(verdict, indent=2, sort_keys=True))
raise SystemExit(2) from None
try:
value = json.loads(raw_text)
except json.JSONDecodeError as exc:
verdict = invalid_snapshot_verdict(
"snapshot_invalid_json",
[f"line {exc.lineno}, column {exc.colno}: {exc.msg}"],
)
print(json.dumps(verdict, indent=2, sort_keys=True))
raise SystemExit(2) from None
try:
snapshot = parse_snapshot(value)
except SnapshotValidationError as exc:
verdict = invalid_snapshot_verdict("snapshot_malformed", exc.errors)
print(json.dumps(verdict, indent=2, sort_keys=True))
raise SystemExit(2) from None
verdict = evaluate_snapshot(snapshot)
print(json.dumps(verdict, indent=2, sort_keys=True))
if verdict["state"] == "unknown":
raise SystemExit(2)
if not verdict["current"]:
raise SystemExit(1)
Loading