Skip to content

feat: add fail-closed prospective maintainer study runner #12

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
VrtxOmega merged 1 commit into from
Jul 14, 2026
Merged

feat: add fail-closed prospective maintainer study runner #12

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
201 changes: 201 additions & 0 deletions docs/PROSPECTIVE_STUDY.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,201 @@
# Prospective Maintainer Study Runner

`pcf study` is a local, fail-closed operator for the frozen independent-
maintainer evaluation protocol. It keeps consent and raw snapshots separate,
locks case selection before PCF evaluation, hides PCF output from raters, and
builds privacy-reduced aggregate receipts through `validate-corpus`.

It does not contact GitHub, send invitations, register a study, promise
compensation, verify institutional compliance, or establish product validity.

## Modes

- `production` is the default. Initialization is allowed for method review, but
consent, issue observation, and case freezing stop unless the activated
protocol contains `status.outreachAuthorized=true` and
`snapshot.restrictedSnapshotStoreCreated=true`.
- `synthetic` supports end-to-end rehearsal. Every consent record must contain
`synthetic=true`. Synthetic output is never participant evidence.

Production mutations use the system clock. Caller-supplied `--now` values are
accepted only in synthetic mode for deterministic testing, and every consent,
observation, freeze, rating, withdrawal, and lock transition must remain in
monotonic time order.

## Private store

The study root must be an explicit absolute path and cannot contain symbolic
links. Directories are created with mode `0700`; files use `0600`. Every read
fails closed if those owner-only permissions drift.

```text
<root>/
config/
protocol.json
sampling-frame.json
study.json
restricted/
participants.json
observations.json
cases.json
snapshots/<case>.json
ratings/<case>/<rater-position>.json
corpus/consented.jsonl
sealed/pcf/<case>.json
blinded/cases/<case>.json
output/analysis.json
audit/events.jsonl
```

`participants.json` is the only file containing consent and contact
references. Blinded case files contain the same canonical issue and policy
context PCF received, but no PCF output, consent reference, contact reference,
or peer rating. The aggregate receipt contains no raw issue text or participant
identifier.

These permissions do not protect data from another process running as the same
operating-system user. Production use still requires an approved restricted
storage location, backups, deletion policy, access logging, and operator model.
Run commands sequentially; the runner is a single-writer local tool.

## 1. Initialize

```bash
pcf study init \
--root /absolute/private/study-root \
--protocol protocol.json \
--sampling-frame sampling-frame.json \
--mode synthetic \
--format json
```

Initialization canonicalizes and hashes the protocol and sampling frame.
Every later command rechecks those hashes.

## 2. Record consent

```bash
pcf study consent --root /absolute/private/study-root --input consent.json
```

The input contains one opaque participant ID, host position, repository,
versioned consent reference, separate contact reference, consent and withdrawal
times, adult and maintainer-role attestations, and validation permission.
Participant IDs and host positions must be unique among active records.

## 3. Lock issue eligibility before PCF

Every public issue number observed after repository activation is recorded once
in strictly ascending order. Inclusion or a frozen objective exclusion is
chosen before any PCF output is inspected.

```json
{
"hostPosition": "H1",
"repository": "owner/repository",
"issueNumber": 123,
"issueCreatedAt": "2026-07-14T11:59:00Z",
"observedAt": "2026-07-14T12:00:00Z",
"disposition": "INCLUDE",
"eligibilityChecked": true,
"pcfOutputInspected": false
}
```

```bash
pcf study observe --root /absolute/private/study-root --input observation.json
```

An exclusion uses `disposition: "EXCLUDE"` and exactly one code from the
frozen protocol. The runner assigns included ordinals from this ledger. A case
cannot be frozen without its prior unmatched `INCLUDE` observation, preventing
post-score cherry-picking.

## 4. Freeze the canonical intake and seal PCF

```bash
pcf study freeze --root /absolute/private/study-root --input case.json --format json
```

The input contains the assigned case/host position, repository, issue ordinal,
initial issue title/body, author association, creation-time labels, creation
time, language, eligibility attestations, and frozen policy context. Comments,
reactions, later labels, assignments, closure, linked patches, and eventual
outcomes are rejected.

The runner:

1. verifies all three assigned positions have active consent;
2. matches the case to the next prior `INCLUDE` observation;
3. canonicalizes the exact intake and computes SHA-256;
4. evaluates that snapshot with the frozen PCF mapping;
5. writes PCF output only under `sealed/pcf/`; and
6. writes the same issue and policy context without PCF output under
`blinded/cases/`.

## 5. Lock ratings

```bash
pcf study rate --root /absolute/private/study-root --input rating.json
```

A rating must match the snapshot hash and one frozen context/external
assignment. External self-rating is rejected. Lane and next actor must be one
of the frozen pairs:

- `review-now` / `maintainer`
- `repair` / `reporter`
- `defer` / `no-action`

Each rater-position file is created exclusively; a second submission cannot
overwrite it. Abstentions remain explicit and never become `defer`.

## 6. Withdraw before aggregate lock

```bash
pcf study withdraw \
--root /absolute/private/study-root \
--participant opaque-participant-id
```

Before aggregate lock, withdrawal deletes that participant's unaggregated
rating files and marks every affected case excluded. After aggregate lock, all
mutation commands fail closed; the consent language governs any later request.

## 7. Inspect, analyze, and lock

```bash
pcf study status --root /absolute/private/study-root --format markdown
pcf study analyze --root /absolute/private/study-root --format json
pcf study analyze --root /absolute/private/study-root --write --lock --format json
```

Aggregate lock requires exactly the protocol target of complete,
non-abstaining cases. It also fails closed until every active participant's
disclosed withdrawal deadline has passed and at least 14 days have elapsed
since the final rating. `--write` stores the metadata-only corpus under
`restricted/corpus/` and the aggregate receipt under `output/`.

Analysis reports:

- counts before rates;
- the three-by-three lane confusion matrix;
- false and missed `review-now` counts and naive Wilson intervals;
- context-versus-external agreement;
- per-stratum confusion counts;
- the existing validator's lane precision/recall, nominal kappa, next-actor
agreement, and fixed score bins; and
- incomplete, abstaining, excluded, and withdrawn cases.

Every analysis remains `INCONCLUSIVE`. Wilson intervals are explicitly naive
because issues cluster within repositories and raters repeat. Timing is not in
the validator corpus, no threshold is tuned, and a later tuned evaluator needs
a new independent holdout.

## Stop conditions

Stop on any consent ambiguity, policy mismatch, suspected security/private
content, out-of-order observation, unapproved exclusion, snapshot corruption,
permission drift, symlink, assignment conflict, self-rating, hash mismatch,
duplicate rating, evaluator-mapping drift, unauthorized production clock,
aggregate-lock mutation, or public-write risk.
5 changes: 3 additions & 2 deletions package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@
],
"scripts": {
"start": "node src/server.mjs",
"check": "node --check scripts/run-pr-gate.mjs && node --check src/server.mjs && node --check src/cli.mjs && node --check src/config.mjs && node --check src/core/api.mjs && node --check src/core/adversary.mjs && node --check src/core/author-context.mjs && node --check src/core/ai-contribution-posture.mjs && node --check src/core/behavioral-signals.mjs && node --check src/core/benchmark.mjs && node --check src/core/calibration.mjs && node --check src/core/candidates.mjs && node --check src/core/contribution-drafts.mjs && node --check src/core/contributor-preflight.mjs && node --check src/core/corpus-validation.mjs && node --check src/core/diff-shape.mjs && node --check src/core/evaluator.mjs && node --check src/core/feedback.mjs && node --check src/core/history.mjs && node --check src/core/issue-form-validator.mjs && node --check src/core/lane-schema.mjs && node --check src/core/lane-status.mjs && node --check src/core/lane-store.mjs && node --check src/core/maintainer-stack.mjs && node --check src/core/pilot-proof.mjs && node --check src/core/policy.mjs && node --check src/core/policy-scan.mjs && node --check src/core/patch.mjs && node --check src/core/queue.mjs && node --check src/core/repository-context.mjs && node --check src/core/repro-gate.mjs && node --check src/core/mcp-submission.mjs && node --check src/core/scout.mjs && node --check src/core/semantic-duplicate-assist.mjs && node --check src/core/serious-scout.mjs && node --check src/core/setup.mjs && node --check src/core/setup-guide.mjs && node --check src/core/shielded-posture.mjs && node --check src/core/text-safety.mjs && node --check src/core/vouch-context.mjs && node --check src/core/watchlist.mjs && node --check src/github/client.mjs && node --check src/github/webhook.mjs && node --check src/github/templates.mjs && node --check src/mcp/core.mjs && node --check src/mcp/server.mjs && node --check scripts/mcp-smoke.mjs && node --check scripts/run-adversary.mjs && node --check scripts/run-benchmark.mjs && node --check scripts/run-maintainer-demo.mjs && node --check scripts/run-public-pilot.mjs && node --check scripts/run-large-bench.mjs && node --check scripts/run-serious-scout.mjs && node --check scripts/run-watchlist.mjs && node --check scripts/verify-ci-workflow.mjs && node --check scripts/verify-repo-hygiene.mjs && node --check public/app.js",
"check": "node --check scripts/run-pr-gate.mjs && node --check src/server.mjs && node --check src/cli.mjs && node --check src/config.mjs && node --check src/core/api.mjs && node --check src/core/adversary.mjs && node --check src/core/author-context.mjs && node --check src/core/ai-contribution-posture.mjs && node --check src/core/behavioral-signals.mjs && node --check src/core/benchmark.mjs && node --check src/core/calibration.mjs && node --check src/core/candidates.mjs && node --check src/core/contribution-drafts.mjs && node --check src/core/contributor-preflight.mjs && node --check src/core/corpus-validation.mjs && node --check src/core/prospective-study.mjs && node --check src/core/diff-shape.mjs && node --check src/core/evaluator.mjs && node --check src/core/feedback.mjs && node --check src/core/history.mjs && node --check src/core/issue-form-validator.mjs && node --check src/core/lane-schema.mjs && node --check src/core/lane-status.mjs && node --check src/core/lane-store.mjs && node --check src/core/maintainer-stack.mjs && node --check src/core/pilot-proof.mjs && node --check src/core/policy.mjs && node --check src/core/policy-scan.mjs && node --check src/core/patch.mjs && node --check src/core/queue.mjs && node --check src/core/repository-context.mjs && node --check src/core/repro-gate.mjs && node --check src/core/mcp-submission.mjs && node --check src/core/scout.mjs && node --check src/core/semantic-duplicate-assist.mjs && node --check src/core/serious-scout.mjs && node --check src/core/setup.mjs && node --check src/core/setup-guide.mjs && node --check src/core/shielded-posture.mjs && node --check src/core/text-safety.mjs && node --check src/core/vouch-context.mjs && node --check src/core/watchlist.mjs && node --check src/github/client.mjs && node --check src/github/webhook.mjs && node --check src/github/templates.mjs && node --check src/mcp/core.mjs && node --check src/mcp/server.mjs && node --check scripts/mcp-smoke.mjs && node --check scripts/run-adversary.mjs && node --check scripts/run-benchmark.mjs && node --check scripts/run-maintainer-demo.mjs && node --check scripts/run-public-pilot.mjs && node --check scripts/run-large-bench.mjs && node --check scripts/run-serious-scout.mjs && node --check scripts/run-watchlist.mjs && node --check scripts/verify-ci-workflow.mjs && node --check scripts/verify-repo-hygiene.mjs && node --check public/app.js",
"mcp": "node src/mcp/server.mjs",
"mcp:smoke": "node scripts/mcp-smoke.mjs",
"setup:pilot": "node src/cli.mjs setup",
Expand Down Expand Up @@ -82,7 +82,8 @@
"demo:issue": "node src/cli.mjs evaluate fixtures/issue-unready.json",
"demo:kernel": "node src/cli.mjs evaluate fixtures/pr-kernel-ready.json --profile kernel-grade",
"preflight": "node src/cli.mjs preflight",
"validate:corpus": "node src/cli.mjs validate-corpus"
"validate:corpus": "node src/cli.mjs validate-corpus",
"study:prospective": "node src/cli.mjs study"
},
"engines": {
"node": ">=22"
Expand Down
134 changes: 133 additions & 1 deletion src/cli.mjs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,19 @@ import {
renderCorpusValidationSummary,
validateCorpusText
} from "./core/corpus-validation.mjs";
import {
ProspectiveStudyError,
analyzeProspectiveStudy,
freezeStudyCase,
initializeProspectiveStudy,
readProspectiveStudyStatus,
recordStudyObservation,
recordStudyConsent,
renderProspectiveStudyMarkdown,
renderProspectiveStudySummary,
submitStudyRating,
withdrawStudyParticipant
} from "./core/prospective-study.mjs";
import { loadConfig } from "./config.mjs";

const args = process.argv.slice(2);
Expand All @@ -22,12 +35,16 @@ if (args.length === 0 || args.includes("--help") || args.includes("-h")) {
}

const command = args[0];
if (!["evaluate", "evaluate-patch", "queue", "setup", "setup-pilot", "preflight", "validate-corpus"].includes(command)) {
if (!["evaluate", "evaluate-patch", "queue", "setup", "setup-pilot", "preflight", "validate-corpus", "study"].includes(command)) {
console.error(`Unknown command: ${command}`);
printHelp();
process.exit(2);
}

if (command === "study") {
await runProspectiveStudy(args.slice(1));
}

if (command === "setup" || command === "setup-pilot") {
const format = readFlag(args, "--format") || "text";
const config = loadConfig(process.cwd());
Expand Down Expand Up @@ -253,6 +270,116 @@ async function readPolicyFiles(path) {
return normalizeRepositoryFiles(Array.isArray(data) ? data : data.repositoryFiles || data.policyFiles || []);
}

async function runProspectiveStudy(studyArgs) {
const action = studyArgs[0];
if (!action || !["init", "consent", "observe", "freeze", "rate", "withdraw", "analyze", "status"].includes(action)) {
console.error("PCF prospective study failed: action must be init, consent, observe, freeze, rate, withdraw, analyze, or status.");
process.exit(2);
}
const root = readFlag(studyArgs, "--root");
if (!root) {
console.error("PCF prospective study failed: --root is required.");
process.exit(2);
}
const format = readFlag(studyArgs, "--format") || "pretty";
if (!["pretty", "json", "markdown"].includes(format)) {
console.error(`PCF prospective study failed: unsupported format '${format}'.`);
process.exit(2);
}
const now = readFlag(studyArgs, "--now");
try {
let result;
if (action === "init") {
const protocolPath = requiredStudyFlag(studyArgs, "--protocol");
const framePath = requiredStudyFlag(studyArgs, "--sampling-frame");
result = await initializeProspectiveStudy({
root,
protocol: await readJsonFile(protocolPath),
samplingFrame: await readJsonFile(framePath),
mode: readFlag(studyArgs, "--mode") || "production",
now
});
} else if (action === "consent") {
result = await recordStudyConsent({
root,
consent: await readJsonFile(requiredStudyFlag(studyArgs, "--input")),
now
});
} else if (action === "observe") {
result = await recordStudyObservation({
root,
observation: await readJsonFile(requiredStudyFlag(studyArgs, "--input")),
now
});
} else if (action === "freeze") {
result = await freezeStudyCase({
root,
input: await readJsonFile(requiredStudyFlag(studyArgs, "--input")),
now
});
} else if (action === "rate") {
result = await submitStudyRating({
root,
rating: await readJsonFile(requiredStudyFlag(studyArgs, "--input")),
now
});
} else if (action === "withdraw") {
result = await withdrawStudyParticipant({
root,
participantId: requiredStudyFlag(studyArgs, "--participant"),
now
});
} else if (action === "analyze") {
result = await analyzeProspectiveStudy({
root,
write: studyArgs.includes("--write"),
lock: studyArgs.includes("--lock"),
now
});
} else {
result = await readProspectiveStudyStatus({ root });
}
printStudyResult(result, format);
process.exit(0);
} catch (error) {
const message = error instanceof ProspectiveStudyError
? error.message
: error?.code === "ENOENT"
? "Required local study file is missing."
: error instanceof SyntaxError
? "Input file contains invalid JSON."
: "Unexpected local study-runner error.";
console.error(`PCF prospective study failed: ${message}`);
process.exit(1);
}
}

function requiredStudyFlag(values, flag) {
const value = readFlag(values, flag);
if (!value) throw new ProspectiveStudyError(`${flag} is required.`);
return value;
}

async function readJsonFile(path) {
return JSON.parse(await readFile(path, "utf8"));
}

function printStudyResult(result, format) {
if (format === "json") {
console.log(JSON.stringify(result, null, 2));
} else if (format === "markdown") {
process.stdout.write(renderProspectiveStudyMarkdown(result));
} else if (result.artifact === "pcf-prospective-study-status" || result.artifact === "pcf-prospective-study-analysis") {
process.stdout.write(renderProspectiveStudySummary(result));
} else {
console.log(`PCF prospective study: ${result.status || "PASS"}`);
if (result.studyId) console.log(`Study: ${result.studyId}`);
if (result.caseId) console.log(`Case: ${result.caseId}`);
if (result.snapshotSha256) console.log(`Snapshot SHA-256: ${result.snapshotSha256}`);
console.log("Decision boundary: INCONCLUSIVE");
}
}

function printHelp() {
console.log(`Usage:
node src/cli.mjs setup [--repository owner/repo] [--base-url http://127.0.0.1:3791] [--public-base-url https://example.tunnel] [--format text|json|markdown]
Expand All @@ -261,6 +388,11 @@ function printHelp() {
node src/cli.mjs evaluate-patch <patch-or-mbox> [--format pretty|json|markdown] [--profile kernel-grade] [--policy policy-files.json]
node src/cli.mjs preflight <payload.json|patch-or-mbox> [--allow-repair] [--format pretty|json|markdown] [--profile standard|kernel-grade] [--policy policy-files.json]
node src/cli.mjs validate-corpus <consented.jsonl|consented.csv|-> [--input-format jsonl|csv] [--format pretty|json|markdown]
node src/cli.mjs study init --root <absolute-path> --protocol <protocol.json> --sampling-frame <frame.json> [--mode production|synthetic] [--format pretty|json|markdown]
node src/cli.mjs study consent|observe|freeze|rate --root <absolute-path> --input <record.json> [--format pretty|json|markdown]
node src/cli.mjs study withdraw --root <absolute-path> --participant <opaque-id> [--format pretty|json|markdown]
node src/cli.mjs study analyze --root <absolute-path> [--write] [--lock] [--format pretty|json|markdown]
node src/cli.mjs study status --root <absolute-path> [--format pretty|json|markdown]
cat queue-payload.json | node src/cli.mjs queue - --format json

Preflight exit codes: 0 = ready to submit, 1 = not ready, 2 = usage error.
Expand Down
Loading