feat: add fail-closed prospective maintainer study runner - #12
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0d2755b723
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| caseId: item.caseId, | ||
| stratum: item.stratum, | ||
| pcfLane: seal.pcf.lane, | ||
| consensusLane: strictMajority(ratings.map((rating) => rating.lane)), |
There was a problem hiding this comment.
When the three raters split across all three lanes, strictMajority(...) returns null, but this case still remains in completed and counts toward targetCases for --lock. The later false/missed review-now counts and per-stratum confusion matrix then treat that null consensus as data while the corpus validator excludes it as ambiguous, so an ambiguous case can be locked as complete and corrupt the reported rates.
Useful? React with 👍 / 👎.
| if (write || lock) { | ||
| await writePrivateFile(join(loaded.paths.corpus, "consented.jsonl"), corpusText); | ||
| await writePrivateJson(join(loaded.paths.output, "analysis.json"), result); |
There was a problem hiding this comment.
If an operator runs pcf study analyze --write without --lock while withdrawal is still allowed, these files are written even though aggregateLockedAt remains unset. A later withdrawStudyParticipant only deletes the participant's rating files and marks cases excluded; it never removes or rewrites restricted/corpus/consented.jsonl or output/analysis.json, so withdrawn ratings can remain in durable aggregate artifacts.
Useful? React with 👍 / 👎.
| for (const position of expected) { | ||
| if (!positions.has(position)) throw new ProspectiveStudyError(`Sampling slot '${slotId}' references unknown position '${position}'.`); |
There was a problem hiding this comment.
This validation checks only the context/external rater positions, but never verifies that slot.hostPosition itself exists in hostPositions or that its per-host ordinal is usable. A sampling frame with a typo in slot.hostPosition initializes successfully, yet consent/observation can never be recorded for that slot because consent rejects unknown host positions, making the target case count impossible to reach after the study has already been initialized.
Useful? React with 👍 / 👎.
1 participant
Summary
pcf studyworkflow for the frozen 48-case independent-maintainer protocolWhy this is narrow
This is one new study-runner core, one focused test file, one operator guide, and the minimum CLI/package wiring. It adds no dependency, network call, GitHub write capability, evaluator threshold change, benchmark expectation change, or live participant data.
Validation
node --test test/prospective-study.test.mjs test/corpus-validation.test.mjs— 11/11 passednpm run ci:gates— 268/268 tests, 77/77 benchmark cases, 29/29 adversarial cases, maintainer demo PASSnpm run mcp:smoke— PASS; no public-write toolsINCONCLUSIVE0d2755b72343dbe93efbbc9cffe24f301adeb7bc— good signature; tree and five file hashes match the prior sealed implementation3035da7826bbe2cf76643b831cbae0ca904e7a8d01cdb11b72e21464851adc08Boundaries
This PR does not activate a study, contact maintainers, register research, promise compensation, collect participant data, establish institutional compliance, or claim product validity. All analysis remains
INCONCLUSIVEuntil a genuinely consented, independently governed prospective corpus exists.