Offline gpt-oss:20b DFIR test + automated scorecard grader - #36
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b09ed6c358
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| const verdict = JSON.parse(readFileSync(verdictPath, "utf8")) | ||
| const r = scoreRun(verdict, caseGt) |
There was a problem hiding this comment.
The grader's CLI reads a standalone verdict.json and immediately scores it, so neither run.manifest.json nor manifest_verify.json/audit-chain custody is checked before returning success. In the documented offline scorecard gate, a stale or fabricated verdict containing the expected techniques plus agent: "verdict (gpt-oss:20b)" can pass --min-score and --require-llm even with no verified manifest, undermining the repository's custody-first guarantee; require a run directory and validate custody before success.
Useful? React with 👍 / 👎.
| const runFixture = (name) => | ||
| JSON.parse(readFileSync(fileURLToPath(new URL(`../fixtures/dfir-scorecard/runs/${name}.verdict.json`, import.meta.url)), "utf8")) |
There was a problem hiding this comment.
The workflow runs node scripts/selftest.mjs under a documented synthetic-only CI policy, but this loader pulls the new scorecard fixtures whose docs/contents identify them as captured real EVTX outputs, including evidence_path values under evidence/real-evtx-20260708 with hosts/accounts/IPs. That makes every CI run depend on non-synthetic evidence and publishes local case data; replace these with synthetic/minimized redacted fixtures or keep the real runs outside the repo/CI.
Useful? React with 👍 / 👎.
Status: shipped + CI green ✅The offline EVTX test is landed and verified:
Honest scope (unchanged): this proves the offline detection floor (the deterministic Next: a real forensic-image benchmark — full disk images (E01/dd), not just EVTX — to exercise the |
Add the automated grader the offline battery lacked, plus real captured evidence from running the investigate pipeline offline on the DGX Spark with gpt-oss:20b forced as the agent (CASEFORGE_FORCE_AGENT=1). - scripts/score-offline-run.mjs: parse a sealed verdict.json, diff elevated MITRE techniques/CVEs vs fixtures/dfir-scorecard/ground-truth.json (HIT/PARTIAL/MISS + score), and report llm_provenance so a passing detection score is never mistaken for the LLM having authored the seal. --require-llm gates non-zero on deterministic fallback. - fixtures/dfir-scorecard/: ground-truth oracle + three real sealed verdict.json runs (win-lateral-movement, attack-samples, bench). - scripts/selftest.mjs: scorecard assertions wired into CI. - docs/offline-llm-test.md: honest results. Result: detection floor is complete (13/13 targets HIT, all custody-sealed), but all three runs fell back to the deterministic find_evil_auto engine after gpt-oss:20b failed to seal on invalid tool calls. The LLM authored none of the seals; the grader's llm_provenance makes that explicit.
b09ed6c to
06986dc
Compare
2 participants
What
Adds the automated grader the offline DFIR battery was missing, plus real captured evidence from running the investigate pipeline fully offline on the DGX Spark GB10 with
gpt-oss:20bforced as the agent (CASEFORGE_FORCE_AGENT=1).Honest headline
Offline detection works; the LLM did not author it. Across all three battery cases the deterministic
find_evil_autoengine elevated 13/13 expected MITRE techniques + CVEs and produced custody-sealed,manifest_verify-PASS cases — but in all three runsgpt-oss:20bfailed to seal (repeated invalid tool calls) and caseforge fell back to the deterministic engine. Every sealedverdict.jsoncarriesagent: "find-evil-auto MVP". This proves offline forensic detection, not that the LLM can drive a sealed offline investigation (it currently cannot on 20b).Changes
scripts/score-offline-run.mjs— parse a sealedverdict.json, diff elevated MITRE techniques/CVEs vs the oracle to HIT/PARTIAL/MISS + score, and reportllm_provenance(llm-agent vs deterministic-fallback).--require-llmexits non-zero when the seal came from the fallback — the gate to flip on once the agent can seal.fixtures/dfir-scorecard/ground-truth.json— machine-readable oracle (vendored from the lab scorecard).fixtures/dfir-scorecard/runs/*.verdict.json— the three real sealed offline runs (evidence + CI fixtures).scripts/selftest.mjs— scorecard assertions wired into CI; encodes the "all fell back" finding as tests.docs/offline-llm-test.md— full honest write-up + reproduction.Test plan
node scripts/selftest.mjs(CI step) covers the grader: 11 scorecard assertions — full technique elevation per case,deterministic-fallbackprovenance flagged on all three, LLM-authored recognition, and a regression guard.CASEFORGE_FORCE_AGENT=1 VERDICT_LLM_MODEL=gpt-oss:20b bash evidence/run-investigate-local.sh evidence/real-evtx-20260708/win-lateral-movement"Offline" = local-only inference (localhost Ollama,
--privacy local-only, no cloud/Claude creds); local-only, not air-gapped.