English | 中文
Most of the judgments we worry about — "should I", "will it work out", "how likely" — never get tested. We say something vague, the world unfolds, we re-narrate the past to fit. Nothing about our judgment actually improves, because there was never a contract with reality in the first place.
This skill makes judgment testable. It walks you through turning each concern into a resolvable forecast (specific event, deadline, settlement criterion), anchors a probability to a reference class instead of a personal story, updates it as evidence arrives, settles it when the deadline hits, and Brier-scores you over time. The ledger is global — every forecast across every project lands in ~/.superforecast/, so your calibration trains on real outcomes, not on memory.
Superforecasting = probabilization + testability. Inspired by Philip Tetlock's Superforecasting and the Good Judgment Project. The book gives the methodology — reference classes, base rates, Bayesian updating, Brier scoring; this project makes the methodology executable. The LLM agent walks you through eight gates (resolvability, Fermi-ization, three-layer reference class, base rate, internal-view adjustments, forecast-vs-decision separation, ...). A deterministic Python engine enforces a state machine that blocks the two most common forecast-killers: predictions without deadlines, and after-the-fact reinterpretation. The goal is not "AI predicts the future" — it is "AI engineers your forecasting process; the human keeps judgment ownership and bears decision responsibility."
User input ("should I leave Beijing?", "will the deal close?", anxiety)
│
▼
[Mode classification] new / update / settle / review / coach
│
▼ (Mode = new)
[Gate 1] Forecast vs decision vs emotion
[Gate 2] Resolvable? — rewrite if not
[Gate 3] Cloud-like? — Fermi-ize into 3-7 sub-questions
[Gate 4] Type: binary (default) / multi_outcome / numeric / decision_bundle
[Gate 5] Three reference classes (broad / medium / narrow)
[Gate 6] External-view base rate (state base BEFORE personal facts)
[Gate 7] Internal-view adjustments (upward/downward × impact bands)
[Gate 8] Forecast vs decision threshold (act / test / pause)
│
▼
[State machine] sf.py persists every step
∅ → DRAFT → SCOPED → ACTIVE → UPDATED* → SETTLED → SCORED (auto Brier)
│
▼
~/.superforecast/forecasts/ ← global ledger, shared across projects
~/.superforecast/reports/ ← calibration reviews
Render → 14-section Markdown forecast card
The skill enforces three engineering constraints that prevent the most common forecast failures: no probability without a deadline + criterion, no rewriting after settlement, no scoring without binary outcome. Together they block open-ended predictions and after-the-fact reinterpretation — the two ways most people's "judgment" never actually improves.
- Eight-gate forecast workflow — every new forecast walks through resolvability, Fermi-ization, three-layer reference class, base rate, internal adjustments, decision threshold
- State machine —
DRAFT → SCOPED → ACTIVE → UPDATED* → SETTLED → SCORED, illegal transitions rejected with clear errors - Append-only event ledger —
events.jsonlis the source of truth;active.jsonand rendered cards are derived - Brier-scored calibration — automatic
(p − outcome)²on settle, per-band calibration report on review - Update with evidence strength buckets — strong (±10–20%) / moderate (±5–10%) / weak (±2–5%), prevents mechanical updating
- Decision ≠ forecast — outputs
act_if_above/test_if_between/pause_if_belowthresholds, never verdicts - Global ledger —
~/.superforecast/shared across all projects and runtimes (Claude Code + Codex see the same history) - Zero dependencies — Python 3.10+ standard library only, no
pip install
- Claude Code CLI or Codex CLI — installed and authenticated
- Python 3.10+ — standard library only, no extras needed
Option A: Git clone (recommended)
git clone https://github.com/deusyu/superforecasting-skill.git ~/.claude/skills/superforecastFor Codex:
git clone https://github.com/deusyu/superforecasting-skill.git ~/.codex/skills/superforecastOption B: Symlink an existing checkout
git clone https://github.com/deusyu/superforecasting-skill.git
ln -s "$(pwd)/superforecasting-skill" ~/.claude/skills/superforecast
ln -s "$(pwd)/superforecasting-skill" ~/.codex/skills/superforecastBoth runtimes share the same skill content and the same global ledger at ~/.superforecast/.
In Claude Code, just say what's on your mind:
我担心三个月后产品留存不行
should I move from Beijing to Shanghai?
how likely is it that we ship by end of Q2?
The skill auto-triggers, walks you through the eight gates, asks for the missing context (resolution date, settlement criterion, reference class anchor), and writes the forecast card.
| Path | What's there |
|---|---|
~/.superforecast/forecasts/events.jsonl |
Append-only event log (source of truth) |
~/.superforecast/forecasts/active.json |
Snapshot of every forecast's current state |
~/.superforecast/forecasts/rendered/<id>.md |
Markdown forecast cards (14-section template) |
~/.superforecast/reports/calibration_*.md |
Calibration review reports |
| Mode | Trigger | Writes ledger? |
|---|---|---|
new |
"should I", "will X", anxiety about a future outcome | ✅ |
update |
New evidence, "the situation changed" | ✅ |
settle |
Outcome happened / didn't, deadline passed | ✅ |
review |
"How am I doing", calibration check | ✅ (aggregate) |
coach |
"Teach me", "what is X" | ❌ |
The gates are a decision tree, not a checklist. Each gate's output feeds the next:
- Forecast / decision / emotion — anxiety reframes to a forecast; should-I splits into multiple sub-forecasts + decision threshold
- Resolvable? — must have a deadline AND a third-party-verifiable criterion, else rewrite
- Cloud-like? — fuzzy words like "happy / succeed / work out" trigger Fermi-ization into 3–7 sub-questions
- Type —
binary(default),multi_outcome,numeric, ordecision_bundle - Three reference classes — broad / medium / narrow, default primary = medium
- External-view base rate — state base from primary reference class before introducing personal facts (this is the single most-violated rule)
- Internal-view adjustments — upward/downward factors with explicit impact bands (
+5% to +10%) - Decision threshold —
act_if_above/test_if_between/pause_if_below, never verdicts
Full gate-by-gate operational detail in references/workflow.md.
∅ ──forecast_created──▶ DRAFT ──question_scoped──▶ SCOPED
│
│ probability_set
▼
ACTIVE ◀──┐
│ │ evidence_update
▼ │
UPDATED ─┘
│ settled
▼
SETTLED ──scored (auto)──▶ SCORED
Side-branch events (no state change; allowed in SCOPED / ACTIVE / UPDATED):
decomposed — Fermi sub-questions
why_wrong_set — reverse-side reasons (Card §10)
update_triggers_set — forward-looking triggers (Card §11)
decision_threshold_set — act / test / pause thresholds (Card §12)
Aggregate event (separate id namespace, written by `sf review`):
reviewed — calibration snapshot, id = review-YYYY-NNN
Three constraints the script enforces:
- No
set-probwithoutscope— every probability must be bound to a deadline + criterion - No
updateorsettlebeforeACTIVE— no probability = nothing to update or settle - No rewriting after
SETTLED— post-hoc revision is rejected
These block the two most common forecast failure modes: open-ended predictions and after-the-fact reinterpretation.
| LLM (agent) handles | Script (sf.py) handles |
|---|---|
| Mode classification | ID / date / probability format validation |
| Eight gates | State machine transition enforcement |
| Reference class selection | Persisting events to events.jsonl |
| Fermi decomposition | Computing Brier Score |
| Evidence strength judgment | Rendering Markdown cards |
| Decision threshold setting | Aggregating calibration reports |
| Narrative + reverse-argument + triggers | Maintaining active.json snapshot |
Principle: semantic judgment → LLM. Deterministic validation / computation / persistence → script. The script refuses to make semantic decisions on the agent's behalf — it will not pick a reference class, judge evidence strength, or rubber-stamp a probability.
For binary forecasts: Brier = (final_probability − outcome)², range [0, 1], lower is better.
sf review --recent N aggregates settled forecasts into probability bands and reports gaps:
- < 20% : ~10% should occur
- 20–40% : ~30% should occur
- 40–60% : ~50% should occur
- 60–80% : ~70% should occur
- ≥ 80% : ~90% should occur
gap > 0 = underconfident; gap < 0 = overconfident; |gap| < 0.05 with n ≥ 10 = well-calibrated in that band. Fence-sitting (≥ 40% of forecasts in the 40–60% band) gets flagged separately.
Full Brier interpretation, evidence-strength buckets, and review-writing guidance in references/scoring.md.
| File | Purpose |
|---|---|
SKILL.md |
Skill definition — workflow, output format, hard constraints, script integration |
scripts/sf.py |
Zero-dependency Python CLI engine — state machine, validation, Brier, render |
references/workflow.md |
Five input modes + eight gates in operational detail |
references/superforecasting_concepts.md |
Terminology and principles glossary |
references/examples.md |
Six worked cases (life, product, business, exam, update, settle) |
references/scoring.md |
Brier interpretation, calibration bands, evidence-strength buckets |
schemas/forecast_event.schema.json |
events.jsonl line format (oneOf discriminator on type; see schema for the canonical event-type list) |
schemas/forecast_card.schema.json |
Render input format (14-section card) |
docs/skill_design.md |
Original design rationale |
docs/concept_understanding.md |
Conceptual foundation document |
sf.py errors are self-explanatory — read them. If it says illegal transition: state X, allowed sources [...], the message names the allowed states; if it says no current probability, the next command is sf set-prob. Coding agents (Claude Code / Codex) can debug these directly without lookup tables.
What's worth listing here are the environment / install / recovery cases the agent can't resolve from the error alone:
| Situation | What to do |
|---|---|
| Skill doesn't auto-trigger in Claude Code / Codex | Verify the symlink: ls -L ~/.claude/skills/superforecast/SKILL.md (or ~/.codex/skills/...). If the file is unreachable, the runtime never loaded the skill. Re-create the symlink. |
| Lost or corrupted ledger | events.jsonl is the only source of truth. If you have it, regenerate active.json and rendered cards by replaying events. If you lost events.jsonl too, the history is gone — back up that one file going forward. |
python3 reports syntax errors on sf.py |
The script requires Python 3.10+ (X | Y union syntax in the __future__ annotations). Upgrade Python or run with an explicit python3.10+ interpreter. |
The MVP covers single-user binary forecasting with manual update/settle. Future work is grouped by independence:
Generalize Brier to multi-class: Σ (p_i − I(outcome=i))² / number_of_classes. Update sf settle to accept --outcome-class for multi_outcome forecasts. Numeric forecasts use absolute or quantile error; record manually until consensus on the right metric.
Today the skill emits update triggers as text in the forecast card. Phase 2 adds sf trigger add <id> --on-date YYYY-MM-DD --message "..." so the agent can re-surface forecasts at the right time. Optional integration with system reminders / cron.
When multiple forecasts cover the same decision (the "leave Beijing" example below cycled through 3 parallel forecasts), the review step should compare them as a portfolio: which path has the highest concentration / lowest variance, where the costs/reversibility differ, what to settle vs. retain. Today the agent does this in narrative; Phase 3 makes it a first-class CLI command.
- Scripts do bookkeeping; LLMs do semantic judgment. State, schemas, dedup, hashing, IO, scoring are deterministic Python. Mode classification, reference-class selection, evidence-strength buckets, decision thresholds are LLM calls.
- Single writer for shared state. Only
sf.pywrites the ledger. The agent never hand-editsevents.jsonloractive.json. - Engineer the failure modes out. No probability without a deadline. No rewriting after settlement. No scoring without binary outcome. The state machine is the enforcement mechanism.
- Forecast ≠ decision. The skill outputs probability + thresholds; the user owns the action.
If you find this project helpful, please consider giving it a Star ⭐!
If this project saves you time or helps you make a better-calibrated decision, consider sponsoring to keep it maintained and improved.
