RL fine-tuning of Inkling against a proper scoring rule (Brier) to produce calibrated win-probability estimates on heads-up Texas Hold'em flop equity questions. Ground truth comes from a CFR+ solver used as a headless reward oracle. Training runs via the Tinker API — no local GPU.
Fine-tuning did not correct base Inkling's pre-existing compression-toward-the-mean calibration pattern at pilot scale. Base Inkling substantially outperforms both a constant-baseline and a preflop-equity heuristic zero-shot; fine-tuning against the Brier score at LR=5e-6 over 3–5 update steps produced aggregate results within seed-to-seed noise of base Inkling, and specifically failed to correct the systematic overprediction on low-equity hands (bins 1-2, +12-15pp too high), which persisted or slightly worsened across every variant tested including one with rebalanced training data.
Full pilot writeup with numbers, methodology, mechanism findings, and
limitations: RESULTS.md.
solver/ wrapper around the CFR+ solver as a headless reward oracle
data/ situation sampler + prompt template + strict parser
reward/ Brier scoring rule, baselines, reward-hacking instrumentation
training/ async rollout engine, LOO advantage, forward_backward + optim_step
inkling/ Tinker sampling client (sync + async, base + LoRA)
scripts/ end-to-end runners: generate_dataset, compute_baselines,
run_pilot, run_eval, run_baseline_c, calibration_curve,
posthoc_advantage_analysis, retroactive_gradient_check
tests/ 65 tests (pytest)
docs/ phase-by-phase design docs (00–06)
results/ committed artifacts — see "Reproducibility" below
RESULTS.md pilot writeup and headline findings
CLAUDE.md working conventions for AI-assisted contributions
Requires Python 3.11+ and PyTorch 2.10+.
pip install "tinker>=0.23.0" "tinker-cookbook[inkling]>=0.5.0" \
"tml-renderers>=0.1.0" torch pytestNote that tml_tokenizers is imported by tinker's Inkling code path but
is not on PyPI as of tinker 0.23.0. The cookbook ships a working
tokenizer facade around tml_renderers.tokenizers.o200k_base_chat,
which is what inkling.client.InklingClient uses directly — no separate
tml_tokenizers install needed. See inkling/client.py for details.
Set TINKER_API_KEY in your environment. The Tinker SDK reads it
automatically; nothing in this repo reads or persists it.
The solver is a separate MIT-licensed repo:
arpjw/regret. All results in this
repo are reproducible against commit a970745 — the commit that
added regret-equity, the JSONL CLI this repo drives.
# Clone regret next to this repo (or anywhere convenient).
git clone https://github.com/arpjw/regret ~/Developer/regret
cd ~/Developer/regret
git checkout a970745 # pinned for reproducibility
# Symlink it into this project.
cd - # back to inkling-calibration/
ln -s ~/Developer/regret solver/regret
# Build the equity CLI.
(cd solver/regret && cargo build --release --bin regret-equity)solver/regret is gitignored — it's a separate git repo with its own
history; do not commit it here. If you clone this repo without setting up
the symlink, oracle-dependent scripts (dataset generation, baselines B/C,
evals) will fail with a clear "regret-equity binary not found" error
pointing you to the build command above.
python3 -m pytest tests/ # 65 tests, should pass in ~4s
python3 scripts/sanity_equity.py # one oracle call, prints the equity
python3 scripts/sanity_inkling.py # one Tinker sample, prints Brier vs oracleEvery number in RESULTS.md is reproducible against the checkpoint URLs
and JSONL artifacts in results/. Reproducing the full pipeline from
scratch (i.e., ignoring committed intermediates and re-running everything
against Tinker) is a ~15-hour, ~13M-output-token job — do this only if
you're intentionally checking the training pipeline itself.
# 1. Generate v1 dataset (naturally-weighted quotas). ~3 min, oracle only.
python3 scripts/generate_dataset.py \
--out results/data/flop_v1 \
--per-bin "0,100,200,200,200,200,200,200,150,50"
# 2. Generate v2 dataset (flat quotas). ~3 min, oracle only.
python3 scripts/generate_dataset.py \
--out results/data/flop_v2_flat \
--per-bin "0,167,167,167,167,167,167,167,167,167"
# 3. Baselines A and B (oracle only). ~2 min.
python3 scripts/compute_baselines.py
# 4. Baseline C — base Inkling on full v1 held. ~40 min at 8-way, ~1.3M tokens.
python3 scripts/run_baseline_c.py --concurrency 8 --label c
# 5. V1 pilot — 5 update steps. ~3–4 hours, ~2.3M tokens.
python3 scripts/run_pilot.py \
--train results/data/flop_v1/train.jsonl \
--out results/pilot --n-steps 5 --seed 0x517E7
# 6. V1 eval — held-out on v1 held. ~2 hours, ~1.3M tokens.
python3 scripts/run_eval.py \
--checkpoints results/pilot/checkpoints.jsonl \
--data results/data/flop_v1 --out results/eval \
--label finetuned_step04 --concurrency 8
# 7. V2 pilots — 2 seeds, 3 steps each. ~4 hours total, ~2.6M tokens.
python3 scripts/run_pilot.py \
--train results/data/flop_v2_flat/train.jsonl \
--out results/pilot_v2_seed1 --n-steps 3 --seed 0x517E7
python3 scripts/run_pilot.py \
--train results/data/flop_v2_flat/train.jsonl \
--out results/pilot_v2_seed2 --n-steps 3 --seed 0xB4B4B4
# 8. V2 evals — both seeds. ~4 hours, ~2.6M tokens.
python3 scripts/run_eval.py \
--checkpoints results/pilot_v2_seed1/checkpoints.jsonl \
--data results/data/flop_v1 --out results/eval_v2 \
--label v2_seed1_step02 --concurrency 8
python3 scripts/run_eval.py \
--checkpoints results/pilot_v2_seed2/checkpoints.jsonl \
--data results/data/flop_v1 --out results/eval_v2 \
--label v2_seed2_step02 --concurrency 8
# 9. Analysis (all free; oracle/Tinker not called).
python3 scripts/calibration_curve.py # 4-way per-bin table
python3 scripts/posthoc_advantage_analysis.py # advantages + parse-fails per bin
python3 scripts/retroactive_gradient_check.py # would-fix vs actual signalTotal Tinker spend for a full reproduction: ~13M output tokens over ~15 hours at 8-way concurrency.
MIT. See LICENSE. The regret solver is separately
MIT-licensed at arpjw/regret.