Literature-seeded islands + parallel backtest runner - #113
Draft
elliotchung wants to merge 9 commits into
Draft
Conversation
Two coupled changes that prepare the research loop for higher-throughput multi-agent operation: Backtest runner (scripts/run_research_backtest.py): - Date-parallel ThreadPoolExecutor with --max-workers flag (default 5). Each worker still spawns an isolated Nautilus subprocess; only the date-major loop is parallelized. Measured 2.33x speedup with 3 workers on the 12-date train window (574s vs ~1338s implied serial). - Cross-session fcntl.flock on data-cache/.backtest.lock so independently launched runs (e.g. per-island /loop drivers) queue safely on the engine instead of racing on S3 sync, data-cache, and host RAM. - Shared baseline cache at data-cache/baseline-results/<baseline>/<hash>/ keyed by strategy_kwargs hash. Auto-populated on every baseline run, auto-read on --use-cached-baseline (with per-algo fallback). Lets every island/researcher hit the same precomputed baseline. - iteration_timeout_seconds preserved; in-flight tasks finish on budget exceeded while pending ones are cancelled. Island researcher (research/config.yaml, .claude/agents/island-researcher.md): - Replaced 3 hand-picked base_algo islands with 5 themed islands seeded from docs/literature/ clusters (Optimal scheduling, Microstructure signals, Volatility estimation, Lead-lag, Inventory and sizing). Each island lists seed_papers; reference_papers are shared diagnostic context not assigned to any island. - Added generation_one_size: 4 so cold-start G1 gets extra loops. - G1L1 reads seed_papers in full and implements from scratch on top of the simple ExecAlgorithm template; later loops walk lineage via the new parent_id field. Loop schema now carries seed_papers + parent_id + theme for structured lineage instead of NOTES.md prose. - Migration synthesis updated: base_specific renamed to theme_specific. Other: - docs/OBJECTIVE.md documents the per-island NOTES.md convention (experiments/island_experiment/<id>/NOTES.md) so concurrent island sessions do not trample the global research/NOTES.md. - pyproject.toml / uv.lock: add zstandard for DBN decode on hosts without the zstd CLI. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Cold-start G1L1 for island-sig (microstructure signals). Two-signal AND gate combining Lipton static top-of-book imbalance with Kolm rolling per-quote-tick OFI — skip an opening order only when both signals point against the trade direction. Result: FAIL vs simple baseline. pnl=-$550.50 vs +$156.00 (-452.88% delta), sharpe -1.97 vs +0.60, trade_count -7.17%. is_weighted_bps improved -16.18% (gate correctly identifies high-arrival-IS moments at the tick scale), but those moments are continuation moves over the oracle 30s horizon — skipping them throws away the strategy best winners. Hypothesis falsified by sign-inversion: Lipton adverse-selection framing applies to passive posting (queue depletion while waiting), not marketable execution against a multi-second alpha. Loop 2 candidates pre-registered in NOTES.md. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
The first island-sig g1l1 loop landed before the new loop-file schema was fully internalized by the agent, so it kept the old base_algo field and omitted theme, seed_papers, and parent_id. Back-fill those fields on the loop and program-database entries (lossless — substantive content was already correct) and add a louder schema-reminder block to procedure step 9 in island-researcher.md so future loops follow the current schema. Also: step 9 previously said "Populate seed_paper (g1l1 only)" which contradicted the schema docs — seed_papers is the same value on every loop in a lineage (identifies the intellectual origin), only parent_id varies. Fixed. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Sign-flipped fork of sig-isl-g1l1. G1L1 same-direction microstructure SKIP gate (skip when Lipton I AND Kolm OFI both push in trade direction) empirically failed (-452.88% PnL): against a 30s oracle horizon, strong same-direction microstructure is a continuation signal -- those are the winners, not the losers. G1L2 inverts the predicate: skip an opening order only when BOTH signals point AGAINST the trade direction (opposite-tape extreme), the regime where the trader fights fresh, observable directional flow on the way in and pays both an adverse arrival print AND an early position drawdown that chews through the 30s alpha. Thresholds preserved at G1L1 defaults (imbalance=0.33, ofi_window=2.0s, ofi=5.0) for clean A/B; added per-side evaluated/skipped counters per G1 migration instrumentation rule. Train (12 dates): pnl +1502.75 vs base +156 (+863.30%), sharpe 5.83 vs 0.60, max_drawdown -0.0472 vs -0.0529 (+10.78%), win_rate +2.66pp, trade_count -7.69% (gate fires on ~7.7% of opens, comparable to G1L1 ~7.2%). is_weighted_bps WORSENED +21.48% while PnL improved dramatically -- reproduces G1 migration IS-vs-PnL dissonance finding. PASS. No snapshot, no push -- operator instruction.
Adds a direct Databento HTTP fetcher alongside the existing S3 sync so backtests can pull single-symbol DBN partitions on demand instead of syncing the full multi-instrument daily file from S3. Selected via `data_source: s3|databento` in research/config.yaml (default `s3`); the S3 codepath is untouched. The Databento branch fetches `mbp-1` only and reuses Nautilus's `include_trades=True` to synthesize trade ticks, matching the legacy S3 partition's record shape exactly — verified by an OOS rerun of vrs-isl-g4l2 producing byte-identical realized_pnl, sharpe, win_rate, trade_count, and vs_baseline_pnl_pct against the May-26 S3 result. Includes a one-shot `build_roll_schedule.py` that derives a per-date front-month roll table from Databento `ohlcv-1d` parent symbology with N-session hysteresis, written to `research/roll_schedule.yaml` for multi-month backtest windows. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Flip research/config.yaml data_source s3 -> databento so the literature-islands experiment pulls market data directly from the Databento Historical API instead of S3. Reverting to S3 is a one-line change back. Requires DATABENTO_API_KEY in the environment. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The download temp file used a single fixed name (<file>.dbn.zst.download), so two workers fetching the same (dataset, schema, symbol, date) concurrently — e.g. the algo and baseline backtests for one date under the date-parallel runner — raced and one crashed with "[Errno 17] File exists". Give each fetch a per-process+thread-unique temp name, and unlink it in a finally block so a failed download leaves no stale temp. os.replace stays atomic; if another worker finishes first, last writer wins with identical payload. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Stack a rolling-spread-p75 OPEN-side SKIP axis ON TOP of sig-isl-g1l2's opposite-tape (Lipton imbalance AND Kolm OFI) gate with OR-skip composition. G1L2's gate is preserved bit-for-bit; spread parameters (60s / q=0.75 / min_samples=50) ported verbatim from island-0 g1l1 - now confirmed verbatim-portable across four bases (ptg, afg, vrs, sig). Per-axis attribution counters added. Result: +$3265 realized PnL (G1L2 +$1502.75, simple +$156). Sharpe 13.45 (G1L2 5.83). Drawdown tightened to -0.0268 (G1L2 -0.0472). Every one of 12 train dates improved vs G1L2; no date harmed. is_weighted_bps normalized from G1L2's +21.48% dissonance back to +0.05% vs base - the spread axis removed the wide-spread slice the opposite-tape gate was leaving intact. PASS pass_gate and PASS refinement.targets on PnL and sharpe simultaneously.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two coupled changes that prepare the research loop for higher-throughput multi-agent operation:
Backtest runner (scripts/run_research_backtest.py):
Island researcher (research/config.yaml, .claude/agents/island-researcher.md):
Other: