[2.0] Add new Frontier-CS 2.0 problem: nanoslm_hybrid_arch_design#170
Open
sijial430 wants to merge 6 commits into
Open
[2.0] Add new Frontier-CS 2.0 problem: nanoslm_hybrid_arch_design#170sijial430 wants to merge 6 commits into
sijial430 wants to merge 6 commits into
Conversation
Hybrid LM architecture-design task (Olmo Hybrid, ~190M): agents submit a model.py scored on held-out val_bpb vs a locked olmo3_190M baseline, trained under a fixed wall-clock budget on a single H100. - Embedding tying + param_cap reconciled: 400M cap in config.yaml and settings.py; baseline_model.py and reference.py both tie embeddings so the arms differ only in the sequence mixer; reference.py analytic/self-check updated to the tied count. - repro/: paper-faithful rig for Olmo Hybrid Table 5 (190M Base-Easy BPB) -- paper-spec pre-norm transformer + GDN-3:1 hybrid, WSD-S schedule, Modal training on the dolma3 mix. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
…ia Modal - settings.py: batch_size 8 -> 2, grad_accum 4 -> 16 (effective batch 32 unchanged). MEASURED: micro-batch 8 at ctx 8192 OOMs an H100 -- the fp32 CE logits are ~26GB and the baseline arm alone tried to allocate 24.48GiB, so the reference could not run under production settings. micro-batch 2 fits both arms (incl. the GDN hybrid) with margin. A fused/chunked CE would let it grow again (noted as a TODO in-file). - modal_app.py: mount the real FineWeb-Edu corpus (Modal Volume nanoslm-corpus) at the data path so the GPU arms train on real tokens instead of data.py's synthetic fallback; add PYTORCH_CUDA_ALLOC_CONF=expandable_segments to reduce allocator fragmentation for the large CE logits. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
… hardened gate Training/eval harness: - Cosine LR now decays over wall-clock fraction (elapsed/train_seconds), not a hardcoded step horizon -- identical schedule for every architecture under the fixed budget; dead max_train_seconds loop check removed. - Modal run_pair_remote honors the agent/final role split: agent role reuses a baseline cached on the corpus Volume (keyed by config fingerprint + train.bin size + CODE_VERSION v8) and trains only the submission (~T); guard rejections return their public reason instead of environment_error. - Baseline SWA layers run FlexAttention's block-sparse kernel, with autocast dtype unification at the flex boundary (SDPA silently unified mixed q/k fp32 vs v bf16; flex refuses -- the old silent mask fallback had been hiding both the bug and a 2x baseline slowdown). - CUDA-only, no fallbacks: reference requires fla or raises; baseline requires flex or raises; FRONTIER_NANOSLM_SMOKE mode removed entirely. - Warmup fast-fails a model that cannot run at the 8192 eval context before spending the 6h budget; training exceptions of any type now classify as guard errors instead of escaping as environment errors. - compile_model added to the config fingerprint. Reference / baseline fidelity (arXiv 2604.03444 + OLMo-core @ fa6c5014): - GDN head_dim 128 kept per the paper's ladder convention ceil_128(0.75*d/h), with the 7B script's divergent int() rule documented; head_dim 48 probed on H100 (works, but slower via Triton autotune). - allow_neg_eigval=True passed explicitly (fla defaults False; paper requires it); GDN o_proj included in the depth-scaled output init. - vocab padded to 100352 (Olmo 3 convention; ids on disk stay < 100278). - reordered_norm documented against the paper's "pre-norm" wording. Data & scoring: - Corpus staged at 2B train tokens for the 6h budget (prep sizing rationale updated); val re-staged with byte-additivity verified. - AST policy gate: strict import allowlist (torch/numpy/fla/einops/triton + pure-stdlib), banned dynamic-access primitives, banned dunder attributes, banned raw-file-reader names; plausibility floor (bpb < 0.4 -> guard) as the runtime backstop. - Docs: problem-structure section in readme, agent README updated (starter is a byte-identical copy of reference.py). Co-Authored-By: Claude Fable 5 <[email protected]>
- score IS the submission's held-out val_bpb (lower better), no scale/clip; bpb_score_scale removed; failure paths return sentinel 9999 - train_seconds 21600 -> 1800 for fast iteration; fingerprinted lr_schedule_seconds=21600 keeps the full 6h cosine (a 30-min run is the prefix of a long run, not a compressed anneal) - opt-in wandb logging (operator-only env gate; scored paths byte-identical) incl. pre-clip grad_norm; stop shipping stale .assets into the GPU image - calibrated @30min (CRN pairs, full-horizon LR): baseline 1.33325, reference 1.31208; 6h numbers unchanged (0.98112 / 0.93795) Co-Authored-By: Claude Fable 5 <[email protected]>
sijial430
changed the title
sijial430
marked this pull request as draft
July 24, 2026 23:19
sijial430
marked this pull request as ready for review
July 24, 2026 23:19
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
nanoslm_hybrid_arch_designlinear-recurrent sequence mixer) that reaches the lowest held-out
bits-per-byte (
val_bpb) when trained from scratch under a fixedwall-clock budget on one H100. The agent submits a single
model.py; thehidden judge drops it into a locked dolma2-BPE train+eval harness, trains it
for wall-clock
T, and scores its rawval_bpb(a lockedolmo3_190Mbaseline is trained under the identical budget and reported for context).
— a 3:1 GDN:attention interleave at the ~190M
olmo3_190Mshape (d=768, L=12,H=12), dolma2 tokenizer.
Measured calibration (real corpus, under 30 mins / 6 hours budget)
Validation
Commands (from repo root):
Validation logs with Codex (+gpt-5.5) (access needs to be granted):
wandb: https://api.wandb.ai/links/sijial-ai2/u86xnx57
results: https://drive.google.com/drive/folders/157qqq--nHlSpkkur7TT-HkCHBjaAFQdH?usp=drive_link