Skip to content

Audit LLM stack and harden training recovery#72

Merged
ppodolsky merged 1 commit into
mainfrom
audit-llm-code-and-repos
Jul 19, 2026
Merged

Audit LLM stack and harden training recovery#72
ppodolsky merged 1 commit into
mainfrom
audit-llm-code-and-repos

Conversation

@ppodolsky

Copy link
Copy Markdown
Contributor

Summary

  • consolidate MAL parsing and model validation around the shared Rust implementation, split training data/checkpoint responsibilities into documented modules, and add the LLM code map
  • rename hermes-mal-py to hermes-mal-python consistently across Cargo, Python packaging, CI, Docker, and contributor docs
  • track the remaining CubeK, CubeCL, and Burn fork pins with upstream PRs and explicit removal conditions instead of vendored source
  • add an idempotent hermes-train relaunch supervisor with process retry, complete-checkpoint validation, versioned GCS publication, stale-remote protection, legacy-backup migration, and final sync
  • supervise W&B with the trainer, use the server's authoritative last history step on resume, and survive metrics-file replacement without duplicate logging

Upstream tracking

The CubeCL and Burn PR Cargo diffs contain no Hermes or personal-repository references.

Verification

  • cargo fmt --all -- --check
  • cargo clippy --all-targets -- -D warnings
  • cargo test
  • shellcheck -x hermes-train/scripts/relaunch.sh hermes-train/scripts/relaunch_test.sh
  • uvx ruff check hermes-train/scripts/wandb_tail.py
  • uvx ruff format --check hermes-train/scripts/wandb_tail.py
  • relaunch recovery suite on macOS and the Linux A100 host
  • live W&B resume verified against retriever-100m-stage1

@ppodolsky
ppodolsky merged commit ab31e27 into main Jul 19, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant