Skip to content

feat: layer freezing PR2 — backward truncation at deepest trainable layer + dx wire elision (#380)#393

Open
LeoBuron wants to merge 3 commits into
developfrom
feat/freezing-pr2-dx-truncation
Open

feat: layer freezing PR2 — backward truncation at deepest trainable layer + dx wire elision (#380)#393
LeoBuron wants to merge 3 commits into
developfrom
feat/freezing-pr2-dx-truncation

Conversation

@LeoBuron

Copy link
Copy Markdown
Member

Part of #380 (PR2 of 3 — do not auto-close the epic). Builds directly on #392 (PR1).

What

Backward dx-chain truncation — the TTE-class memory/compute win PR1's freezing enables:

  • propLoss == NULL grads-only contract on the five parameter-layer backwards: compute weight/bias (gamma/beta) grads exactly as normal, touch no dx memory. LN/GN SYM paths skip pass B entirely when NULL (dx requant, plQC->scale refresh, entry validations, empty-geometry writes).
  • Truncated backward loop (calculateGradsImpl): deepestTrainableIndex finds the closest-to-input layer whose params still train; the loop runs backwardIndex → D only, passes NULL propLoss at D (its dx wire is never allocated), and nothing below D executes. A model with no trainable params skips the entire backward (loss value still computed).
  • Trace semantics documented (TraceApi.h): lossgrad/agrad only for the executed range; nothing below D; no backward events at all for all-frozen models.

Behavior change disclosure (deliberate, epic-spec'd)

The cut is at the deepest parameter layer. Consequences beyond frozen models:

  • Leading non-param heads (Flatten/Pool/Dropout/Quant — the repo's standard data-shape convention) no longer run backward at all, and their previously-fired agrad events vanish from traces. Concretely: kws_raw's pool0.agrad parity probe disappears; trace_compare.py does not flag one-sided files (follow-up below).
  • Fully-trainable param-first models save layer-0's previously computed-and-discarded dx.
  • Weight/bias gradients are bit-identical in all cases (pinned by an upper-grad equivalence twin test); forward, loss values, and all optimizer behavior are unchanged.

Testing

  • TDD with mutation checks throughout (guard inversions, loop-bound widening, deepestTrainableIndex corruption — all killed by named tests).
  • 7 NULL-twin tests (grads byte-identical with/without dx, per layer incl. LN/GN SYM), truncation trace-count test, upper-grad equivalence twin, all-frozen edge test. PR1's e2e gate ran green untouched throughout (it exercises truncation live).
  • Whole-branch review (independent): guard completeness verified by exhaustive propLoss-> audit; gradNext ownership re-derived across all three model shapes (no leak/double-free); 208 tests across touched binaries re-run clean.
  • Full local CI green in the workspace: ctest 81/81, pytest 29/29; examples preset all targets (incl. trace_c_kws_raw).

Known minors (deliberate)

  • New test fixtures follow their files' existing leak-tolerant idioms (CI runs ASan with detect_leaks=0, LSan tracked under Enable LeakSanitizer in CI after establishing cleanup convention #82).
  • CE-degenerate note: a trainable param layer as the LAST layer under CROSS_ENTROPY (contract violation — Softmax must be last) would seed a lossgrad without a loop iteration; unreachable under the documented CE contract.

Follow-up candidates (not filed yet)

  • Frozen-sandwich loop-level test ([trainable, frozen, trainable] composition) and LN/GN-at-deepest loop coverage.
  • trace_compare.py: report one-sided (missing-pair) files instead of silently skipping.

🤖 Generated with Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant