feat(tensor): byteConversionAppend — bit-offset packing for mixed-width streams

[LeoBuron]

Context

A student is building delta compression (wide base values + narrow deltas, bit-dense) and hit two limits of the bit-packing layer: byteConversion can only start writing at bit 0 of a byte pointer, and its leading memset clobbers previously written bits in a shared byte. Appending mixed-width segments at non-byte-aligned boundaries is therefore impossible today. (In-tree chunked packing is unaffected: elemOffset % 8 == 0 keeps every chunk start byte-aligned for any qBits — see the _Static_assert in TensorConversion.c.)

Change

• byteConversionAppend(dataIn, dataInBits, dataOut, dataOutBits, numValues, dstStartBit) (Tensor.h/Tensor.c): identical loop machinery, out cursor seeded from dstStartBit, no memset. Bits outside the written range are preserved; numValues == 0 is a safe no-op.
• writeByte: OR → clear-then-set. The [startbit, endbit) range is now fully defined regardless of stale in-range bits, making appends robust on previously written buffers. Behavior-identical for all existing callers — every production write lands on memset-zeroed in-mask bits.
• byteConversion delegates to byteConversionAppend(..., 0) after its memset — single source of truth for the loop.
• Second commit — latent underflow fixed: byteConversion with numValues == 0 used to underflow its memset size (n*bits-1)/8+1 to ~SIZE_MAX (crash; one guard away via convertSymTensorToInt32Tensor + tensor: N=0 element allocation has implementation-defined behavior #160 N=0 tensors). Now: early return for numValues == 0 (no memset — N=0 tensor data may be NULL), and the size expression uses the canonical ceiling idiom (bits+7)/8, which is identical for bits > 0 and safely 0 for dataOutBits == 0.
• One-line contract note in docs/conventions/tensor.md: zero-fill-on-widen / sign-extend caveat applies to the append entry point too.

Verification

• TDD: all 9 new tests watched failing first (stub RED / SIGSEGV-RED for the underflow pair), gold fixtures derived via an independent LSB-first reference packer (not the implementation).
• 7-round mutation check: every new test kills at least one wound (OR-writeByte, ignore dstStartBit%8, ignore dstStartBit/8, unconditional clobber, broken input reseed, skip-zero-fill, restored underflow formula). The skip-zero-fill mutation was surfaced by review and is covered by the widening-into-stale-buffer test.
• Prototype of the same design passed 300k+ randomized differential fuzz trials (mixed widths 1–32, offsets to 5M bits, packed sub-byte input, ASan/UBSan clean) during the analysis phase.
• ctest --preset unit_test_debug: 81/81; examples build 395/395; uv run pytest: 29/29.

Notes

• The delta-compression consumer itself (caller tracks a size_t bit cursor; signed deltas must unpack via the unpackSignExtend idiom; overflow → fit-guard abort per numerics: SYM int16×int16 products overflow int32 in Linear/Matmul/LayerNorm (Conv #45 fixed via int12 operands) #227) is intentionally out of scope.

🤖 Generated with Claude Code