fix: hash UNet engine cache key so ONNX export path stays under Windows MAX_PATH#10
Conversation
…ws MAX_PATH EngineManager.get_engine_path encoded every UNet build flag verbatim into the on-disk directory name. With a realistic engine_dir (fp8 + static batch + pin_cache_frames + optlvl + resolution), the folder grew to 246 chars, and the derived unet.engine.onnx / unet.engine.opt.onnx paths hit 263/267 chars -- over Windows' 260-char MAX_PATH. Because the *directory* fit but the *file* didn't, mkdir succeeded while torch.onnx.export's open(onnx_path, "wb") raised FileNotFoundError, which wrapper.py's OOM-only fallback doesn't catch, so the build failed with "Acceleration has failed: [Errno 2] No such file or directory: ...unet.engine.onnx". Fix: for EngineType.UNET only, hash the fully-assembled flag prefix (sha1, first 12 hex chars) using the existing _lora_signature hashing idiom, keeping model name + fp8 + resolution human-readable in the folder name. Cache-key precision is unchanged (identical configs still hash identically, differing configs still diverge) -- only the on-disk encoding is compacted, from ~267 chars down to ~132. VAE/ControlNet directory naming is untouched, so existing VAE engines are not invalidated; UNet engines rebuild once under their new hashed folder name. Also adds a belt-and-suspenders warning if a derived path would still approach MAX_PATH (e.g. an unusually deep user engine_dir), and casts the "filename" config lookup to str to keep the type checker precise about self._configs' heterogeneous per-key value types. Adds a regression test (test_engine_path_length.py) that reproduces the exact crash-log config and asserts the derived .onnx / .opt.onnx paths stay under MAX_PATH, plus determinism and no-collision checks. Co-Authored-By: Claude Opus 4.8 <[email protected]>
pin_cache_frames/cache_maxframes are a 040-release-only feature not present on SDTD_032_dev's get_engine_path yet. Filtering _CRASH_UNET_KWARGS against inspect.signature keeps the cherry-picked regression test portable across branches instead of hand-editing it per port; the UNet short_hash/MAX_PATH fix itself is unconditional and unaffected.
|
Claude finished @forkni's task in 3m 6s —— View job Code ReviewTasks:
Summary
FindingsNo issues found. Traced through the logic in detail:
One thing worth calling out for the PR description/reviewers (not a defect, just an expected consequence): since the UNet cache-key directory name changes from the verbose form to the hashed form, previously-built UNet engines will be cache-missed and rebuilt once on first run after this lands — expected and unavoidable for this kind of fix, not something to change. |
Summary
Cherry-picks
153669bfromfix/unet-engine-path-maxpath(dotsimulate upstream PR cumulo-autumn#29, still open/unmerged) ontoSDTD_032_dev. Without this, ansd-turbo+fp8+tensorrtbuild with a nestedengine_dirproduces a UNet cache-key directory name long enough that the derivedunet.engine.onnx/.opt.onnxpaths exceed Windows' 260-char MAX_PATH —mkdirsucceeds buttorch.onnx.export'sopen(path, "wb")raises a bareFileNotFoundError, surfaced to users as "Acceleration has failed".The fix hashes the full UNet prefix (same idea as the existing
_lora_signatureLoRA-hash pattern) into a short, stable, collision-free directory name, and adds a warning if the derived path would still exceed MAX_PATH.src/streamdiffusion/acceleration/tensorrt/engine_manager.py— UNet-only:short_hash = sha1(canonical_prefix)[:12],prefix = f"{base_name}{fp8_tag}--h{short_hash}{res_tag}", plus alongest_derived >= 250warning.tests/unit/test_engine_path_length.py— 3 regression tests. Adapted from the upstream commit:SDTD_032_dev'sget_engine_pathdoesn't yet have thepin_cache_frames/cache_maxframesparams (a 040-release-only feature), so_CRASH_UNET_KWARGSis filtered againstinspect.signature(get_engine_path)before use — keeps the test portable across branches without hand-editing it per cherry-pick. The UNet hashing fix itself is unconditional and unaffected by this filtering.Companion PR on
SDTD_040_beta_release: #9.Does not touch
fix/unet-engine-path-maxpathor upstream PR cumulo-autumn#29 — that PR's merge decision is dotsimulate's; this only lands the fix on our fork's own branches.Test plan
python -m py_compileon both changed filespytest tests/unit/test_engine_path_length.py -v— 3 passed (validated against this branch'ssrc/, not the stale editable install, viaPYTHONPATHoverride)claude-code-review) green before merge🤖 Generated with Claude Code