Skip to content

fix: validate cached ONNX before reuse to prevent empty-graph TensorRT build crash#16

Merged
forkni merged 1 commit into
SDTD_040_beta_releasefrom
fix/onnx-cache-validation
Jul 19, 2026
Merged

fix: validate cached ONNX before reuse to prevent empty-graph TensorRT build crash#16
forkni merged 1 commit into
SDTD_040_beta_releasefrom
fix/onnx-cache-validation

Conversation

@forkni

@forkni forkni commented Jul 19, 2026

Copy link
Copy Markdown
Owner

Summary

  • Fixes a crash during TensorRT engine compilation: Exception: Acceleration has failed: 'NoneType' object has no attribute 'graph'
  • Root cause: an interrupted/failed ONNX export (OOM, process kill, crash mid-write) can leave a syntactically-valid-but-empty protobuf (0 nodes, no/invalid opset) at the cached onnx_path. The export cache check in EngineBuilder.build() was existence-only (os.path.exists(onnx_path)), so the corrupt file was silently reused instead of being re-exported.
  • That empty graph flows into Optimizer.fold_constants() → polygraphy fold_constants, whose ORT symbolic shape inference refuses opset < 7 and returns None; polygraphy then calls get_num_nodes(None), raising the opaque NoneType error deep in a third-party dependency.
  • Confirmed via runtime log from an affected machine: Found cached model: ...vae_decoder.engine.onnx immediately followed by VAE decoder: original .. 0 nodes, 0 tensors, 0 inputs, 0 outputs.

Changes

  • src/streamdiffusion/acceleration/tensorrt/builder.py: adds _onnx_cache_valid(path) — checks file size > 0, ≥1 graph node, opset ≥ 7 (loaded structure-only, no external weight data, so it's cheap even for >2GB SDXL models). The export cache-hit check now requires this in addition to os.path.exists; on a stale/corrupt hit it logs a warning, deletes the bad file, and falls through to re-export.
  • src/streamdiffusion/acceleration/tensorrt/models/models.py: BaseModel.optimize() now fails fast with a clear RuntimeError if the incoming graph has 0 nodes, instead of letting it reach polygraphy and surface as an unreadable NoneType crash.
  • Generic to every engine type: all five entry points (compile_vae_encoder/decoder/safety_checker/unet/controlnet) route through EngineBuilder.build(), and every model subclasses BaseModel.

Test plan

  • Both files pass ast.parse syntax check
  • Verified via code-search that no equivalent ONNX-content validation existed anywhere in the codebase prior to this change
  • Confirmed all 5 compile_* entry points route through EngineBuilder.build() where the new guard lives
  • End-to-end: on a machine with the CUDA/TensorRT stack, truncate a cached engine's .onnx to an empty-but-valid graph and confirm the build now logs Cached ONNX invalid, re-exporting and completes successfully (not runnable in this dev environment — no GPU/TensorRT here)

🤖 Generated with Claude Code

…T build crash

An interrupted/failed ONNX export leaves a syntactically-valid-but-empty
protobuf at onnx_path. The export cache check was existence-only, so the
empty file was reused, feeding a 0-node/opset-less graph into polygraphy
fold_constants -> ORT shape inference bails on opset<7 -> returns None ->
get_num_nodes(None) crashes as "'NoneType' object has no attribute 'graph'".

builder.py: add _onnx_cache_valid() (size>0, >=1 node, opset>=7, loaded
structure-only) and gate the export cache-hit on it; delete + re-export a
bad cache entry. models.py: fail fast in BaseModel.optimize() on a 0-node
graph. Applies to every engine type (UNet/VAE enc+dec/ControlNet/safety).

Co-Authored-By: Claude Sonnet 5 <[email protected]>
@forkni
forkni requested a review from Copilot July 19, 2026 17:48

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@forkni
forkni merged commit 49d1d24 into SDTD_040_beta_release Jul 19, 2026
1 check passed
@forkni
forkni deleted the fix/onnx-cache-validation branch July 19, 2026 17:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants