Skip to content

feat: support HRNet DarkPose pose estimation (w32/w48, COCO body & wholebody)#258

Merged
jamjamjon merged 8 commits into
jamjamjon:mainfrom
wep21:hrnet
May 30, 2026
Merged

feat: support HRNet DarkPose pose estimation (w32/w48, COCO body & wholebody)#258
jamjamjon merged 8 commits into
jamjamjon:mainfrom
wep21:hrnet

Conversation

@wep21

@wep21 wep21 commented May 29, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds HRNet top-down heatmap-based pose estimation, supporting COCO body (17 keypoints) and COCO-WholeBody (133 keypoints) with the w32 / w48 backbones at 256x192 and 384x288 input resolutions.

What's included

  • src/models/vision/hrnet/impl.rsHRNet model. Reuses the top-down affine preprocessing (person crop → fixed input size), and decodes the model's heatmap output [batch, nk, h, w] via per-keypoint argmax plus quarter-pixel sub-pixel refinement (shift 0.25 toward the larger neighbor), then maps heatmap coordinates back to the crop using center/scale.
  • src/models/vision/hrnet/config.rsConfig::hrnet*() builders covering w32/w48 × body/wholebody × 256x192/384x288.
  • src/models/vision/hrnet/mod.rs, module registration and re-export in src/models/vision/mod.rs.
  • examples/pose-estimation/hrnet.rs + main.rshrnet subcommand with --width w32|w48, --is-coco, --hires, and --model (local file override). Uses a YOLO person detector for the top-down crops.

Verification

Built clean (cargo build --lib and cargo build --example pose-estimation) and ran end-to-end against a sample end2end.hrnet_w48 COCO-WholeBody ONNX (input [1,3,384,288] → output [1,133,96,72]) on assets/bus.jpg; rendered skeletons align accurately on all detected people.

cargo run --example pose-estimation -- --source ./assets/bus.jpg hrnet --width w48 --is-coco false --model ./end2end.hrnet_w48.onnx

🤖 Generated with Claude Code

Signed-off-by: wep21 <[email protected]>
@wep21 wep21 marked this pull request as draft May 29, 2026 08:17
wep21 and others added 3 commits May 29, 2026 17:20
@wep21 wep21 changed the title feat: support HRNet pose estimation (w32/w48, COCO body & wholebody) feat: support HRNet DarkPose pose estimation (w32/w48, COCO body & wholebody) May 29, 2026
- Add per-model hub_owner/hub_repo overrides to ORTConfig so a model can
  fetch weights from a GitHub release other than the default jamjamjon/assets,
  while preserving the existing dtype (fp16/etc.) candidate resolution.
- Point HRNet configs at the wep21/assets 'hrnet' release.
- Rename weight files to the usls/assets hyphen convention with a '-dark'
  qualifier so the DarkPose variant is identifiable
  (e.g. hrnet-w32-coco-256x192-dark.onnx).
- Fix hrnet_w32_133 to 256x192 input to match the available wholebody weight
  (w48 wholebody stays 384x288).

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
@wep21

wep21 commented May 29, 2026

Copy link
Copy Markdown
Contributor Author

HRNet (DarkPose) inference results

All weights are fetched on-demand from the wep21/assets hrnet release into the local cache (the per-model hub_owner/hub_repo override added in this PR). fp16 variants resolve automatically via --dtype fp16.

Source image: ./assets/bus.jpg. Person detection by YOLO; pose by HRNet.

COCO body (17 keypoints)

Variant Command Result
w32 · 256×192 cargo run --release --example pose-estimation -- hrnet --width w32 --is-coco true --hires false w32-256
w32 · 384×288 cargo run --release --example pose-estimation -- hrnet --width w32 --is-coco true --hires true w32-384
w48 · 256×192 cargo run --release --example pose-estimation -- hrnet --width w48 --is-coco true --hires false w48-256
w48 · 384×288 cargo run --release --example pose-estimation -- hrnet --width w48 --is-coco true --hires true w48-384

COCO-WholeBody (133 keypoints)

Variant Command Result
w32 · 256×192 cargo run --release --example pose-estimation -- hrnet --width w32 --is-coco false w32-wb
w48 · 384×288 cargo run --release --example pose-estimation -- hrnet --width w48 --is-coco false w48-wb

Add --dtype fp16 to any command to use the half-precision weights.

Verified on CPU (macOS). All six configs fetch, run, and annotate successfully. The fp16 variants were also confirmed (e.g. hrnet --width w48 --is-coco false --dtype fp16 resolves and runs …-dark-fp16.onnx, producing the same 133-keypoint result).

@wep21 wep21 marked this pull request as ready for review May 29, 2026 16:22
@wep21

wep21 commented May 29, 2026

Copy link
Copy Markdown
Contributor Author

@jamjamjon Hi, is it applicable to add this new pose heatmap model? Currently, onnx is fetched from my assets repo.

…hub approach

Adopts the HRNet DarkPose pose estimation model (w32/w48, COCO body &
wholebody, 256x192/384x288) contributed by @wep21 in PR jamjamjon#258.

Key adjustments to the original PR:

- Rejected the per-model `hub_owner`/`hub_repo` config fields to keep
  `ORTConfig` lean and consistent with the existing architecture.

- Instead, HRNet variants now use explicit GitHub release URLs pointing
  to `wep21/assets`.

- Fixed a URL bug in `hrnet_w48_17_384()` where the w48 variant
  incorrectly referenced a w32 ONNX file.

- Extended `ORTConfig::try_commit()` so that explicit GitHub release
  URLs also support automatic dtype variant lookup (e.g. `-fp16`,
  `_fp16`, `.fp16`). Previously this only worked for bare filenames;
  now both bare filenames and full URLs benefit from it.

This keeps the configuration API surface minimal while fully supporting
third-party model hosting and dtype-aware downloads.
@jamjamjon

Copy link
Copy Markdown
Owner

@jamjamjon Hi, is it applicable to add this new pose heatmap model? Currently, onnx is fetched from my assets repo.

Hi @wep21, thanks for the great PR! The HRNet model is definitely applicable and the code quality is solid. I merged it with a few adjustments:

  • Dropped the hub_owner/hub_repo fields to keep ORTConfig lean — explicit GitHub URLs work just as well for third-party assets.
  • Fixed a small URL bug in hrnet_w48_17_384() (was referencing w32 instead of w48).
  • Extended ORTConfig::try_commit() so explicit GitHub URLs also get automatic dtype variant lookup (e.g. -fp16), which previously only worked for bare filenames.

The ONNX files stay in your wep21/assets repo — no need to move them. Everything compiles clean and CI passes. Thanks again for contributing!

@jamjamjon jamjamjon merged commit 9c37153 into jamjamjon:main May 30, 2026
17 checks passed
@wep21 wep21 deleted the hrnet branch May 30, 2026 02:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants