Skip to content

Revert find_reshape_cont guard relaxation from PR#4858#5052

Open
tamahedi wants to merge 18 commits into
developfrom
fix/revert-reshape-cont-guards
Open

Revert find_reshape_cont guard relaxation from PR#4858#5052
tamahedi wants to merge 18 commits into
developfrom
fix/revert-reshape-cont-guards

Conversation

@tamahedi

@tamahedi tamahedi commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Motivation

There is this ticket right now: AIRADSW-665. When I did a regression search for it I found it came from migraphx and this PR #4858. After going through each changes in that PR I found the bug came from the changes in src/simplify_reshapes.cpp. I reverted those changes here in this PR.

Technical Details

src/simplify_reshapes.cpp has had its guards find_reshape_cont struct reverted to more strict checking because it allowed some checks on strides to go through that were causing TDR.

Changelog Category

Add a CHANGELOG.md entry for any option other than Not Applicable

    • Added: New functionality.
    • Changed: Changes to existing functionality.
    • Removed: Functionality or support that has been removed. (Compared to a previous release)
    • Optimized: Component performance that has been optimized or improved.
    • Resolved Issues: Known issues from a previous version that have been resolved.
    • Not Applicable: This PR is not to be included in the changelog.

@tamahedi
tamahedi requested review from a team and causten as code owners July 9, 2026 14:03
@Zhaeong
Zhaeong requested a review from TedThemistokleous July 9, 2026 14:22
@tamahedi
tamahedi requested review from TedThemistokleous and pfultz2 and removed request for TedThemistokleous July 9, 2026 15:57
tamahedi and others added 2 commits July 9, 2026 12:10
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
@gh-app-migraphx-bot-pr-write

gh-app-migraphx-bot-pr-write Bot commented Jul 9, 2026

Copy link
Copy Markdown
Test Batch New Rate (1216aa) Old Rate (c76451) Diff Status
torchvision-resnet50 64 1,272.86 3,139.16 -59.45% 🔴
torchvision-resnet50_fp16 64 2,250.68 6,636.66 -66.09% 🔴
torchvision-densenet121 32 726.66 2,693.97 -73.03% 🔴
torchvision-densenet121_fp16 32 4,375.82 4,527.37 -3.35%
torchvision-inceptionv3 32 1,248.11 1,796.21 -30.51% 🔴
torchvision-inceptionv3_fp16 32 2,759.26 2,815.29 -1.99%
cadene-inceptionv4 16 822.91 820.81 0.26%
cadene-resnext64x4 16 615.60 782.70 -21.35% 🔴
slim-mobilenet 64 6,252.44 8,384.58 -25.43% 🔴
slim-nasnetalarge 64 128.25 228.89 -43.97% 🔴
slim-resnet50v2 64 2,817.53 3,163.24 -10.93% 🔴
bert-mrpc-onnx 8 1,167.74 1,168.84 -0.09%
bert-mrpc-tf 1 143.11 489.70 -70.78% 🔴
pytorch-examples-wlang-gru 1 465.91 477.74 -2.48%
pytorch-examples-wlang-lstm 1 445.97 385.30 15.75% 🔆
torchvision-resnet50_1 1 377.57 754.75 -49.97% 🔴
cadene-dpn92_1 1 62.17 443.77 -85.99% 🔴
cadene-resnext101_1 1 360.65 366.34 -1.55%
onnx-taau-downsample 1 111.58 400.08 -72.11% 🔴
dlrm-criteoterabyte 1 nan 32.42 nan
dlrm-criteoterabyte_fp16 1 44.23 51.82 -14.63% 🔴
agentmodel 1 6,842.05 9,667.44 -29.23% 🔴
unet_fp16 2 17.05 57.07 -70.12% 🔴
resnet50v1_fp16 1 132.35 1,020.33 -87.03% 🔴
resnet50v1_int8 1 209.07 951.60 -78.03% 🔴
bert_base_cased_fp16 64 441.82 1,097.49 -59.74% 🔴
bert_large_uncased_fp16 32 318.44 345.45 -7.82% 🔴
bert_large_fp16 1 92.22 203.19 -54.61% 🔴
distilgpt2_fp16 16 2,030.55 2,086.55 -2.68%
yolov5s 1 122.53 561.85 -78.19% 🔴
tinyllama 1 18.49 45.80 -59.63% 🔴
vicuna-fastchat 1 44.10 43.20 2.09%
whisper-tiny-encoder 1 414.01 411.64 0.58%
whisper-tiny-decoder 1 410.11 405.55 1.12%
llama2_7b 1 20.96 20.81 0.72%
qwen1.5-7b 1 13.93 3.65 281.76% 🔆
phi3-3.8b 1 26.81 15.77 70.02% 🔆
llama3-8b 1 8.60 21.78 -60.49% 🔴
whisper-large-encoder 1 10.07 7.32 37.54% 🔆
whisper-large-decoder 1 110.79 105.64 4.87%
mistral-7b 1 23.88 23.74 0.60%
FLUX.1-schnell 1 792.77 130.96 505.37% 🔆

Regressions detected 🔴

@gh-app-migraphx-bot-pr-write

gh-app-migraphx-bot-pr-write Bot commented Jul 9, 2026

Copy link
Copy Markdown
Test Status Result
bert-mrpc-onnx PASSED: MIGraphX meets tolerance
bert-mrpc-tf PASSED: MIGraphX meets tolerance
pytorch-examples-wlang-gru PASSED: MIGraphX meets tolerance
pytorch-examples-wlang-lstm PASSED: MIGraphX meets tolerance
dlrm-criteoterabyte PASSED: MIGraphX meets tolerance
agentmodel PASSED: MIGraphX meets tolerance
unet PASSED: MIGraphX meets tolerance
resnet50v1 PASSED: MIGraphX meets tolerance
bert_base_cased_fp16 PASSED: MIGraphX meets tolerance
bert_large_uncased_fp16 🔴 FAILED: MIGraphX is not within tolerance - check verbose output
bert_large PASSED: MIGraphX meets tolerance
yolov5s PASSED: MIGraphX meets tolerance
tinyllama PASSED: MIGraphX meets tolerance
vicuna-fastchat PASSED: MIGraphX meets tolerance
whisper-tiny-encoder PASSED: MIGraphX meets tolerance
whisper-tiny-decoder PASSED: MIGraphX meets tolerance
distilgpt2_fp16 PASSED: MIGraphX meets tolerance
llama2_7b PASSED: MIGraphX meets tolerance
qwen1.5-7b PASSED: MIGraphX meets tolerance
phi3-3.8b PASSED: MIGraphX meets tolerance
llama3-8b PASSED: MIGraphX meets tolerance
whisper-large-encoder PASSED: MIGraphX meets tolerance
whisper-large-decoder PASSED: MIGraphX meets tolerance
mistral-7b PASSED: MIGraphX meets tolerance
FLUX.1-schnell PASSED: MIGraphX meets tolerance

@ivarusic-amd ivarusic-amd added Windows Related changes for Windows Environments bugfix Fixes a bug found in the code. labels Jul 10, 2026
ivarusic-amd and others added 7 commits July 10, 2026 11:20
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
migraphx::make_op("multibroadcast", {{"out_lens", {2, 2, 2, 3}}}), trans_x);
auto sub = m2.add_instruction(migraphx::make_op("sub"), rsp_y, mb);
auto rsp_out =
m2.add_instruction(migraphx::make_op("reshape", {{"dims", {1, 1, 4, 3, 2}}}), sub);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reshape should come before the sub.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When i do this for the current pass with the changes in the if statements for this PR, the optimize_resize_ndims_unequal test fails so I'd have to change back everything for the reshape to come before the sub.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Then removing the .lens() is not the right fix.

// auto add_ins = m2.add_instruction(migraphx::make_op("add"), y_trans, x_brcst);
// auto r = m2.add_instruction(migraphx::make_op("reshape", {{"dims", {2, 4, 6}}}),
// add_ins); m2.add_return({r});
// }

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We shouldnt be disabling the tests.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That one was failing with the old changes. It was added in the PR where the guards were originally added. #4858

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We shouldn't disable a test though - we can modify them if say the IR has changed but removing this isn't the right call since it ensures other runs/models using this matcher won't behave the same now.

@pfultz2

pfultz2 commented Jul 10, 2026

Copy link
Copy Markdown
Collaborator

Can you add the test case where find_reshape_cont crashes or hangs?

@tamahedi

tamahedi commented Jul 10, 2026

Copy link
Copy Markdown
Contributor Author

Can you add the test case where find_reshape_cont crashes or hangs?

add a test case for "customer model usecase "?

@tamahedi
tamahedi requested a review from pfultz2 July 10, 2026 16:59
@codecov

codecov Bot commented Jul 10, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #5052      +/-   ##
===========================================
+ Coverage    92.82%   92.89%   +0.06%     
===========================================
  Files          597      603       +6     
  Lines        32145    32449     +304     
===========================================
+ Hits         29838    30141     +303     
- Misses        2307     2308       +1     
Files with missing lines Coverage Δ
src/simplify_reshapes.cpp 98.15% <100.00%> (+<0.01%) ⬆️

... and 13 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

tamahedi and others added 3 commits July 13, 2026 14:57
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
tamahedi and others added 3 commits July 14, 2026 08:35
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
@tperry-amd

Copy link
Copy Markdown
Contributor

Can you add the test case where find_reshape_cont crashes or hangs?

add a test case for "customer model usecase "?

I think he means to distill the part of the model into a unit test case that can act as a minimum repro of the scenario. There are tools to slice onnx models or just generally construct some tensors/ops in a unit test that reproduce the failing case if you know what the failing graph structure is like.

@TedThemistokleous

TedThemistokleous commented Jul 22, 2026

Copy link
Copy Markdown
Collaborator

Can you add the test case where find_reshape_cont crashes or hangs?

add a test case for "customer model usecase "?

I think he means to distill the part of the model into a unit test case that can act as a minimum repro of the scenario. There are tools to slice onnx models or just generally construct some tensors/ops in a unit test that reproduce the failing case if you know what the failing graph structure is like.

@tperry-amd @tamahedi correct - we have examples of this in simplify_reshapes_test

If this is related to say some sort of onnx input or model you can add or generate the exact onnx using our gen_onnx.py script.- thats how all our onnx tests are generated.
https://github.com/ROCm/AMDMIGraphX/blob/develop/test/onnx/gen_onnx.py

Regardless you'll need to add the case you're seeing here, parse it to ensure no issues with parser and then create a new unit test in simpify_reshapes_test.cpp. If its not something within the operator but a pattern you're seeing reproduce it as MIGraphX IR

If its just a pattern you're trying to optimize - reproduce that into the new unit test you'll create in simplify_reshapes_test.cpp

Then you can build MIGraphX and run bin/test_simplify_reshapes_test -c from your build folder and that will run all the simplify reshape unit tests locally

Once that passes without regressions then run `test_optimize_module_test -c' this runs simplify_reshapes + simplify algebra and checks some patterns that use both matchers to ensure consistency and expected output.

A finally a sanity would be make -j$(nproc) check which would build all the unit tests locally and run them to give you an idea of any other gotchas. This will rerun the two tests but is much longer but will run any other passes that may interact with the matcher you modified.

Ensure you do this for a Release build flag, and Debug build flag incase any static_asserts trigger - CI will test this anyway (this is what our debug builds do) but its better to catch this early since you can test/verify everything locallt with the binary tests. If you're on linux you can use ccmake . if you have done apt install cmake-curses-gui which has a decent interface for selecting different compile flag with cmake

If you want to regen tests quickly you can also build make -j$(nproc) tests to just rebuild the unit tests to speed things up if you're just working on your test

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bugfix Fixes a bug found in the code. Windows Related changes for Windows Environments

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants