Skip to content

DNM - dsr1 Mi355x test#1283

Closed
Oseltamivir wants to merge 88 commits intomainfrom
amd/mi355x-dsfp4-april14-copy
Closed

DNM - dsr1 Mi355x test#1283
Oseltamivir wants to merge 88 commits intomainfrom
amd/mi355x-dsfp4-april14-copy

Conversation

@Oseltamivir
Copy link
Copy Markdown
Collaborator

@Oseltamivir Oseltamivir commented May 5, 2026

No description provided.

billishyahao and others added 30 commits March 16, 2026 08:36
@billishyahao
[AMD] add dsr1 mxfp4 v2 sweep points
0383696
@billishyahao
fix
18e05b1
@ZhaiFeiyue @billishyahao
Fix tokenizer mismatch between benchmark client and sglang server on …
32b5d3d 
…transformers v5

Transformers v5 incorrectly rebuilds pre_tokenizer/decoder components for
models like DeepSeek-R1 that use LlamaTokenizerFast with a non-Llama
tokenizer architecture. The sglang server fixes this at startup, but the
benchmark client loads the tokenizer without these fixes, causing a ~5x
token count inflation (e.g. 7000 tokens -> 35000 tokens) and false
performance regressions in TTFT and throughput benchmarks.

Apply the same tokenizer fixes (pre_tokenizer/decoder restoration and
add_bos_token recovery) that sglang server applies, so client and server
tokenize identically. No-op on transformers v4.

Made-with: Cursor
@billishyahao
change mtp model to fp8
0bd347f
@billishyahao
change fp8 image
754e53c
@billishyahao
bump image to 0327
f29f2d0
@billishyahao
remove specv2
a44c7eb
@billishyahao
consolidate dsr1 fp4 configs
2514136
@billishyahao
Merge remote-tracking branch 'inf/main' into amd/mi355x-dsfp4-march15
3b4d4ab
@billishyahao
bump fp8 image to 0327
682a4ab
@billishyahao
fix crash
64bf100
@billishyahao
fix env
c44e175
@billishyahao
cleanup
0a41f89
@billishyahao
add perf change log
7282748
@billishyahao
add deprecate comments
e6d4b32
@billishyahao
add spec v2 env
b7dd65f
@billishyahao
bump the docker image
12a4ba0
@billishyahao
add stream control to eliminate cpu overhead
597a458
@billishyahao
tune the config
f715e47
@billishyahao
bump image
2ea82d5
@billishyahao
tune config
16384e7
@billishyahao
add new exp config
4d733e7
@billishyahao
enable log level info
83af743
@billishyahao
fix mori env
0c3083e
@billishyahao
bump image
1c61622
@billishyahao
fix log
e2d2ac9
@billishyahao
bump the image
d2a7988
@billishyahao
fix
b09ae6c
@billishyahao
fix
2c3ee04
@billishyahao
fix
69102f7
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 5, 2026

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=25395388503
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=25395388503

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 5, 2026

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=25397018357
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=25397018357

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 5, 2026

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=25399743027
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=25399743027

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 5, 2026

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=25401398398
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=25401398398

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 5, 2026

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=25402882560
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=25402882560

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 5, 2026

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=25403887062
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=25403887062

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 6, 2026

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=25409238836
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=25409238836

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 6, 2026

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=25413029415
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=25413029415

@Oseltamivir Oseltamivir force-pushed the amd/mi355x-dsfp4-april14-copy branch from 4eab18e to b8b305b Compare May 6, 2026 21:52
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 6, 2026

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=25464374364
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=25464374364

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 6, 2026

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=25466303569
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=25466303569

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 7, 2026

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=25467958241
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=25467958241

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 7, 2026

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=25468709876
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=25468709876

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 7, 2026

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=25469344105
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=25469344105

1 similar comment
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 7, 2026

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=25469344105
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=25469344105

@Oseltamivir Oseltamivir closed this May 7, 2026
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 7, 2026

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=25471538848
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=25471538848

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@claude claude[bot] claude[bot] left review comments

@billishyahao billishyahao Awaiting requested review from billishyahao billishyahao is a code owner

@chunfangamd chunfangamd Awaiting requested review from chunfangamd chunfangamd is a code owner

@seungrokj seungrokj Awaiting requested review from seungrokj seungrokj is a code owner

@yctseng0211 yctseng0211 Awaiting requested review from yctseng0211 yctseng0211 is a code owner

@1am9trash 1am9trash Awaiting requested review from 1am9trash 1am9trash is a code owner

Assignees

No one assigned

Labels

sweep-enabled

Projects

InferenceMAX Board
Status: Done

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@Oseltamivir @billishyahao @ZhaiFeiyue