feat(cco): honor reqs.sdmaQueueCount, clamp to HW max, guard out-of-range queueId#501
Open
jhchouuu wants to merge 1 commit into
Open
feat(cco): honor reqs.sdmaQueueCount, clamp to HW max, guard out-of-range queueId#501jhchouuu wants to merge 1 commit into
jhchouuu wants to merge 1 commit into
Conversation
…ange queueId reqs.sdmaQueueCount was declared but never consumed: the SDMA channel count came only from MORI_SDMA_NUM_CHANNELS because queues were materialized in ccoCommCreate, before reqs is available. - Defer cco SDMA queue setup to ccoDevCommCreate, where reqs is known. ccoSdmaSetupCommQueues now resolves the count from reqs.sdmaQueueCount (0 = env default) and is idempotent per comm (first DevComm fixes the count). - EnsureSdmaTransport takes the requested channel count and exposes the count actually connected via SdmaChannels(); cco adopts it so device-side indexing stays in bounds. shmem's no-arg call is unchanged (defaults to env). - Clamp requests above the CDNA per-pair hardware max (kMaxSdmaChannelsPerPair = 8) with a warning instead of aborting in hsaKmtCreateQueueExt. - Guard thread-scope put/commit/quietQueue against queueId >= sdmaNumQueue (out-of-range is now a safe no-op, not an OOB access). Co-Authored-By: Claude Opus 4.6 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
ccoDevCommRequirements.sdmaQueueCountwas declared but never consumed. TheSDMA channel count could only be set via
MORI_SDMA_NUM_CHANNELS, because thequeues were materialized in
ccoCommCreate— beforereqsis available — andccoDevCommCreatemerely copiedcomm->sdmaNumQueue. Settingreqs.sdmaQueueCounthad no effect.Additionally, thread-scope
put/commit/quietQueuedid no bounds check onqueueId, so aqueueId >= sdmaNumQueuewas undefined behavior (wrong-queueuse or an out-of-bounds handle/signal access).
Changes
reqs.sdmaQueueCount. SDMA queue setup is deferred toccoDevCommCreate, wherereqsis known.ccoSdmaSetupCommQueuesresolvesthe count from
reqs.sdmaQueueCount(0 = env default) and is idempotent percomm (the first DevComm fixes the count; later ones reuse the handles).
EnsureSdmaTransport(int requestedChannels = 0)connects with therequested count (0 =
MORI_SDMA_NUM_CHANNELS) and exposes the count actuallyconnected via
SdmaChannels(); CCO adopts it so device-side indexing stays inbounds. SHMEM's existing no-arg call is unchanged (defaults to env).
(
kMaxSdmaChannelsPerPair = 8, i.e. 4 queues/engine × 2 engines) are clampedwith a warning rather than failing in
hsaKmtCreateQueueExt.queueIdin thread-scopeput/commit/quietQueue:now a safe no-op. warp/block scope already clamp via the lane/thread → queue
mapping.
Test Plan
Validated on MI355X (gfx950, 2 GPU, own build) with a 2-rank harness that sets
reqs.sdmaQueueCountdirectly and probesdevComm.sdma.sdmaNumQueue:reqs.sdmaQueueCount = 8→ 8 distinct queues; put+quiet PASS.= 0, env unset → default 2; PASS.= 0,MORI_SDMA_NUM_CHANNELS=4→ 4; PASS.= 16→ clamped to 8 (warn, no crash); PASS.queueId(n+3) → recv untouched, kernel returns (no hang/fault).sdmaQueueCount) unchanged.mori_ccoandmori_shmemboth build with the newEnsureSdmaTransportsignature.
🤖 Generated with Claude Code