KV cache: seeded runs produce different logical file-I/O workloads

[jay-tau]

Running the KV cache benchmark twice with the same seed and workload settings did not produce the same filenames or file contents.

Tested on current main (2dc78d3):

python3 kv-cache.py \
  --model tiny-1b \
  --num-users 1 \
  --max-requests 4 \
  --duration 30 \
  --gpu-mem-gb 0 \
  --cpu-mem-gb 0 \
  --storage-capacity-gb 1 \
  --generation-mode none \
  --cache-dir "$CACHE_DIR" \
  --seed 42 \
  --output "$OUTPUT"

Both runs completed 4 requests and generated 1,243 tokens. Their file sizes matched, but their names and hashes differed:

run A:
conv_user_0000_1784555733870_turn_1.npy
141066368 bytes
sha256: 3a73b832...

run B:
conv_user_0000_1784555748474_turn_1.npy
141066368 bytes
sha256: 8e4412b2...

The immediate cause appears to be conversation.py:55, where the conversation ID contains the current wall-clock time. The ID becomes part of the cache filename and is also used when generating the file contents.

There are a few additional sources of nondeterminism:

• generator and processing threads share module-global RNGs;
• queue tie-breaking uses time.time();
• concurrent requests may already be in flight when --max-requests is reached.

For a fixed seed, configuration, input data, and request count, it would be useful to reproduce:

• request, conversation, and cache identities;
• relative filenames, file sizes, and contents;
• workload decisions such as token counts, prefix caching, and multi-turn behavior;
• the logical per-file read/write/delete pattern;
• the exact number of admitted requests.

This would still use the real storage backend. Latency, throughput, syscall chunking, physical block placement, and completion timing would remain nondeterministic.

Would the project be open to a PR adding this guarantee, either to --seed or through a separate deterministic-workload option?

[FileSystemGuy]

Thanks for the detailed report — I walked through the code on main (2dc78d3) and can confirm the diagnosis. Every source you list is real, and there's one more.

Root cause (dominant)

conversation.py:55:

conv_id = f"conv_{user_id}_{int(time.time()*1000)}"

The wall-clock millisecond is baked into the conversation ID, which then flows to two places:

1. Filename — the ID is the on-disk name (conv_user_0000_1784555733870_turn_1.npy), so names differ every run.
2. File contents — the turn key {conversation_id}_turn_N is passed to KVCacheGenerator._seed_from_key() (cache.py:42-49), which computes SHA256(key) XOR global_seed to seed the content RNG and XOR stamp. Since the key carries the timestamp, the derived content seed changes → different SHA-256 despite identical size.

That single line explains the exact symptom (matching sizes/token counts, differing names + hashes).

Notably, the correct pattern already exists in the same file: request_id is built from a locked monotonic counter — f"{user_id}_req_{req_id:04d}" (benchmark.py:293-309), not from a timestamp. The conversation ID is simply inconsistent with that; giving it a deterministic monotonic identity (mirroring request_id) is the primary fix — no new machinery needed.

Secondary sources (all confirmed)

• Global RNGs shared across threads — cli.py:437-438 seed the module-global random / np.random, but generation, per-user workers, process_requests, RAG ingest (rag.py:225), and a ThreadPoolExecutor all draw from those globals concurrently (benchmark.py:517,1226,1233,1327). Draw order is thread-timing-dependent, so re-seeding the globals alone won't make the sequence reproducible — the robust fix is a per-stream np.random.default_rng(...) derived from the stable ID (which cache.py already does for content) rather than global seeding inside threads.
• Queue tie-break on time.time() — benchmark.py:319,407,426: priority_tuple = (-priority, time.time()) into a PriorityQueue. On equal QoS priority, ordering falls to wall-clock. (Minor extra risk: if two time.time() values tie, the queue falls through to comparing the InferenceRequest objects, which can raise.) Replacing the tie-break with the monotonic request_counter value fixes both.
• In-flight requests at --max-requests — benchmark.py:635 stops on requests_completed >= max_requests, but the generator keeps producing and workers may be mid-flight, so the exact admitted set/count is timing-dependent.

One additional source

Conversation eviction is wall-clock-ordered. _evict_oldest_conversation() (conversation.py:113-119) sorts by (last_access, created_at), both datetime.now(). When max_conversations is exceeded, which conversation is evicted depends on wall-clock — affecting cache reuse in the logical workload. It won't fire in the 1-user/4-request repro, but matters for larger configs.

On the proposed guarantee

The scoping is right: a deterministic logical workload (request/conversation/cache identities, relative filenames, sizes, contents, per-file R/W/D pattern, admitted count) while leaving latency, throughput, syscall chunking, and completion timing nondeterministic. Making the admitted count reproducible interacts with --duration (inherently wall-clock), so I'd suggest exposing this as a distinct deterministic-workload mode rather than silently strengthening --seed, since it also constrains scheduling.

A PR along these lines would be welcome. Suggested shape:

1. Monotonic conversation ID mirroring request_id (removes the timestamp from name + content).
2. Per-conversation / per-request RNGs seeded from stable IDs; stop relying on process-global random / np.random inside worker threads.
3. Counter-based queue tie-break instead of time.time().
4. Deterministic eviction ordering.

[FileSystemGuy]

@jay-tau I agree with your analysis (see Claude's post just above). but I'm not sure that we should make a code change for this at this time. There are two reasons for my hesitance to change the code:

1. The non-determinism of the RNG affects the file names and their contents, but not their sizes. The benchmark as a whole is explicitly trying to ensure non-dedupeable and non-compressible data in order to level the playing field between solutions that do dedupe and compression with solutions that do not do them, so the changing data content and filenames shouldn't affect the workload itself.
2. We are very late in the submission window and attempting a fix that would result in full determinism would involve a much larger code change than I think we have time for.

@dslik @hazemawadalla Do you want to add anything to the discussion?

[dslik]

Based on the analysis, I do not believe that any of these identified run-to-run differences will affect results. Full determinism was not a design goal, just equivalence of results. If you think this results in major changes to the results (> 1%) across runs, that would be a higher priority, but still not something we would likely choose to address in the 3.0 release.

[FileSystemGuy]

@jay-tau Please close this after you see it if it answers your questions, or add more comments or questions here...

[jay-tau]

Would it be something that the team is open to merging in the next release cycle?
I do not believe it materially impacts performance, but it does help with making the seed more true to name, what what it currently does.

[dslik]

Yes, this is something we would consider merging after this submission round closes.

[FileSystemGuy]

@jay-tau @dslik Agreed, after we get passed the v3.0 submission round.