DISKANN 10M×1536-dim recall requires search_list=4000 to reach 95%, causing unacceptable latency and throughput

[Raysmond]

DISKANN 10M×1536-dim recall requires search_list=4000 to reach 95%, causing extreme latency and low throughput

Environment

Item	Value
Repo / branch	mlcommons/mlperf_storage, main (June 2026)
Milvus	v2.6.x, standalone Docker
Storage	Solidigm D7-PS1010 TLC 7.68TB NVMe (XFS)
Host	377 GB RAM, Xeon
Dataset	10,000,000 vectors × 1536-dim, uniform random
Metric	COSINE
Index	DISKANN, max_degree=64, search_list_size=200
Shards	10
Benchmark	vdbbench --runtime 300, 8 workers, batch_size=10

Problem

With a correctly-constructed DISKANN index (max_degree=64, the standard value; not the erroneous --max-degree default of 16), reaching 95% recall@10 on 10M × 1536-dim uniform vectors still requires search_list=4000. At this search depth, QPS collapses to 18 and per-query latency exceeds 400 ms — unacceptable for any practical workload.

Results

search_list	QPS	Recall@10	Mean (ms)	P99 (ms)
200	588	59.4%	13.3	14.6
400	356	70.7%	22.3	23.1
800	144	80.5%	55.3	58.4
1200	84	85.7%	94.6	102.6
2000	43	90.6%	183.5	207.5
4000	18	95.2%	435.2	511.2

Root Cause Hypothesis

1536-dimensional uniform vectors are nearly equidistant (nearest-neighbor distance / 100th-neighbor distance ≈ 0.96, vs ≈ 0.7 for SIFT 128-dim). Without clustering structure, DISKANN's graph traversal must explore a large fraction of the graph to find true neighbors — there are no shortcuts. This may be a fundamental limitation of graph-based ANN on high-dimensional uniform data, not a parameter-tuning issue.

Questions

1. Has anyone achieved 95% recall at search_list ≤ 200 on 10M×1536-dim uniform DISKANN? If so, what configuration?
2. Is this expected for uniform high-dimensional data? Should the benchmark use a more structured dataset?

[wvaske]

This is expected for uniform data. We did an analysis a while back that showed the access pattern for diskann with uniform data was the same as with real data. But it's is very different for hnsw. It's reasonable to use uniform synthetic data with diskann but we'll need a different process for any other index type.

…

On Wed, Jun 10, 2026 at 9:51 AM Raysmond ***@***.***> wrote: *Raysmond* created an issue (mlcommons/storage#425) <#425> DISKANN 10M×1536-dim recall requires search_list=4000 to reach 95%, causing extreme latency and low throughput Environment Item Value Repo / branch mlcommons/mlperf_storage, main (June 2026) Milvus v2.6.x, standalone Docker Storage Solidigm D7-PS1010 TLC 7.68TB NVMe (XFS) Host 377 GB RAM, Xeon Dataset 10,000,000 vectors × 1536-dim, uniform random Metric COSINE Index DISKANN, *max_degree=64*, search_list_size=200 Shards 10 Benchmark vdbbench --runtime 300, 8 workers, batch_size=10 Problem With a correctly-constructed DISKANN index (*max_degree=64*, the standard value; not the erroneous --max-degree default of 16), reaching 95% ***@***.*** on 10M × 1536-dim uniform vectors still requires *search_list=4000*. At this search depth, QPS collapses to 18 and per-query latency exceeds 400 ms — unacceptable for any practical workload. Results search_list QPS ***@***.*** Mean (ms) P99 (ms) 200 588 59.4% 13.3 14.6 400 356 70.7% 22.3 23.1 800 144 80.5% 55.3 58.4 1200 84 85.7% 94.6 102.6 2000 43 90.6% 183.5 207.5 *4000* *18* *95.2%* *435.2* *511.2* Root Cause Hypothesis 1536-dimensional uniform vectors are nearly equidistant (nearest-neighbor distance / 100th-neighbor distance ≈ 0.96, vs ≈ 0.7 for SIFT 128-dim). Without clustering structure, DISKANN's graph traversal must explore a large fraction of the graph to find true neighbors — there are no shortcuts. This may be a fundamental limitation of graph-based ANN on high-dimensional uniform data, not a parameter-tuning issue. Questions 1. Has anyone achieved 95% recall at search_list ≤ 200 on 10M×1536-dim uniform DISKANN? If so, what configuration? 2. Is this expected for uniform high-dimensional data? Should the benchmark use a more structured dataset? — Reply to this email directly, view it on GitHub <#425?email_source=notifications&email_token=AFCHLUL3MNG6BT3R5HE64WD47F7YVA5CNFSL4Z3JMQ5C6L3HNF2C22DVMIXUS43TOVSS6NBWGMZDOOBXHE2TJJTSMVQXG33OVJZXKYTTMNZGSYTFMSSWK5TFNZ2KYZTPN52GK4S7MNWGSY3L>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFCHLUJMOASTHCLVEEPOB4347F7YVAVCNFSNUABFKJSXA33TNF2G64TZHM2DKOJXGYZTMMBTHNEXG43VMU5TINRTGI3TQNZZGU2KC5QC> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[Raysmond]

Thanks @wvaske, that's helpful context -- it confirms that uniform data is fundamentally the issue here, not a parameter-tuning problem.

Building on this, I'd suggest we step back and ask: what is this benchmark actually measuring? With uniform 1536-dim data, no graph-based ANN index can achieve high recall at low search depth. The recall-vs-QPS curve is dominated by the "dimensionality curse," not by the storage system. So the benchmark becomes a contest of which system can push more IOPS — not which ANN + storage stack delivers the best recall-per-dollar or recall-per-GB-of-RAM.

I think we should look at how VectorDBBench approaches this. They use open-source raw datasets (LAION, SIFT1M/GIST1M, Cohere, etc.) that have natural clustering structure. With real data, recall saturates at much lower search depths, so the QPS numbers at a target recall (e.g., 95% @ 10) are meaningful — they reflect the index quality, not just raw storage throughput.

This matters especially for DiskANN, whose entire design rationale is:

Trade latency for memory savings. DiskANN puts the graph index on SSD and uses a small in-memory cache (SearchCacheBudgetGBRatio), aiming to serve ANN queries at competitive QPS with a fraction of the RAM that HNSW requires. The key question isn't "how fast can it go at sl=4000?" — it's "can it hit 95% recall@10 while using only 10–15 GB of RAM on a dataset where HNSW needs 80+ GB?"

On uniform data, you can't answer that question because the recall curve has no knee — you never reach a regime where recall is "good enough" at a reasonable search depth. Real datasets with clustering would let us measure:

Metric	What it tells you
Recall@k at fixed search_list	Index quality (graph construction)
QPS at 95% recall target	End-to-end performance at a meaningful operating point
RSS at 95% recall target	Memory efficiency — DiskANN's raison d'être
QPS per GB of RAM	Overall cost efficiency

Without a recall target, the benchmark just answers "which storage is faster?" — and that's a question you can answer with fio, not a vector database benchmark.

Potential proposal: adopt the VectorDBBench approach — use raw (not pre-indexed) open datasets with known ground truth, define per-index recall targets, and report throughput at those targets. This makes results comparable across index types (DISKANN vs HNSW vs AISAQ) and captures what users actually care about: "what recall can I get at my RAM budget?"

[wvaske]

The fundamental problem with open datasets is size scaling.

…

On Thu, Jun 11, 2026 at 10:02 PM Raysmond ***@***.***> wrote: *Raysmond* left a comment (mlcommons/storage#425) <#425 (comment)> Thanks @wvaske <https://github.com/wvaske>, that's helpful context -- it confirms that uniform data is fundamentally the issue here, not a parameter-tuning problem. Building on this, I'd suggest we step back and ask: *what is this benchmark actually measuring*? With uniform 1536-dim data, no graph-based ANN index can achieve high recall at low search depth. The recall-vs-QPS curve is dominated by the "dimensionality curse," not by the storage system. So the benchmark becomes a contest of which system can push more IOPS — not which ANN + storage stack delivers the best recall-per-dollar or recall-per-GB-of-RAM. I think we should look at how *VectorDBBench* approaches this. *They use open-source raw datasets* (LAION, SIFT1M/GIST1M, Cohere, etc.) that have natural clustering structure. With real data, recall saturates at much lower search depths, so the QPS numbers at a target recall (e.g., 95% @ 10) are meaningful — they reflect the index quality, not just raw storage throughput. This matters especially for *DiskANN*, whose entire design rationale is: Trade latency for memory savings. DiskANN puts the graph index on SSD and uses a small in-memory cache (SearchCacheBudgetGBRatio), aiming to serve ANN queries at competitive QPS with a fraction of the RAM that HNSW requires. The key question isn't "how fast can it go at sl=4000?" — it's "can it hit 95% ***@***.*** while using only 10–15 GB of RAM on a dataset where HNSW needs 80+ GB?" On uniform data, you can't answer that question because the recall curve has no knee — you never reach a regime where recall is "good enough" at a reasonable search depth. Real datasets with clustering would let us measure: Metric What it tells you ***@***.*** at fixed search_list* Index quality (graph construction) *QPS at 95% recall target* End-to-end performance at a meaningful operating point *RSS at 95% recall target* Memory efficiency — DiskANN's raison d'être *QPS per GB of RAM* Overall cost efficiency Without a recall target, the benchmark just answers "which storage is faster?" — and that's a question you can answer with fio, not a vector database benchmark. *Potential proposal:* adopt the VectorDBBench approach — use raw (not pre-indexed) open datasets with known ground truth, define per-index recall targets, and report throughput *at those targets*. This makes results comparable across index types (DISKANN vs HNSW vs AISAQ) and captures what users actually care about: "what recall can I get at my RAM budget?" — Reply to this email directly, view it on GitHub <#425?email_source=notifications&email_token=AFCHLUORGHPXJFPO27O6KWL47N6G7A5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTINRYG4ZDSNRRGU22M4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLDGN5XXIZLSL5RWY2LDNM#issuecomment-4687296155>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFCHLUPTVWCYWRU442WBCYL47N6G7AVCNFSNUABFKJSXA33TNF2G64TZHM2DKOJXGYZTMMBTHNEXG43VMU5TINRTGI3TQNZZGU2KC5QC> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[Raysmond]

The fundamental problem with open datasets is size scaling.
…

We want to make it clear that the v3 vdb benchmark will only support uniform datasets ?
And btw, do we have CLOSED submission rules for VDB tests?

[FileSystemGuy]

@idevasena Ping, can you confirm the uniform dataset?

@Raysmond Yes, we do have submission rules for CLOSED now, see section 5 in the Rules.md file.

[Raysmond]

@FileSystemGuy Thanks, Curtis. We've run some initial tests with the CLI commands, but we have a follow-up question regarding the methodology: Why is HNSW listed as an index_type for CLOSED scenarios? Since HNSW is fundamentally an in-memory indexing algorithm, the query process doesn't interact with the underlying disk or storage system. I’m curious to understand what exactly we are benchmarking when using HNSW in this context?