perf(knn): reduce memory for batch flat vector search

[LeoReeYang]

Summary

• Reduce memory usage of batch flat KNN when the final projection does not include the vector column (Closes perf: High memory usage for batched vector query on flat KNN #6940).
• Heap candidates no longer retain whole scan RecordBatches (and the full vector column) in the common columns=[...] case.
• When the vector column is projected, retain only a per-row copy of the vector (not the entire scan batch).

Implementation notes

• Scanner::flat_knn computes retain_vector from the final projection and plumbs it into KnnBatchParams.
• KNNVectorDistanceExec batch mode uses a two-tier candidate representation:
  • RowIdOnly { query_index, distance, row_id } when retain_vector=false
  • WithVector { ..., slim_batch, row_index, vector_row } when retain_vector=true

Memory (unit test)

Synthetic workload matching test_batch_knn_heap_memory: num_rows=4096, dim=512, m=10, k=10.

What is measured: sum of Arrow get_array_memory_size() over candidate heap entries (not peak RSS or end-to-end query memory).

Strategy	Candidate heap (array bytes)	vs baseline
Baseline (full scan batch per candidate)	842,167,200 (~803 MiB)	1×
RowIdOnly (no vector in projection)	0	≪ baseline (< baseline/50)
WithVector (per-row vector copy)	257,664 (~252 KiB)	~3,270× smaller (< baseline/10)

Notes:

• Baseline simulates pre-fix behavior: each of m×k candidates holds a shallow clone of the full scan batch (vector + _rowid for all num_rows). In this synthetic setup the same batch is cloned repeatedly, so the counter sums shared buffers per candidate (worst-case retention model).
• WithVector is on the order of m×k×d×4 (= 204,800 bytes for vector data) plus one deduplicated slim batch (_rowid only).
• Indexed batch KNN and other large columns are out of scope for this PR.

Latency (local, secondary)

Release build (uv run maturin develop --uv --release), use_index=false, with_row_id=true.

Workload: rows=200_000, dim=256, m=10, k=10 (medians over 8 rounds, 2 warmup).

	separate	batch	batch speedup
Baseline (upstream-main)	0.1262s	0.09365s	1.35×
This PR	0.1272s	0.05466s	2.33×

Batch latency improved ~1.71× vs baseline in this workload; separate-query path unchanged.

Test plan

• cargo test -p lance --lib batch_knn
• cargo clippy -p lance --tests -- -D warnings
• cd python && uv run pytest python/tests/test_vector_index.py -k batch

[codecov]

Codecov Report

❌ Patch coverage is 83.33333% with 50 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance/src/io/exec/knn.rs	82.31%	39 Missing and 10 partials ⚠️
rust/lance/src/dataset/scanner.rs	95.65%	0 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

[LeoReeYang]

Superseded by #6950 after branch rename to perf/batch-flat-knn-memory and rebase onto latest main.