feat: add segmented BTree index merge_segments support

[zhangyue19921010]

Closes: #6979

Also refactored scalar index optimize logic in append.rs:

1. Unified merge_scalar_indices — consolidates all scalar-index merge logic from merge_indices_with_unindexed_frags into one function with explicit segment selection and a single capability-based decision tree.

2. Added BTreeIndex::merge_segments — a k-way segment merge primitive, enabling multi-segment consolidation without re-reading the dataset.

3. num_indices_to_merge is now honored uniformly for all scalar types — previously scalar indices ignored it and always merged into one segment. Now append (=0) builds a delta only for the unindexed fragments, merge_N merges the latest N segments, and default/merge_1 merges the newest delta — matching the inverted/vector semantics.

4. Multi-segment handling is capability-based, never a hard error — merge_scalar_indices picks per case:

   1. Append delta (no segment selected) → build a new segment over the unindexed fragments, keep the old ones.
   2. Supports N:1 merge (like BTree), ≥1 selected → k-way merge into one segment.
   3. No N:1 merge, exactly 1 selected → single-segment update().
   4. No N:1 merge, ≥2 selected → rebuild a fresh segment over the covered fragments (no k-way primitive, and lossy summary indexes can't re-emit rows).

Compatibility Testing

Context

Main branch ignores num_indices_to_merge for scalar indices — every mode merges the single old segment + new data → 1 segment. A multi-segment scalar index is a state that cannot exist on main.

Test Results (single-segment starting state — the only state main can produce)

Behavior is identical across all scalar types here (capability only matters once ≥2 segments exist):

Mode	Main	PR	Diff
default / merge_1 / merge_2	1 seg (param ignored)	1 seg (merge newest + new)	✅ Same result
append	1 seg (param ignored)	2 seg (delta only)	⚠️ Changed

The only behavior change is append mode: it now produces a delta segment instead of silently merging. All other modes are unchanged from main.

PR-only new behavior: multi-segment scenarios (unreachable on main)

After accumulating N segments via repeated append-mode optimizes (applies to all scalar types):

Starting state	Mode	Result
3 segments + new data	default / merge_1	3−1+1 = 3 segments (merge newest delta)
3 segments + new data	merge_2	3−2+1 = 2 segments
3 segments + new data	append	3+1 = 4 segments

For merge_2 (≥2 selected): types that support N:1 merge (like BTree) use the k-way merge primitive; types that don't consolidate via a full rebuild over the covered fragments. Both reach the same segment count.

Query correctness: All combinations verified — no data loss or incorrect results. Added tests for the delta-append, single-segment update, and multi-segment rebuild paths.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[codecov]

Codecov Report

❌ Patch coverage is 89.70013% with 79 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance/src/index/scalar/btree.rs	71.31%	25 Missing and 12 partials ⚠️
rust/lance-index/src/scalar/btree.rs	59.52%	12 Missing and 5 partials ⚠️
rust/lance/src/index/append.rs	96.15%	4 Missing and 13 partials ⚠️
rust/lance/src/index.rs	40.00%	6 Missing ⚠️
rust/lance/src/index/create.rs	98.59%	2 Missing ⚠️

📢 Thoughts on this report? Let us know!

[zhangyue19921010]

Hi @Xuanwo This pr is ready for review. please take a look. thanks!

[Xuanwo]

This makes a publicly reachable state fail during optimize. Multi-segment non-BTree scalar indexes can be committed today, but this guard rejects them instead of defining whether they should be unsupported at commit time or rebuilt during optimize.

[zhangyue19921010]

Nice Catch~

Removed the guard — optimize now handles multi-segment non-BTree gracefully instead of failing. merge_scalar_indices decides per case:

1. Append delta (no segment selected) → build a new segment over the unindexed fragments, keep the old ones.
2. Type supports N:1 merge, ≥1 selected → k-way merge into one segment.
3. Type not supports N:1 merge, exactly 1 selected → single-segment update.
4. Type not supports N:1 merge, ≥2 selected → rebuild a fresh segment over the covered fragments.

[zhangyue19921010]

Hi @Xuanwo Thanks a lot for your help and review. Comments are all addressed. and CI passed. PTAL~

[Xuanwo]

Thank you for working on this!