perf(index): reduce describe indices fragment scans

[hfutatzhanghb]

Summary

Optimize describe_indices() row-count aggregation by building a single fragment_id -> logical_rows map from manifest metadata and then iterating each segment's RoaringBitmap directly.

This avoids calling Dataset::get_fragments() inside the per-segment loop and adds trace-level instrumentation for the number of dataset fragments, segment fragment references, missing fragment references, and computed indexed rows.

Results

We use the same test envirnment to compare the optimize result. See below:

before optimization (425 seconds):

[2026-05-27T21:39:10] dataset_describe_indices describe_indices count=1 seconds=425.149

after optimization（0.701 seconds）

[2026-05-28T10:40:20] dataset_describe_indices describe_indices count=1 seconds=0.701

Why

The previous implementation computed rows_indexed by looping over every dataset fragment for every index segment:

for segment in index_segments:
    for fragment in all_dataset_fragments:
        if segment.fragment_bitmap.contains(fragment.id):
            rows_indexed += fragment.logical_rows

That makes the work proportional to num_index_segments * total_dataset_fragments. On a large table, for example 10,000 index segments and 1,000,000 fragments, this can turn a metadata-only description into billions of membership checks even when each segment covers only a small fragment bitmap.

There was also avoidable allocation in the inner loop. Dataset::get_fragments() clones the dataset into an Arc and clones each manifest fragment into a FileFragment wrapper. Because it was called once per segment, those clones and wrappers were rebuilt repeatedly while describing one logical index.

The new path builds fragment row metadata once with Dataset::iter_fragments() and then performs work proportional to:

total_dataset_fragments + sum(segment.fragment_bitmap.cardinality)

For lance-ray style scheduling, this keeps the driver-side metadata planning step much lighter before slicing work across workers.

Validation

• cargo fmt --all
• cargo test -p lance test_describe_indices_total_size -- --nocapture
• cargo clippy --all --tests --benches -- -D warnings
• git diff --check

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[hfutatzhanghb]

@Xuanwo @BubbleCal @jiaoew1991 Hi, could you please help review this PR when free? Thanks!

[codecov]

Codecov Report

❌ Patch coverage is 94.20290% with 8 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance/src/index.rs	94.20%	6 Missing and 2 partials ⚠️

📢 Thoughts on this report? Let us know!

[LuciferYang]

The ? operator propagates Error::internal if any fragment in the manifest lacks complete row metadata (physical_rows or num_deleted_rows). This iterates all fragments upfront, including those never referenced by any index bitmap. The old code only called fast_logical_rows() for fragments that appeared in the bitmap, so fragments with incomplete metadata were tolerated as long as they were not indexed.

[wjones127]

Realistically, any modern Lance dataset will have physical_rows or num_deleted_rows. Generally, though, we try to use the async version to be backwards compatible. If it's not too much trouble, I'd prefer we kept with that.

This iterates all fragments upfront, including those never referenced by any index bitmap. The old code only called fast_logical_rows() for fragments that appeared in the bitmap, so fragments with incomplete metadata were tolerated as long as they were not indexed.

TBH I don't this this is a huge deal. Realistically, you will have indexes covering most if not all fragments, so you'll use all or most of the fragment row counts. I think the speed benefit of collecting them into a contiguous allocation (versus computing on demand) might outweigh the cost of slight over-computation.

[wjones127]

We could keep this is as-is if we do the error message change I suggest in: https://github.com/lance-format/lance/pull/6958/changes#r3314011721

[hfutatzhanghb]

@wjones127 Thanks for your valuable suggestion. Have applied.

[hfutatzhanghb]

Hi, @wjones127 @LuciferYang After apply this optimization, we reduce describe_indices cost time significantly from 400+ second to 0.7 seconds within 500B rows lance dataset . I think this is a very useful PR.

[LuciferYang]

The single assert_eq!(desc.rows_indexed(), 4) exercises one fragment, no deletions, no missing-bitmap entries. Per the project standard ("Include multi-fragment scenarios for dataset operations"), at least one of the following should be covered, ideally in a separate #[tokio::test] to keep the existing test focused:

• Multiple appended fragments — verifies the HashMap aggregation sums across fragments.
• A fragment with deletions — verifies num_rows() returns physical_rows - num_deleted_rows, not raw physical rows.
• A stale segment bitmap (e.g., after compaction) referencing a fragment that no longer exists in the manifest — exercises the new missing_fragment_refs branch.

[hfutatzhanghb]

@LuciferYang Thanks for suggestions. Will add unit tests laterly.

[hfutatzhanghb]

@LuciferYang @wjones127 Have added unit tests. please check again. Thanks.

[LuciferYang]

+1, LGTM