feat(lance-linalg): runtime SIMD dispatch for pre-Haswell x86_64 from-source builds#2
Draft
tobocop2 wants to merge 189 commits into
Draft
feat(lance-linalg): runtime SIMD dispatch for pre-Haswell x86_64 from-source builds#2tobocop2 wants to merge 189 commits into
tobocop2 wants to merge 189 commits into
Conversation
5 tasks
tobocop2
changed the title
tobocop2
changed the title
tobocop2
added a commit
that referenced
this pull request
Apr 26, 2026
…ch dot_u8.rs convention Per-function doc comments on `*_scalar`/`*_avx`/`*_avx_fma`/`*_avx2`/`*_avx512` inner functions are now one-liners matching the existing convention in `dot_u8.rs` (see e.g. its `pub unsafe fn dot_u8_avx2` comment). Drops the redundant "Caller must ensure..." precondition lines — those are implicit from `unsafe fn` + `#[target_feature]`. Module-level `//!` docs and public-API `///` docs are left detailed per project convention. Refs #1, #2.
This was referenced Apr 26, 2026
tobocop2
force-pushed
the
fix/runtime-simd-multiversion
branch
from
April 28, 2026 04:59
a3df856 to
9193496
Compare
tobocop2
changed the title
tobocop2
force-pushed
the
fix/runtime-simd-multiversion
branch
from
April 28, 2026 05:43
58325e8 to
7db5171
Compare
tobocop2
force-pushed
the
fix/runtime-simd-multiversion
branch
from
April 28, 2026 05:46
7db5171 to
26aa7f4
Compare
tobocop2
force-pushed
the
fix/runtime-simd-multiversion
branch
3 times, most recently
from
June 10, 2026 16:56
126a6a5 to
26a520b
Compare
…p-table (lance-format#7361) ## What The mem-wal primitives sophon's drop-table two-phase commit needs: - **`ShardWriter::abort(&self)`** — shut down the background flush tasks (`task_executor.shutdown_all()`) *without* flushing, discarding buffered memtable state. Unlike `close(self)` it takes `&self` (so it's callable through the `Arc<ShardWriter>` callers hold) and does no object-store IO. The caller must quiesce writes first (documented). Idempotent. Acked data is **not** lost — it's durable in the WAL log and replays on the next claim, which is what makes the drop's prepare phase reversible. - **`ShardStatus { Active | Sealed }` on `ShardManifest`** — a durable, reversible lifecycle marker (proto + struct + serde). `claim_epoch` refuses a `Sealed` manifest with a **distinguishable** error instead of minting a new epoch, so a shard mid-drop can't be re-claimed — even by a caller that skips its own status check — and a reader can tell an in-doubt drop apart from an ordinary epoch fence. Set/cleared through the existing epoch-guarded `commit_update` CAS; carried across claims via `..base`, so only the genuinely-fresh constructions default it to `Active`. ## Why A WAL-enabled table's drop spans two durable resources — the owning pod's fresh-tier state and the catalog/object-store data — so sophon's teardown is a two-phase commit. Before the dataset directory is removed, the owning pod must: - `abort` the writer so its background flush task can't re-create `_mem_wal/` under the just-deleted directory (a graceful `close()` would flush it back), and - durably mark the shard `Sealed` so the drop is in-doubt across a pod crash or a Maglev rehome — which the in-memory fence flag cannot survive. The seal is reversible (rollback clears it back to `Active`), making the prepare phase abortable without data loss. Both build on existing machinery (`shutdown_all`, the manifest CAS) — thin exposure, not new infrastructure. > **Changed since first draft:** this PR previously also added `Session::invalidate_dataset`; it has been **dropped**. Base-table read freshness rides the recreate's new object-store `e_tag` (the QN→PE and lance metadata caches are `e_tag`-keyed and miss the stale entry), so cache invalidation isn't load-bearing — only `abort` and the `Sealed` marker remain. ## Tests - `test_abort_discards_without_flushing_and_is_idempotent` — `abort` leaves no new L0 generation (contrast with `close`), idempotent on a second call. - `test_claim_epoch_refuses_sealed_manifest` — a `Sealed` manifest is refused with the distinguishable error and left untouched (no epoch minted); rolling the status back to `Active` makes the shard claimable again (reversibility). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
## What Adds `LsmPointLookupPlanner::lookup_keep_tombstone` and `lookup_many_keep_tombstone` — point-lookup variants that carry the `_tombstone` marker through instead of filtering deleted keys out. ## Why The existing `lookup` / `lookup_many` collapse two distinct states — **deleted in the fresh tier** and **never written** — into a single `None`. That's correct for a normal reader, but a caller doing a read-on-write merge needs to tell them apart: a fresh-deleted PK must be treated as *absent* (so a later partial update resurrects it from carried columns + NULLs), while a never-written PK falls back to the base row. These variants return the deleted row with `_tombstone = true`; absent keys still return `None`. Implemented by refactoring `plan_lookup` to share a `plan_lookup_coalesced` helper and skipping the post-coalesce tombstone filter, so the hot read path is unchanged. ## Tests - `test_lookup_keep_tombstone_returns_deleted_row_with_marker` - `test_lookup_many_keep_tombstone_includes_tombstoned_keys` ## Context Prerequisite for a downstream WAL partial-column-update resurrection fix: a partial update arriving after a delete must resurrect carried columns + NULL untouched, never stale pre-delete base data. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
lance-format#7495) ## What The `lance-index` **test build** is currently broken on `main`: `error[E0046]: not all trait items implemented, missing: with_io_priority` at `rust/lance-index/src/scalar/fmindex.rs`. `with_io_priority` was added to the `IndexStore` trait in lance-format#7449. The `FailNewFileStore` test fixture was added later in lance-format#7422 (which branched before lance-format#7449 landed), so it never implemented the new method. Each PR was green on its own; together on `main` the test build fails — a semantic merge conflict. It only surfaces in the `Build tests` step because the fixture is `#[cfg(test)]`. ## Fix Implement `with_io_priority` on `FailNewFileStore` by delegating to the inner store and re-wrapping, mirroring how the fixture already delegates its other read methods while keeping the injected `new_index_file` failure. ## Verification - `cargo check -p lance-index --tests` — compiles (was failing on `main`). - `cargo test -p lance-index --lib fmindex` — 24 passed. - `cargo fmt -p lance-index -- --check` and `cargo clippy -p lance-index --tests` — clean.
## What changed This PR narrows OSS-1346 to codec replacement work only. It adds Lance-owned `BitPacker4x` and internal `BitPacker8x` implementations in `lance-bitpacking`, and switches existing FTS posting-list users from the external `bitpacking` runtime dependency to `lance-bitpacking`. The external `bitpacking` crate remains only as a dev-dependency for byte-for-byte compatibility tests against the legacy 4x/8x formats. ## Why FTS currently depends on an external bitpacking crate for 128-value posting blocks. Keeping the implementation in `lance-bitpacking` lets Lance own the codec while preserving existing 128-block wire compatibility. The 8x implementation is included as an internal algorithm building block only; this PR does not expose a new block-size option. ## How it works - Exposes `BitPacker` / `BitPacker4x` from `lance-bitpacking` with the same call shape used by existing FTS code. - Adds owned backends for 128-value 4x blocks: - SSE3 on `x86_64` - NEON on `aarch64` - scalar fallback when SIMD backends are unavailable - Adds an internal 256-value `BitPacker8x` algorithm: - AVX2 on `x86_64` - NEON on little-endian `aarch64` - scalar fallback when AVX2 or NEON is unavailable - not re-exported from `lance-bitpacking` - Does not add or change FTS `block_size` parameters, metadata, Python, Java, or docs. ## Validation - `cargo fmt --all` - `cargo test -p lance-bitpacking -- --nocapture` - `cargo check -p lance-index --tests` - `cargo check -p lance --tests` - `cargo clippy -p lance-bitpacking --tests -- -D warnings` - `cargo clippy -p lance-index --tests -- -D warnings` - `cargo clippy -p lance --tests -- -D warnings` - `cargo clippy --all --tests --benches -- -D warnings` - `cargo metadata --manifest-path python/Cargo.toml --locked --format-version=1 --no-deps` - `CARGO_TARGET_DIR=/tmp/lance-target-oss1346-aarch64 cargo check -p lance-bitpacking --tests --target aarch64-apple-darwin` - `CARGO_TARGET_DIR=/tmp/lance-target-oss1346-x86 cargo check -p lance-bitpacking --tests --target x86_64-apple-darwin` - `git diff --check`
…#7483) ## What Adds `ShardWriter::delete_no_wait`, the delete analog of `put_no_wait`: it inserts the tombstone into the in-memory tier (visible to reads on this writer the instant it returns) and hands back the durability watcher *without* awaiting it. `delete` becomes a thin wrapper that awaits the watcher. ## Why Lets a caller hold an external lock across only the in-memory tombstone insert and await durability after releasing it — matching how `put_no_wait` is already used so flushes still coalesce. A downstream WAL serializes deletes against partial-update merges under a per-bucket lock; with the blocking `delete` it would hold that lock across durability, and `delete_no_wait` lets it release first. ## Tests - `test_shard_writer_delete_no_wait_visible_before_durability` 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
) The Merge arm only dropped indices for fields removed from the schema, so a field rewritten in place (new backing data file, same field id) kept its index covering the fragment with stale entries. Index-served queries then silently returned wrong results -- a filter on the rewritten column matches the old values rather than the current data. Prune the rewritten fragment from any index over a field whose backing data file changed, reusing the same primitive the DataReplacement arm already uses.
## Summary Add `lance/arrow.py` and `tests/test_arrow.py` to the pyright include list and fix the resulting type errors. Closes lance-format#3292. - **arrow.py**: narrow union types via typed locals / `typing.cast`, annotate the internal decoder return types, switch the fallback to `pyarrow.compute as pc`, and add a single `# pyright: ignore[reportOptionalCall]` for `tf.stack` (typed as `None` by tensorflow's incomplete stubs). - **lance/lance/__init__.pyi**: fix the `bfloat16_array` stub signature from `List[str | None]` to `Sequence[float | None]` — the Rust function takes `Vec<Option<f32>>`, and `Sequence` (covariant) lets `list[float]` callers type-check. - **test_arrow.py**: add `import lance.arrow` so the `lance.arrow.*` attribute accesses resolve. ## Test plan `tensorflow` is linux-only in `pyproject.toml`, so it was installed locally to mirror the CI (Linux) lint environment before checking. - `pyright` → **0 errors** (15 warnings, all pre-existing missing-stub warnings for pyarrow/pandas/etc.) - `ruff format --check` + `ruff check` → pass - `pytest python/tests/test_arrow.py` → **20 passed**
This fixes low recall for IVF_SQ and IVF_HNSW_SQ indexes using Dot distance when the scalar quantization bounds include negative values. The previous SQ Dot path computed dot distance directly in quantized u8 code space. That drops the affine offset from scalar quantization, so zero-centered embeddings can be ranked incorrectly. The updated path computes Dot distance against the dequantized SQ value model while keeping the stored-code HNSW path allocation-free and able to use the u8 dot kernel. The change also adds regression coverage for a crafted negative-bound SQ Dot case and end-to-end IVF_SQ / IVF_HNSW_SQ Dot recall on negative-valued vectors.
…format#7484) ## Problem An indexed merge-insert delete by a composite primary key whose columns are **all** indexed silently removes nothing. `merge_insert(when_matched = Delete, use_index = true)` reports `deleted = 0`, the matched rows stay live in the table, and resurface in later reads. ## Root cause `WhenMatched::Delete` is only implemented in the v2 plan (`DeleteOnlyMergeInsertExec`), which never uses a scalar index. When every join column is indexed, `can_use_create_plan` routes the merge to the legacy `Merger` instead (the only engine with the indexed-scan probe). That engine's matched-row handler only ever distinguished `DoNothing` from "update" — it folded **both** `Delete` and `Fail` into the update path: - a fully-indexed `Delete` rewrote the matched rows in place → 0 deletes; - a fully-indexed `Fail` silently updated instead of erroring. A single indexed column hits the same path, so this was never composite-specific — composite keys just make it the common case (partial-column updates require an index on every PK column). ## Fix Make the legacy `Merger` dispatch every `WhenMatched` variant explicitly, mirroring the v2 classifier (`merge_insert_action`): - **`Delete`** collects matched row ids as deletions and emits no replacement batch. - **`Fail`** aborts on any match, with the same message as the v2 path. - A **delete-only commit branch** drains the merger and applies the deletions (resolving row ids → addresses via the row-id index for stable row ids) without writing fragments — this keeps the O(keys) indexed delete instead of falling back to a full table scan. - The partial-schema (column-rewrite) commit branch physically cannot express row deletions, so combining `Delete` with inserts from a partial-schema source now returns a descriptive error rather than silently dropping the deletes. ## Tests `cargo test -p lance --lib dataset::write::merge_insert` (152 passing), covering: - composite-key indexed delete (index on first / second / both / neither — the both-indexed case is the original bug); - single-column indexed delete; - multi-fragment + stable-row-id variants, including an appended fragment neither index covers (unindexed-remainder union); - `Delete` combined with `InsertAll` (not delete-only); - `Fail` on an indexed key (match aborts, no-match inserts cleanly). `cargo fmt` and `cargo clippy -p lance --lib --tests -- -D warnings` are clean. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
…-format#7223) ## Summary `LabelListIndexPlugin::train_index` unconditionally rejected fragment-scoped training (`fragment_ids.is_some()`), so a LabelList index could not be built through the distributed / segmented path (per-fragment `execute_uncommitted` + `merge_existing_index_segments`) the way BTree, Inverted, Bitmap, and FM already are. This makes LabelList a first-class distributed scalar index. ## Changes - `lance-index/src/scalar/label_list.rs`: `train_index` ignores `fragment_ids` and builds over the already fragment-scoped stream (mirrors `FMIndexPlugin`); a partial index over a fragment subset is correct since it covers exactly those rows. Add `merge_label_list_indices`, which unions the per-segment bitmap states and the `list_nulls` row sets. LabelList wraps a `BitmapIndex` plus a null-row set and distributed segments cover disjoint rows, so this is a cheap union (the same operation `LabelListIndex::update` performs) — no source re-scan. Mirrors `merge_bitmap_indices` but also carries `list_nulls`. - `lance/src/index/scalar/label_list.rs` (new): `merge_segments` opens each source segment as a `LabelListIndex` and calls `merge_label_list_indices` (mirrors `scalar/bitmap.rs`). - `lance/src/index.rs`: `merge_existing_index_segments` routes `all_label_list` segments (details `LabelListIndexDetails`) to the new merge; add `segment_has_label_list_details`. - `lance/src/index/scalar.rs`: declare the `label_list` module. Covered by `test_label_list_merge_existing_index_segments`: one LabelList segment per fragment, merged via `merge_existing_index_segments`, answers `array_has_any` across both fragments with the same row count as a pre-index full scan.
Summary: - Improve FM `contains` queries so normal reads demand-load needed wavelet blocks and page nearby blocks instead of prewarming whole partitions. - Add chunked explicit FM prewarm, parallel partition loading/search, and resumable partition builds. - Add FM `contains` benchmark and FM index management tooling. Optimizations applied: - Parallel FM search across partitions/segments. - Removed query-time full-partition prewarm; explicit prewarm still fully warms the index. - Chunked contiguous wavelet-row prewarm with `LANCE_FMINDEX_PREWARM_CHUNK_BYTES` and `LANCE_FMINDEX_PREWARM_CHUNK_CONCURRENCY`. - Cold-read demand paging with `LANCE_FMINDEX_DEMAND_PAGE_BYTES` to reduce object-store RPS. - Parallelized FM metadata/partition loading. - Added resumable FM partition creation and explicit `--index-uuid` recovery support. - Final benchmark layout: 1 logical FM segment, `LANCE_FMINDEX_PARTITION_ROWS=100000`, large partition-byte cap, 1,000 partitions. Performance: Dataset: 100M-row `az://datasets/mmlb/mmlb_100m_fts_en_fm_20260626.lance`. Query workload: 4 sampled 5-term patterns from `summary_in_image`, `contains(full_content, pattern)`, `k=100`, `_rowid` only (`projection=[]`, `row_id=true`). | Run | Index layout | Prewarm | Query result | | --- | --- | --- | --- | | Baseline | 6 segments / 10k partitions | 6,525s / 108.8m | 1t: 1.58 qps, mean 633ms, p95 768ms; 8t: 12.48 qps, mean 320ms, p95 320ms | | Demand-load + chunked prewarm | 6 segments / 10k partitions | 2,182s / 36.4m, 3.0x faster | 1t: 5.38 qps, mean 185ms, p95 307ms; 8t: 12.12 qps, mean 326ms, p95 330ms | | Single segment | 1 segment / 100k-row partitions | 356s / 5.94m, 18.3x faster than baseline | 1t: 22.38 qps, mean 44.6ms, p95 54.4ms; 8t: 84.40 qps, mean 42.8ms, p95 47.3ms | Index size: - Final FM index UUID `78600545-625f-40e2-8790-204f35097ec0`: 1,000 partition files, 920,064,767,792 bytes / 856.9 GiB / 0.837 TiB. - Previous 6-segment FM layout was 920,202,650,467 bytes / 857.0 GiB, so the final relayout is effectively size-neutral.
…nce-format#7717) A valid empty value is stored as {position: 0, size: 0} while nulls pack their non-zero rep/def levels into position. The page scheduler skips size-0 rows when scheduling reads, but the load task assigned read results to every def == 0 blob, so an empty value consumed the next blob's payload and the last consumer crashed on an exhausted iterator ("Expected option to have value"). Give empty values their zero-length bytes at scheduling time and only assign read results to blobs that scheduled a read. Fixes lance-format#7716 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Fixed decoding/round-trip handling for empty blob (`LargeBinary`) values so they’re preserved and remain correctly aligned when mixed with non-empty payloads and nulls. * Prevented empty blob byte data from being overwritten during page loading and ensured empty values are treated as present during decoding. * **Tests** * Added an async round-trip test covering empty (`LargeBinary`) values interleaved with nulls when blob metadata is enabled. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
Hoisting the tier choice out of the per-vector loop is only half the job. The
first attempt also replaced the kernel with the auto-vectorized scalar one,
on the theory that an AVX2-baseline build makes the SIMD kernel redundant.
Benchmarks say otherwise: at dim 8, `L2(simd,f32x8)` went from -41.6% (before
this work) to +156% against upstream/main. Removing a cheap branch is worthless
if it costs the kernel behind it.
The baseline already guarantees avx2+fma, so call the AVX+FMA kernel directly:
no runtime check, and the `#[target_feature]` contract is satisfied statically,
so it inlines into the loop.
Measured on an AMD EPYC 7B13 (avx2, fma, no avx512f), against upstream/main:
before this work with this commit
L2(simd,f32x8) dim 8 -41.6% -45.4%
L2(f32, simd) dim 1024 +1.5% +0.4%
Dot(f32, SIMD) -0.1% -0.5%
The iterator stays a bare `Map`: `Map<ChunksExact, _>` is `TrustedLen`, so
`.collect()` preallocates, and `Map::fold` drives `ChunksExact` in one inlined
loop. Any wrapper — trait object or enum — loses both.
`cfg(target_feature = "avx2")` does not imply `fma` — `-C target-feature=+avx2` sets one and not the other. The batch path guarded on it calls a kernel declared `#[target_feature(enable = "avx,fma")]`, so such a build emitted `vfmadd` while claiming a baseline without FMA. Gate the direct-kernel arm on both features, and make the runtime-dispatch arm its exact complement so the two remain total. Verified by building the crate at `+avx2`, `+avx2,+fma`, `+avx`, `target-cpu=haswell`, and the default baseline.
Bumps the pinned toolchain from 1.94.0 to 1.97.0 in `rust-toolchain.toml` (the `python/` and `java/lance-jni/` copies are symlinks to it) and fixes the clippy lints newly reported by 1.97 across the workspace. New lints fixed: - `manual_filter`, `manual_checked_ops`, `question_mark` - `useless_conversion`, `useless_borrows_in_formatting` - `collapsible_match`, `while_let_loop`, `explicit_counter_loop`, `unnecessary_sort_by` All fixes are behavior-preserving. `cargo clippy -- -D warnings` is clean across the main workspace (all features, all targets), `python/`, and `java/lance-jni/`; `cargo fmt --check` passes on all three. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved handling of floating-point `NaN` values in range queries for more conservative, reliable results. * Prevented potential division-by-zero issues in decoding and statistics calculations. * Preserved correct behavior when processing all-null data and missing fields. * **Maintenance** * Updated the Rust toolchain used to build the project. * Simplified internal processing across indexing, scanning, encoding, storage, and query-planning components without changing expected behavior. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
`dot_scalar`/`l2_scalar` chunk by 16 lanes. At or below that width the chunking
degenerates to the scalar remainder loop and vectorizes nothing, which is why
the explicit AVX kernel is worth ~40% at dim 8 — the width PQ sub-vectors use.
Above 16 lanes the autovectorizer is already good, and on Zen 3 the 8-wide
kernel loses to it.
Measured on an AMD EPYC 7B13 (avx2, fma, no avx512f), two repetitions each,
against upstream/main:
kernel everywhere this commit
L2(simd,f32x8) -42.2% / -41.4% (unchanged, kernel)
Dot(f32, SIMD) +1.9% / +3.6% (scalar, == base)
So above the threshold keep exactly the kernel the pre-dispatch code used. The
wide path is then byte-identical to base and cannot regress; the narrow path
keeps the win. The branch is loop-invariant.
## Performance issue No-position FTS prewarm currently decodes posting rows into a `Vec<PostingList>` object graph. This substantially amplifies the index cache for workloads dominated by singleton and small posting lists. Measured before this change: - 10M globally unique terms: 2.76 GB cache for a 239.6 MB index (11.53x) ## How this improves performance This PR: - replaces index-time posting-group metadata with runtime synthetic groups, so existing indexes benefit without rebuilding; - stores no-position prewarmed groups as compact Arrow-backed posting rows instead of materializing every posting list; - creates a lightweight posting-list view only for the term used by a query, sharing the cached posting buffers; - keeps score and length metadata in the posting reader for prewarmed groups; - retains the materialized fallback for legacy and position-bearing paths; - versions the cache codec while continuing to read the previous materialized group representation. The persistent FTS index format and public API do not change. `prewarm_index` still populates the posting cache for subsequent queries. ## Benchmark Fresh `c4-standard-16` VM, no-position format-v2 indexes created once by baseline Lance `5dbd1400d`, then reused unchanged by the target. The target uses the default runtime group size of 128. Each prewarm result is the mean of five runs. | Dataset | Main cache | Target cache | Change | Cold prewarm | |---|---:|---:|---:|---:| | 10M unique terms | 2.762 GB | 250.3 MB | -90.94% | 6.440s -> 0.895s | | Wikipedia-40M | 8.423 GB | 3.724 GB | -55.79% | 15.697s -> 6.860s | A 64/128/256 sweep selected 128: compared with 256 it halves cache-group granularity for a 2.36% cache increase on 10M unique terms and 0.29% on Wikipedia-40M. Size 64 increased the 10M cache by 7.07% and cold prewarm time by 30.8%. Three repeated, prewarmed query runs at group size 128 showed no materialization regression. Mean QPS changed by -0.7% to +0.6% across the 10M unique-term and Wikipedia k=10/k=100 workloads, within run-to-run variance. ## Validation - `cargo fmt --all -- --check` - `cargo test -p lance-index scalar::inverted --lib --profile release-with-debug` (197 passed) - `cargo clippy --all --tests --benches -- -D warnings` - same-index benchmark validation: every query run reported `index_action=reused` and `prewarm_with_position=false` Addresses OSS-1409. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added runtime synthetic posting-list grouping for non-legacy v2 indexes. * Introduced a more compact packed posting-list cache format and packed-group prewarming. * **Bug Fixes** * Preserved backward-compatible decoding by falling back to the legacy layout. * Added consistency validation for packed cache contents and ensured BM25 correctness. * **Refactor** * Removed persisted posting-group boundary bookkeeping; grouping is now computed at runtime. * **Tests** * Updated and extended tests for packed vs materialized groups, including roundtrip and zero-copy coverage. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Yang Cen <[email protected]>
…ance-format#7682) ## What & why `lance-core` keeps a process-global CPU thread-pool runtime behind `CPU_RUNTIME: AtomicPtr<Runtime>`. `global_cpu_runtime()` dereferenced that raw pointer into `&'static mut Runtime`. Its only caller, `spawn_cpu`, runs concurrently on many worker threads, so multiple `&'static mut` references to the same `Runtime` could be live at once — a violation of `&mut` uniqueness, which is undefined behavior on the Rust abstract machine even though every `Runtime` method actually used takes `&self` and never mutates. It is latent UB the compiler is permitted to miscompile under noalias assumptions (Miri flags it), not an observed crash today. ## How & why it is correct Return `&'static Runtime` and dereference with `&*ptr` instead. Three facts make this sound: - The only method called is `Runtime::spawn_blocking`, which takes `&self` — the `&mut` was never needed. - `Runtime: Sync`, so many threads may hold `&'static Runtime` concurrently without UB, which is exactly the aliasing the old code got wrong. - The runtime is created via `Box::into_raw` and never reclaimed, so the `&'static` lifetime is genuine (no use-after-free); `atfork_tokio_child` only nulls the pointer, it does not free. Both `unsafe` derefs now carry `// SAFETY:` comments documenting these invariants. ## Compatibility Fully backward compatible, zero behavior change. `global_cpu_runtime` is private, so the return-type change is internal-only; `spawn_cpu`'s public signature is unchanged. `&mut Runtime` already auto-reborrowed to `&self` for `spawn_blocking`, so narrowing to `&Runtime` is byte-for-byte identical at runtime — purely tightening a gratuitous, unsound `&mut` into a shared `&`. ## Test plan - `cargo test -p lance-core` - `cargo clippy -p lance-core --tests -- -D warnings` - `cargo fmt --all -- --check`
…at#7723) ## Bug Fix ### What is the bug? `test_simple_index_nearest_centroid` builds a small HNSW index in parallel and then requires an exact nearest-centroid result. HNSW node insertion mutates the shared graph concurrently, so Rayon scheduling can produce different graph topologies even though node-level RNG is seeded. ### What incorrect behavior does the bug cause? The approximate search occasionally fails the strict `id == 42` assertion and returns a nearby centroid such as 43 or 44. This makes the test flaky, especially under CPU contention, without indicating a production regression. ### How does this PR fix the problem? Build the test's 100-point HNSW index inside a dedicated single-thread Rayon pool. This preserves the exact assertions while making graph construction deterministic. Production index construction and the binary nearest-centroid test are unchanged. ## Validation - `cargo test -p lance-index test_simple_index_nearest_centroid -- --nocapture` - 3,000 concurrent stress-test runs with 16 processes and `RAYON_NUM_THREADS=8`: 0 failures - `cargo fmt --all -- --check` - `cargo clippy -p lance-index --tests -- -D warnings` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Tests** * Updated vector index testing to run index construction within a controlled single-threaded environment. * No user-facing behavior changes. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Co-authored-by: Yang Cen <[email protected]>
) Adds a specification for **data overlay files**: small files attached to a fragment that supply new values for a subset of `(row offset, field)` cells without rewriting the base data files. They make cell-level updates cheap when only a small fraction of rows and/or columns change. This PR is **spec + proto only** — no read/write implementation yet. It is also explicitly *experimental*. The released libraries will not produce tables with this feature enabled. Once the implementation is done in the library, we will vote on the final design before releasing. This is similar to how we have done file format updates. ## Changes - **`protos/table.proto`** - Rework `DataOverlayFile`: a `oneof coverage { bytes shared_offset_bitmap | FieldCoverage field_coverage }` to support both dense (rectangular) and sparse overlays; add the `FieldCoverage` message. - Rename `read_version` → `committed_version` (`uint64`), with effective/commit-stamped semantics so overlay-vs-index ordering is correct. - Drop the in-file offset key column in favor of rank-based addressing off the coverage bitmap. - Document reader feature flag `64` (and previously-undocumented `16`/`32`). - **`docs/src/format/table/data_overlay_file.md`** (new): full specification — coverage/resolution, deletion precedence, NULL-override, layout + rank addressing, dense vs. sparse, versioning, field-aware index exclusion with flat re-evaluation, the correctness invariant, both compaction modes, row lineage, a worked example (write → read → index query → sparse write → read → compaction), and a guidance stub with open questions. - **`docs/src/format/table/index.md`**: concise overview + link to the new spec (replacing the earlier inline sketch). ## Out of scope / follow-ups - Write transaction shape (new `Operation` variant in `transaction.proto` + Rust). - Writer support for unequal-length columns (needed for single-file sparse overlays). - Coverage bitmap external spill for very large coverage. - Per-fragment vs. per-table overlays / LSM analogy (open question in the doc). 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added documentation for experimental Data Overlay Files, including storage, versioning, querying, compaction, and transaction behavior. * Added format and transaction schema support for describing data overlays. * Added navigation links to the new documentation. * **Bug Fixes** * Datasets using unsupported data overlays are now explicitly rejected instead of risking stale results. * **Documentation** * Expanded guidance on invalidated index results and overlay-related filtering scenarios. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]> Co-authored-by: Weston Pace <[email protected]>
…alue buffers (lance-format#7460) Closes lance-format#7459 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **Bug Fixes** - Zstandard compression now correctly honors the configured compression level for large variable-width values. - Improved consistency between compression settings and the resulting encoded data. - **Tests** - Added coverage to verify the compression level is preserved for large per-value Zstandard compression. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
…cy (lance-format#7632) ## Background & Motivation Under concurrent-scan workloads like Spark, a single large worker (Executor) typically runs many Tasks in parallel (in Lance's read path, **1 Task = 1 fragment = 1 scan**). The problem: **each Task independently sizes its own scan concurrency from the core count of the current Executor / host**. Concretely in Lance, the decode concurrency of the v2 read path `FilteredReadExec` (plan name `LanceRead`) — the `try_buffered(num_threads)` — is unconditionally set to `get_num_compute_intensive_cpus()` = `num_cpus − 2`. ``` total decode concurrency per Executor ≈ (concurrent Tasks) × (num_cpus − 2) ``` ## Why It Was Deprecated, and the Gap It Left Behind `Scanner.batch_readahead` is marked `Ignored in v2 and newer format` — a deliberate decision, not an oversight. In v1 it controlled two things: **prefetch depth** and **decode concurrency**. v2 replaced prefetch with a byte-budget model (`io_buffer_size` + `fragment_readahead`), so its prefetch role became meaningless and was rightly dropped. The gap: v2 split the **decode-concurrency** role into `FilteredReadExec`'s threading mode (`try_buffered(num_threads)`), hard-coded to `get_num_compute_intensive_cpus()`. The only override today is the process-wide `LANCE_CPU_THREADS` env var — far too coarse, since it governs *every* compute-intensive path at once (vector/KNN search, index building, `take`, update/merge-insert, …), not just scan decode concurrency. With no per-scan knob, there's no way to rein in the over-parallelization above. ## What This Change Does & Its Impact Have `new_filtered_read` pass `Scanner.batch_readahead` through as `FilteredReadThreadingMode::OnePartitionMultipleThreads(batch_readahead)`, reattaching `batch_readahead` to the decode-concurrency dimension that v2 had left without a knob. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Invalid `batch_readahead` / `batchReadahead` values (0 or less) are now rejected consistently across Java, Python, and Rust. * Error messages now clearly state `batch_readahead` must be greater than 0. * Filtered scan execution now applies `batch_readahead` to control decoding parallelism more reliably. * **Tests** * Strengthened `batch_readahead` tests to validate deterministic scanned content in addition to row counts. * Added Rust coverage for default behavior, custom values, and rejection of zero-parallelism configurations. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
## Feature Linear: [OSS-1344](https://linear.app/lancedb/issue/OSS-1344/make-fts-index-block-size-configurable) ### What is the new feature? FTS inverted index creation now accepts a `block_size` parameter for compressed posting blocks. Supported values are `128` and `256`. ### Why do we need this feature? The posting block size was previously fixed at `128`, which made the block-max granularity impossible to tune for different datasets and query profiles. ### How does it work? - Adds `block_size` to `InvertedIndexParams`, protobuf details, posting-list schema metadata, and cache headers. - Uses `128` as the default for newly created indexes. - Treats older serialized params, schema metadata, and cache entries that omit `block_size` as legacy `128`. - Rejects unsupported values, including `512`, with a clear validation error. - Uses Lance-owned `BitPacker4x` for physical 128-value posting blocks and `BitPacker8x` for physical 256-value posting blocks. - Marks `block_size=256` as experimental in public API docs because it may introduce breaking changes. - Keeps position-stream packing on the legacy 128-value block format. - Keeps downgrade compatibility tests on explicit legacy `block_size=128`, since older wheels cannot read current-created physical 256 FTS posting blocks. - Threads the configured block size through FTS build, read, iterator, WAND, cache, and MemWAL flush paths. - Exposes the parameter in Python and Java FTS index creation APIs, with docs and focused tests. ## Validation - `cargo fmt --all` - `cargo fmt --all --check` - `git diff --check` - `CARGO_TARGET_DIR=/tmp/lance-target-a479-no512 cargo test -p lance-index block_size -- --nocapture` - `CARGO_TARGET_DIR=/tmp/lance-target-a479-no512 cargo clippy -p lance-index --tests -- -D warnings` - `uv run make build` from `python/` - `uv run pytest python/tests/test_scalar_index.py::test_create_scalar_index_fts_block_size` from `python/` - `uv run ruff format --check python/tests/test_scalar_index.py python/lance/dataset.py` from `python/` - `uv run ruff check python/tests/test_scalar_index.py python/lance/dataset.py` from `python/` - `CARGO_TARGET_DIR=/tmp/lance-target-a479-merge-main cargo test -p lance-index block_size -- --nocapture` - `CARGO_TARGET_DIR=/tmp/lance-target-a479-merge-main cargo test -p lance-index test_256_posting_block_uses_single_physical_bitpack_chunk -- --nocapture` - `CARGO_TARGET_DIR=/tmp/lance-target-a479-merge-main cargo test -p lance-bitpacking` - `CARGO_TARGET_DIR=/tmp/lance-target-a479-merge-main cargo clippy -p lance-bitpacking -p lance-index --tests -- -D warnings` - `uv run ruff format --check python/tests/compat/test_scalar_indices.py` from `python/` - `uv run ruff check python/tests/compat/test_scalar_indices.py` from `python/` - `uv run pytest --run-compat -vvv -s python/tests/compat/test_scalar_indices.py::test_FtsIndex_downgrade --durations=30` from `python/` - `CARGO_TARGET_DIR=/tmp/lance-a479-target cargo test -p lance-index test_new_training_request_defaults_missing_block_size_to_128` - `CARGO_TARGET_DIR=/tmp/lance-a479-target cargo test -p lance-index block_size` - `uv run ruff format --check python/lance/dataset.py` from `python/` - `uv run ruff check python/lance/dataset.py` from `python/` Not run locally: Java focused test / spotless check, because this machine has no Java Runtime installed (`Unable to locate a Java Runtime`). --- ## Update: all V3 breaking changes consolidated here Per review direction, every breaking change for the 256-doc block format now lands in this single PR (the follow-up stack lance-format#7602/lance-format#7603/lance-format#7604/lance-format#7624/lance-format#7625/lance-format#7629 carries none). On top of the configurable block size and PFOR frequency encoding, this PR now also includes: - **Quantized doc-length scoring (Lucene norm semantics), 256-doc blocks only.** BM25 doc lengths are quantized to a SmallFloat-style byte code (4 mantissa bits: 0-7 exact, <= 6.25% relative error, decode = bucket floor). The byte-norm slab bakes lazily per loaded DocSet and quarters the doc-length bytes scoring pulls through the cache (200M docs: 800MB -> 200MB). 128-block indexes keep exact-length scoring bit-for-bit. Measured top-k overlap vs exact scoring on the (score-clustered, synthetic) mmlb corpus: 98.1% mean for phrase, 89.7% for 3-word AND; corpora with more score spread shift less. - **256-doc posting blocks drop the leading block-max-score f32** (~1.5G on a 200M-doc index; 131G -> 130G). Block layout: `[first_doc u32][doc num_bits u8][docs][pfor freqs]`; `posting_block_score_prefix_len(block_size)` keys every reader/writer. The impact skip data from the stacked lance-format#7602 supplies a tighter per-block bound; until it lands, 256-block block-max pruning falls back to the (valid, looser) list-level max score. **BREAKING:** 256-doc-block (v3) indexes must be rebuilt; v3 is unreleased so no migration is provided. BM25 scores on v3 differ from exact-length BM25 by the norm quantization, matching Lucene's norm semantics. The format discussion lance-format#7606 documents the final layout and scoring semantics. Additional validation for this update: bulk-vs-classic A/B under quantized scoring is score-identical (both paths quantize identically); the full stack's warm benchmarks vs Lucene 10.4 on mmlb-200m: OR k10 0.0249s/318qps and OR k100 0.0467s/170qps (both ahead of Lucene sliced), AND k10 0.0443s (1.29x), AND k100 0.0883s (1.94x). --- ## Standalone results vs main (per-branch-tip wheels) **Legacy (128) read-path parity.** Threading a runtime `block_size` through `PostingIterator` initially replaced the compile-time `BLOCK_SIZE` division (a shift) with real `div` instructions in the `doc()`/`next()` hot loops, measured as +11-14% on 3-word OR against the 200M legacy index (`PostingIterator::next` grew from 16.5% to 23.6% of the profile). Block sizes are validated powers of two, so the iterator now derives block indices with `trailing_zeros` shifts and masks; after that fix the legacy path is at parity with main: 3-word OR k10 0.131s (main 0.132-0.134s), k100 0.255-0.256s (main 0.256-0.257s), single-term 0.025s (main 0.027s) across 3 warm passes on the 200M legacy index, 400G cache. **block_size=256 index size** (5M-doc controlled build, same wheel, `with_position=false`): postings shrink **3.33 GiB → 2.62 GiB (−21%)** from PFOR frequencies + no per-block max-score prefix + half the block headers. Index build 192s → 210s (+10%, PFOR encode cost). Top-10 overlap vs the 128 exact-length scoring on 10 3-word OR queries: 95% mean (5/10 identical sets; the rest differ by 1-2 near-tie docs, from the quantized-norm scoring). **Query wins for 256 land in the stacked PRs.** A 256 index without impact skip data prunes on the (valid, looser) list-level max and is *slower* than 128 — e.g. classic AND k10 0.547s until lance-format#7602's impacts restore block-granular bounds (0.115s), and lance-format#7603/lance-format#7604/lance-format#7624/lance-format#7625/lance-format#7629 take the same index to 0.025s OR k10 / 0.045s AND k10. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added configurable FTS posting `block_size` (128/256) to scalar index creation, including updated examples and APIs. * Enabled FTS format version 3 (`v3`) for `block_size=256`, with quantized doc-length scoring for v3. * **Bug Fixes** * Enforced `block_size`/`format_version` compatibility (invalid combinations now error). * Persisted and restored FTS metadata for format version and posting block size, with legacy indexes defaulting to `128`. * **Documentation** * Updated full-text-search and quickstart guides and parameter docs for `block_size`, defaults, accepted values, and the experimental `256`/`v3` behavior. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Yang Cen <[email protected]> Co-authored-by: Claude Fable 5 <[email protected]>
## Summary This adds scan-based support for `List<Blob>` and nested blob leaves such as `Struct<List<Blob>>`. Descriptor scans preserve the original Arrow nesting and expose blob leaves as descriptors without loading payload bytes. `BlobHandling::AllBinary` materializes only blob leaves to binary values while keeping the surrounding `List` / `Struct` layout intact. This keeps the existing one-blob-per-row APIs unchanged and avoids adding a public random-access API for list blob values. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added Blob V2 binary view support for list and nested fields, with correct scan/filtered read/take output schema behavior. * Improved Blob V2 payload resolution for inline, dedicated, packed, and external sources. * **Bug Fixes** * Stricter Blob V2 detection and more consistent conversion/unload behavior. * Improved projection and schema intersection handling for Blob V2, including nested cases and better type-ignore behavior. * Preserved nulls, list offsets, and descriptor validity; ensured the binary-view marker is handled consistently. * **Tests** * Expanded Blob V2 coverage for binary/view materialization, descriptor projection, and nested list/struct scenarios. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
Lance releases frequently, which can make users worry that the Lance format itself is unstable. This PR updates the README to distinguish SDK/API release cadence from the dataset `data_storage_version` compatibility contract. It explicitly states that stable storage versions remain readable by future Lance releases, that a dataset's storage version is fixed at creation, and that `next` is only for experimentation.
Part of lance-format#7750 This isolates and names the existing dense mini-block rep/def page-budget decision so later sparse auto-selection can consume an honest contract. It renames the planner result and related encoder methods and fields without changing the underlying decision or page splitting. There is no format or behavior change. Validation: - `cargo fmt --all` - `cargo test -p lance-encoding` (427 passed, 5 ignored; doc-tests: 0 passed, 1 ignored) - `cargo clippy -p lance-encoding --tests --benches -- -D warnings` - `cargo clippy --all --tests --benches -- -D warnings` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved handling of pages with large repetition and definition-level data. * Prevented encoding failures when a single row exceeds mini-block limits. * Added safer fallback behavior for oversized rows during page encoding. * Improved page splitting to keep data within encoding budgets while preserving correctness. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
Part of lance-format#7750 Inline bitpacking mini-block decompression currently enters the non-empty unchunk path for zero values, which expects an inline bit-width header and panics on an empty payload. Return an empty `FixedWidthDataBlock` with the configured bit width instead. This is the generic zero-value codec prerequisite used when sparse pages project to no leaf values. It contains no sparse format or layout behavior. Validation: - `cargo fmt --all -- --check` - `cargo test -p lance-encoding encodings::physical::bitpacking::test` - `cargo test -p lance-encoding` - `cargo clippy -p lance-encoding --tests --benches -- -D warnings` - `cargo clippy --all --tests --benches -- -D warnings` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Fixed decompression for empty bitpacked data blocks. * Empty miniblocks now return correctly formatted empty results without attempting to read missing metadata. * **Tests** * Added coverage to verify empty miniblock handling. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
Labeling incoming issues by hand as bug / feature / performance is tedious. This applies the label automatically through three complementary layers, so it works no matter how an issue is opened. The aim is reliable labeling, not enforcing issue structure. ## 1. Issue templates (web UI) Templates under `.github/ISSUE_TEMPLATE/`, each auto-applying its label: - **Bug report** → `bug` — a light form: description, reproduction, Lance version, language binding (env and logs optional). - **Feature request** → `feature` — free-form. - **Performance issue** → `performance` — free-form. `config.yml` keeps blank issues enabled and routes usage questions to Discord. ## 2. Content-based labeler (everything else) Templates only fire for the web-UI chooser. Issues opened via `gh issue create`, the REST API, or an agent bypass them and land unlabeled. `.github/workflows/issue-labeler.yml` closes that gap: it runs `srvaroa/labeler` (the same pinned action the PR labeler uses) on `issues` events, keyed by `.github/labeler-issues.yml`. - Primary signal: a leading marker in the **title** (`bug:`, `[feature]`, `perf -`, …), which people commonly type by hand. - Fallback: body keywords (panic/crash → bug, regression/OOM → performance, …). - Unmatched issues get **no** label and are left for human triage rather than guessed at. `appendOnly` means it never strips a template/human label and is idempotent on edit. ## 3. Agent contract (AGENTS.md) A template can't bind an agent, so `AGENTS.md` now instructs agents to pass an explicit `--label bug|feature|performance` (and prefix the title) when opening issues. ## Notes - Templates can't be previewed from a diff — worth clicking through the "New issue" chooser on a fork before merging. - Left blank issues enabled so no one is forced through a template. - A future refinement could add an LLM classify step in the labeler workflow for the unmatched residual; out of scope here. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Documentation** * Added structured templates for bug, feature, and performance issue reports. * Added guidance for filing issues, including recommended labels and title prefixes. * **Chores** * Enabled automatic issue labeling based on titles and descriptions. * Added a contact link directing usage questions to the community Discord. * Enabled blank issue creation. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
…ance-format#6980) # Summary Fixes an HNSW-IVF index build bug analogous to lance-format#6957, but in the legacy HNSW partition writer path. # Changes - Keeps the `TempStdDir` guard alive for the full `write_hnsw_quantization_index_partitions` function. - Prevents scratch `hnsw_part_*` files from disappearing before they are read back into the final index files. - No new test added: existing `index::vector::ivf::tests::test_create_index_nulls` already covers `IVF_HNSW_PQ` and `IVF_HNSW_SQ` with `IndexFileVersion::Legacy`. # Testing - `cargo test -p lance test_create_index_nulls -- --nocapture` - Result: 10 passed, including the legacy `IVF_HNSW_PQ` and `IVF_HNSW_SQ` cases. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved reliability when building IVF_HNSW indexes with concurrent per-partition jobs. * Ensured temporary scratch directories and intermediate files persist correctly during in-flight work, and are cleaned up consistently after errors. * Improved error propagation so index build failures stop background work promptly and report late task issues accurately. * **Tests** * Added Tokio-based coverage for scratch-directory lifetime, failure draining/abort behavior, and cleanup under isolated temporary directories. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
Builds on the merged configurable posting block size work in lance-format#7466. Format discussion: lance-format#7606. Store per-block `(freq, doc_len)` impact frontiers alongside compressed posting blocks, plus one level-1 entry per 32 blocks, and drive block-max WAND pruning from them instead of build-time scores that can become stale as index statistics drift. - Both 128- and 256-doc blocks use the same compact impact codec: quantized `u8` document-length norms, delta-encoded frequencies, and an omitted norm byte for the common `+1` delta. - Scorer-specific bounds bake once into cache-shared state; query-local caches reuse the slab without sharing stale bounds across different corpus statistics. - Impact data survives packed prewarm and persistent-cache round trips. Posting-list cache versions are bumped while older versions remain readable. - Packed posting views share both impact-derived state and the block-head cache introduced by lance-format#7466. - Malformed or missing impact entries fall back to conservative infinite bounds instead of enabling unsafe WAND skips. - Public posting cache-key struct literals remain source-compatible; impact-bearing entries use an internal namespaced key. - V3 indexes without impacts retain the finite BM25 ceiling, while custom scorers without a declared safe bound fall back to `INFINITY`. The query benchmark below predates this codec follow-up; the V3 scoring bounds are unchanged, but impact size and decode cost need to be remeasured. ## Benchmark Measured before this restack against lance-format#7466 using per-branch-tip wheels: 200M-doc V3/256 index, 24 partitions, 1000 warm queries at 8 concurrent requests. | query | lance-format#7466 list-max fallback | this PR | |---|---|---| | OR 3w k10 | 0.363s / 22 qps | **0.103s / 76 qps (3.5x)** | | OR 3w k100 | 0.392s / 20 qps | **0.198s / 40 qps (2.0x)** | | AND 3w k10 | 0.547s / 15 qps | **0.115s / 69 qps (4.8x)** | | AND 3w k100 | 0.558s / 14 qps | **0.245s / 33 qps (2.3x)** | ## Validation - `cargo test -p lance-index`: 773 passed, 2 ignored; doctest passed - After the final rebase to current `main`, `cargo test -p lance-index --lib scalar::inverted`: 249 passed - `cargo check --workspace --tests --benches` - `cargo clippy --all --tests --benches -- -D warnings` - `cargo fmt --all -- --check` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Full-text search now optionally uses impact skip data to improve block-max scoring and pruning when index partitions provide it. - Impacts are carried through compressed and packed posting data, with impacts-aware WAND routing and scorer-weighted bound caching. - **Bug Fixes** - Improved decoding/validation of impacts envelopes, including safe behavior for malformed, null, or truncated data. - More robust posting component extraction and cache-key isolation to prevent mixing impact vs non-impact partitions. - **Tests** - Expanded roundtrip, cache isolation, and backward/forward compatibility tests for impact-enabled and legacy postings. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Yang Cen <[email protected]>
…at#7758) ## Summary `hamming_clustering_for_ivf_partition` and `get_ivf_partition_info` resolved the target index with `find(name)` — the first physical segment only. On a logical IVF_FLAT index made of multiple segments (delta segments from append-mode `optimize_indices`, or distributed builds committed via `commit_existing_index_segments`), this silently processed only the oldest segment: appended rows were excluded and cross-segment duplicate pairs were never compared, with no error raised. Both functions now cover **all** segments of the logical index. Delta segments of one IVF index share centroids, so partition `p` denotes the same centroid region in every segment; the correct clustering unit is the union of partition `p`'s rows across all segments in a single pairwise pass (representative = min row id, so results are order-independent). ## Changes - **Rust** (`lance-index::vector::hamming`): all-segments behavior for both functions. Every selected segment is validated to share byte-identical global IVF centroids; a mismatch (or missing centroids) raises a descriptive error naming the diverging segment UUIDs and partition counts, advising a retrain. New `hamming_clustering_for_ivf_partition_segments` / `get_ivf_partition_info_segments` accept explicit segment UUIDs, following the `prewarm_index_segments` precedent (lance-format#7677). - **Python**: keyword-only `index_segments` argument on `hamming_clustering_for_ivf_partition` and `get_ivf_partition_info` (`None` = all segments, the default). Segment-id normalization used in three places (these wrappers, `prewarm_index`, `ScannerBuilder.with_index_segments`) is consolidated into `lance.util._normalize_index_segment_ids`. ## Notes - Fixes a latent bug: `get_ivf_partition_info` returned `size: 0` for every partition on v3 index files because the loaded IVF model carries no partition lengths. Sizes now come from partition storage via `VectorIndex::partition_size`. - Behavior tightening: `get_ivf_partition_info` now validates the indexed column is `FixedSizeList<UInt8, 8>`, a check it previously skipped. This is consistent with the clustering function, which always required 8-byte hashes. - An empty explicit segment selection is rejected rather than silently returning "no duplicates", to avoid a silent data-quality hazard in dedup pipelines. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added support for running IVF_FLAT Hamming clustering and partition analysis across multiple index segments. * Added optional segment selection to limit clustering and partition statistics to specific physical segments. * Segment identifiers now accept UUID strings or UUID objects with validation for invalid selections. * **Bug Fixes** * Improved handling and validation of index segment identifiers across dataset and scanner operations. * Added validation to ensure selected segments are compatible before processing. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
…lance-format#7535) > **Supersedes lance-format#7407.** Relocated into `lance-format/lance` (head + base in-repo) so the OSS-1324 PR can stack on it directly and show a clean, specific diff. Data model, feature flag, and write/commit foundation for [Data Overlay Files](lance-format#7381) (OSS-1322). **Stacked on lance-format#7381** (spec/proto). To keep this diff clean, the base is an in-repo mirror of lance-format#7381's branch (`will/data-overlay-spec-base`); retarget to `main` once lance-format#7381 lands. (lance-format#7406 / OSS-1323, needed for the sparse overlay sink, has merged.) ## What's here - **Data model** (`lance-table/src/format/overlay.rs`) — `DataOverlayFile` + `OverlayCoverage` on `Fragment.overlays`, dense (`shared_offset_bitmap`) and sparse (per-field). Lives in its own module: a small public surface (the two types, `dense`/`sparse`, `coverage_for_field`) with the coverage / rank / parse-once / newest-last invariants documented at the module level and the serde/proto/Roaring plumbing kept private. Coverage bitmaps are parsed once on load into `Arc<RoaringBitmap>` (not re-deserialized per access), and a fragment's overlays are stable-sorted by `committed_version` (newest last) on load so resolution can rely on the ordering. Protobuf/serde round-trip + `coverage_for_field` tested. - **Feature flag 64** (`FLAG_DATA_OVERLAY_FILES`) — set when any fragment has overlays. Release-gated: treated as an unknown flag in release builds (so release readers/writers refuse overlay datasets) unless `LANCE_ENABLE_DATA_OVERLAY_FILES` is set; debug builds understand it. Tested (release-gating policy is asserted profile-independently). - **`Operation::DataOverlay` transaction** — appends overlays to a fragment (preserving concurrently-written ones) and stamps `committed_version` at commit, re-stamped on retry. Protobuf round-trip + multi-fragment `build_manifest` (distinct targets + untargeted pass-through) tested. - **Conflict resolution** (v2 `CommitConflictResolver`) — permissive like `DataReplacement`: compatible with append / column-rewrite / index-build / data-replacement / other overlays, and with deletes/updates that leave the overlaid fragment in place. Retryable when a concurrent op row-rewrites or consumes the overlays on an overlaid fragment (`Rewrite`/`Merge`) or *removes* one (`Update`/`Delete` removal); incompatible with whole-dataset `Overwrite`/`Restore` and with `UpdateMemWalState` (matching the spec — both commit directions now agree). Tested (both-direction matrix, including remove-fragment and `Rewrite`×overlay). The fragment read path refuses overlays until the scan merge lands (OSS-1324). ## Follow-ups on this stack - [ ] `DataOverlayFileWriter` — streaming dense+sparse sink, used internally behind `update`/`merge_insert`. - [ ] `validate()` overlay checks — value-column length vs. coverage cardinality, dtype vs. schema (I/O-bound; cheap structural invariants are enforced at parse/commit time). - [ ] Scan/take merge (OSS-1324), resolved at read time fetching only the touched coverage ranks. Blocks OSS-1324 (take/scan), OSS-1325 (index masking), OSS-1326 (compaction). 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added experimental data overlay file support with opt-in release gating via an environment variable. * Introduced first-class `DataOverlay` transaction support to append per-fragment overlay updates during commits, including newest-last overlay ordering validation. * **Bug Fixes** * Improved overlay conflict handling across rewrites, data replacements, and row-moving updates, including deferred row-overlap validation. * Prevented reads from proceeding on fragments containing overlay data when overlay merge is in progress. * **Documentation** * Updated transaction compatibility docs with clearer overlay stacking rules, rewrite/replacement overlay interactions, and added conflict scenarios. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
… to avoid role chaining (lance-format#7757) ## Problem The credential vendor's `api_key` and static flows assume the target role with `AssumeRole`, signed by the process's **ambient** credentials. When the vendor runs with temporary credentials — e.g. an IRSA / EKS pod role — that is **role chaining**, which STS **hard-caps to 1 hour** regardless of the role's `MaxSessionDuration`. In that setup the returned `Expiration` can also exceed the token's *real* validity, so downstream credential caches (`StorageOptionsAccessor`, `CachingCredentialVendor`) keep serving a token S3 already rejects with `ExpiredToken` and never refresh — their expiry checks trust the inflated `expires_at_millis`. ## Change Add an optional **pod web-identity** path. When `pod_web_identity_token_file` is configured — via property `credential_vendor.aws_assume_via_pod_web_identity=true` (resolving the EKS-injected `AWS_WEB_IDENTITY_TOKEN_FILE`), or an explicit `credential_vendor.aws_pod_web_identity_token_file` — the scoped assume uses `AssumeRoleWithWebIdentity` with the pod's projected service-account OIDC token. `AssumeRoleWithWebIdentity` is a **direct** (non-chained) assumption, so STS honors the role's `MaxSessionDuration` (up to 12h) and reports an accurate expiration. - Per-table scoped session policy, permission handling, and the default chained `AssumeRole` behavior are **unchanged**. - The token file is re-read on every vend (kubelet rotates it). - Requires the target role's trust policy to federate the cluster OIDC provider for the service account. This change also improves error handling to propagate the object store error message in namespace error messages. ## Tests - `test_config_builder` extended for the new field. - `test_pod_web_identity_path_reads_token_file` — verifies the web-identity branch is selected and reads the token file. 🤖 Co-authored with Claude Opus 4.8 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added AWS pod-based web identity authentication for credential vending. * New configuration option to set the projected service-account token file path (with optional opt-in fallback to the standard AWS env var). * When configured, scoped role vending uses web identity instead of the legacy chained assume-role flow. * **Bug Fixes** * Preserves existing ambient-credential behavior when web identity is not configured. * **Tests** * Added coverage to ensure vending fails with a clear token-file read error when the configured token file path is missing. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
…7698) Fixes lance-format#7697 ## Summary Fix Lance-generated DataFusion field-path expressions so schema-derived column names preserve exact casing. Previously, `field_path_to_expr` used DataFusion `col(...)`, which lowercases unquoted identifiers. This caused generated nullable-vector filters like `VECTOR IS NOT NULL` to resolve as `vector`, breaking case-sensitive schemas during vector index creation / append optimization. ## Changes - Build root field-path expressions with `Expr::Column(Column::new_unqualified(...))` instead of `col(...)`. - Preserve existing nested field handling through `field_newstyle(...)`. - Add regression coverage for: - uppercase nullable vector column indexing / append optimization - exact-case root column expression generation - mixed-case root plus escaped nested field path containing dots ## Testing ```bash cargo test -p lance-datafusion logical_expr::tests::test_field_path_to_expr cargo test -p lance test_optimize_append_preserves_case_sensitive_nullable_vector_column cargo fmt --all <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved query parsing to correctly preserve case sensitivity for root column names and handle escaped nested field paths (including dots inside escaped segments). * Fixed an issue where appending data and optimizing indexes could leave unindexed fragments, affecting vector scans. * Vector search results now remain consistent after append + index optimization when using case-sensitive nullable vector columns. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
The per-batch `dot_batch_f32_avx*` / `l2_batch_f32_avx*` wrappers are only called from the runtime-dispatch path, which is compiled out when the build baseline already guarantees avx2+fma (the repo's `target-cpu=haswell` default). Under that baseline the wrappers, their test helper, and their tests were unused, so `cargo clippy -D warnings` failed on dead_code. Gate the definitions and their tests with the same `not(all(target_feature = "avx2", target_feature = "fma"))` cfg as their caller, so definition, dispatch, and test compile together. No behavior change: at the avx2+fma baseline `dot_batch`/`l2_batch` already inline the kernel directly; below it the runtime dispatch and these wrappers remain.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds runtime SIMD dispatch to lance-linalg's hot distance kernels so a from-source build with a lower x86_64 baseline produces a working binary on pre-Haswell hardware (Sandy Bridge / Ivy Bridge / Steamroller). fast-by-default with a documented from-source override for legacy users.
Today,
import lancedbSIGILLs on AVX-without-AVX2 CPUs because the wheel bakes AVX2 into every compiled function with no runtime guard. numpy and pyarrow handle the same hardware via runtime dispatch. This PR brings lance to parity for the from-source legacy build path.Summary
match *SIMD_SUPPORT+mod x86 { #[target_feature(enable=...)] pub unsafe fn ... }shape asdot_u8.rs/cosine_u8.rs/l2_u8.rs. On the haswell baseline, dispatch always lands on AVX2 — modern compile output is unchanged from today.lance.simd_info()Python API mirroringpyarrow.runtime_info()for tier introspection.qemu-pre-haswellCI gate that builds withRUSTFLAGS="-C target-cpu=x86-64-v2"(env-var-scoped to that one job — workspace.cargo/config.tomlis unchanged) and runs lance-linalg tests under qemu Nehalem.CONTRIBUTING.mddocuments the legacy build:RUSTFLAGS="-C target-cpu=x86-64-v2" cargo build --release.Zero new external dependencies. Dispatch shape extends an idiom lance already uses for u8 and f16/bf16 kernels rather than introducing a new convention. Recent precedent: @justinrmiller's #6540, #6517, #6506, #6510.
Verification
cargo test -p lance-linalg --lib— 83/83 on aarch64 dev box.is_x86_feature_detected!()so each runs on hosts that can execute its tier.cargo clippy --all-targets -- -D warningsclean;cargo fmt --checkclean;Cargo.lockunchanged.RUSTFLAGSoverride (via companion lancedb wheel — seetobocop2/lancedb#2for the verification PASS output).change:lines once I find a host that can hold the fullcargo bench -p lance-linalg --bench {cosine,dot,l2,norm_l2}suite (Codespace's 30-min idle timeout killed my last attempt mid-run).Closes #1.