build: bump lance to runtime-dispatch fix; document RUSTFLAGS override for pre-Haswell by tobocop2 · Pull Request #2 · tobocop2/lancedb

tobocop2 · 2026-04-26T01:47:47Z

🚧 Fork-internal PR for end-to-end verification. Lance dep is pinned to tobocop2/lance rev 026346c2; will swap to an upstream lance-format/lance release before this is retargeted at lancedb/lancedb upstream.

Companion to tobocop2/lance#2, which adds runtime SIMD dispatch to lance-linalg. Per westonpace's review on lancedb/lancedb#3324: published wheels stay at target-cpu=haswell; pre-Haswell users build from source with a documented RUSTFLAGS override.

Summary

Bump 14 lance crate deps from lance-format/lance.git tag=v6.0.0-beta.1 → tobocop2/lance.git rev=026346c25892fbeb91a4144ecf67d739f987c9d2. Version bumped to =6.0.0-beta.3 so a cargo update against the upstream tag doesn't silently overwrite the fork-rev pin.
python/README.md documents the from-source legacy build path: RUSTFLAGS="-C target-cpu=x86-64-v2" maturin build --release.
.cargo/config.toml baselines unchanged (target-cpu=haswell + +avx2,+fma,+f16c for both linux-gnu and linux-musl). Published wheels stay fast-by-default; only the from-source path opts into the lower baseline.
net.git-fetch-with-cli = true so cargo honors git config when fetching the fork dep — comes out once we depend on a lance release.

Verified end-to-end on Sandy Bridge Xeon E5-2609

$ grep -m1 'model name' /proc/cpuinfo
model name      : Intel(R) Xeon(R) CPU E5-2609 0 @ 2.40GHz

$ grep -m1 '^flags' /proc/cpuinfo | tr ' ' '\n' | grep -E '^(sse4_2|avx|avx2|fma|avx512f)$' | sort -u
avx
sse4_2

$ RUSTFLAGS="-C target-cpu=x86-64-v2" maturin build --release
$ pip install ./target/wheels/lancedb-*.whl
$ python verify.py
PASS: import + simd dispatch + table create + vector search all work.

Pre-PR (default pip install lancedb): Illegal instruction (core dumped) at import lancedb.

Full reproducer (deps + clone + RUSTFLAGS-override build + verification): https://gist.github.com/tobocop2/2e341358b55c143527416edfdb1e37df

Test plan

cargo check --features remote clean on aarch64-apple-darwin
Wheel build + import + vector search all pass on Sandy Bridge Xeon E5-2609 with the RUSTFLAGS override
Wheel .cargo/config.toml baselines unchanged (haswell)
cargo test, cargo clippy, cargo fmt --check (CI)
AvxFma tier confirmed working on AMD FX-7500 (Steamroller) — pending friend's run

Use lance.simd_info() (added in tobocop2/lance#2) to introspect what tier each from-source build hits.

Closes #1.

## Summary - Updates Lance Rust dependencies to `6.0.0-beta.4` using `ci/set_lance_version.py`. - Updates the Java `lance-core.version` property to `6.0.0-beta.4`. - Triggering Lance tag: https://github.com/lance-format/lance/releases/tag/v6.0.0-beta.4 ## Verification - `cargo clippy --workspace --tests --all-features -- -D warnings` - `cargo fmt --all`

## Summary - Update `rustls-webpki` 0.103.10 → 0.103.13 to fix RUSTSEC-2026-0104 (reachable panic in CRL parsing) - Add advisory ignore for the legacy `rustls-webpki` 0.101.7 copy pinned to the aws-smithy/rustls 0.21 chain (same chain already exempted for RUSTSEC-2026-0098/0099) Fixes the `deny` CI job failure seen in lancedb#3325. ## Test plan - [x] `cargo deny check advisories` passes locally 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]>

github-actions · 2026-04-28T05:02:06Z

ACTION NEEDED

Lance follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.

For details on the error please inspect the "PR Title Check" action.

Adds manifest_enabled for local/native connections so directory namespace manifests can be the source of truth, including migration from directory listing and Azure credential vending feature wiring. Also exposes the option through Rust, Python, and Node bindings with focused validation.

## Summary - Update Lance Rust dependencies to `6.0.0-beta.7` using `ci/set_lance_version.py`. - Update Java `lance-core.version` to `6.0.0-beta.7`. - Align Arrow/DataFusion/PyO3 dependency versions and apply required compatibility fixes for the Lance upgrade. Triggering tag: [v6.0.0-beta.7](https://github.com/lance-format/lance/releases/tag/v6.0.0-beta.7) ## Verification - `cargo clippy --workspace --tests --all-features -- -D warnings` - `cargo fmt --all`

…#3340) Hi, the hybrid query error message looks like it can use a space, just added it. ```python def _validate_query(self, query, vector=None, text=None): if query is not None and (vector is not None or text is not None): raise ValueError( "You can either provide a string query in search() method" "or set `vector()` and `text()` explicitly for hybrid search." "But not both." ) ```

…ailure (lancedb#3285) The `build:release` command already outputs the `*.node` files directly to the `dist/` directory via the `--output-dir dist` flag. Therefore, the `postbuild:release` script, which attempts to copy `*.node` files from the `lancedb/` source directory, fails with a "no such file or directory" error because the source files do not exist there. This commit removes the redundant `postbuild:release` script to resolve the build failure. fix lancedb#3284 Signed-off-by: qingfeng-occ <[email protected]>

…ancedb#3335) ## Summary When pytorch is used with multiprocessing and the mp mode is spawn then the Permutation needs to be pickled. It could not be pickled because `Table` and `Connection` are not serializable. This PR adds pickle support to Permutation without adding general pickle support to `Table` or `Connection`. To add general support we probably need to start by adding serialization in the namespace client. In the meantime this PR enable pickling by adding special cases for: * In-memory tables (just serialize as Arrow IPC) * Native tables (serialize the URI) If a user is not using one of the above cases (e.g. using a remote connection) then they will need to provide a connection factory that can be pickled. ## Breaking change `PermutationBuilder.persist(...)` is removed from the Python bindings; the permutation table is now always in-memory. The underlying Rust `PermutationBuilder::persist` API is untouched and can be re-exposed later if needed. It probably won't make sense to do that until we have a way to serialize `Table` and `Connection`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>

…rs (lancedb#3339) ## Summary PyTorch's `DataLoader` uses fork-based multiprocessing by default on Linux, but threads do not survive `fork()`. LanceDB's Python bindings drive async work through two threaded layers, both of which become inert in a forked child: - `BackgroundEventLoop` runs an asyncio loop on a Python `threading.Thread`. - `pyo3-async-runtimes::tokio` holds a global multi-threaded tokio runtime whose worker threads also die on fork — and its runtime lives in a `OnceLock` that cannot be replaced after first use. As a result, any `Permutation` (or other async API) used inside a fork-based `DataLoader` worker hangs indefinitely. This PR makes both layers fork-safe so `Permutation` works as a `torch.utils.data.Dataset` with `num_workers > 0`. ## Approach ### Rust — new `python/src/runtime.rs` Mirrors the pattern used in [Lance's Python bindings](https://github.com/lance-format/lance/blob/456198cd6f42be07f99617a6d7e39d6209cdf3cc/python/src/lib.rs#L139), adapted for the async-bridge use case. - `LanceRuntime` implements `pyo3_async_runtimes::generic::Runtime + ContextExt`, backed by an `AtomicPtr<tokio::runtime::Runtime>` we own (sidestepping `pyo3-async-runtimes`'s frozen `OnceLock` global). - A `pthread_atfork(after_in_child)` handler nulls the pointer; the next `spawn` rebuilds the runtime in the child. The previous runtime is intentionally **leaked** — calling `Drop` would try to join now-dead worker threads and hang. - `runtime::future_into_py` is a drop-in for `pyo3_async_runtimes::tokio::future_into_py`. All ~80 call sites in `arrow.rs` / `connection.rs` / `permutation.rs` / `query.rs` / `table.rs` are updated to route through it. - `python/Cargo.toml` adds `libc = "0.2"` and the tokio `rt-multi-thread` feature. ### Python — `lancedb/background_loop.py` - Refactors `BackgroundEventLoop.__init__` to a reusable `_start()` method. - An `os.register_at_fork(after_in_child=…)` hook calls `LOOP._start()` to give the singleton a fresh asyncio loop and thread **in place**. This matters because the rest of the codebase imports `LOOP` via `from .background_loop import LOOP` — rebinding the module attribute would leave those references holding the dead loop. ### Python — `lancedb/__init__.py` Removes the `__warn_on_fork` pre-fork warning (and the now-unused `import warnings`). Fork is supported. ## Test plan - [x] New `test_permutation_dataloader_fork_workers` in `python/tests/test_torch.py`: runs a `Permutation` through `torch.utils.data.DataLoader(num_workers=2, multiprocessing_context="fork")` inside a spawn-isolated child with a 30s hang detector. **Pre-fix**: timed out at 36s. **Post-fix**: passes in ~3.6s. - [x] New `test_remote_connection_after_fork` in `python/tests/test_remote_db.py`: forks a child that creates a fresh `lancedb.connect(...)` against a mock HTTP server and calls `table_names()`; passes in <1s, validates the runtime reset is sufficient for fresh remote clients. - [x] All 62 tests in `test_torch.py` + `test_permutation.py` pass. - [x] All 35 tests in `test_remote_db.py` pass. - [x] `test_table.py` (87) + `test_db.py` + `test_query.py` (157, minus one unrelated `sentence_transformers` import skip) — 244 passing. - [x] `cargo clippy -p lancedb-python --tests` clean. - [x] `cargo fmt`, `ruff check`, `ruff format` all clean. ## Known limitation (follow-up) This PR makes a **freshly-built** `lancedb.connect(...)` work in a forked child. An **inherited** `Connection` from the parent still carries an inherited `reqwest::Client` whose hyper connection pool references socket FDs and TCP/TLS state shared with the parent — using it from the child after fork is unsafe (especially with HTTP/1.1 keep-alive). The recommended pattern for fork-based `DataLoader` workers that hit a remote DB is to construct a new connection inside the worker. Auto-clearing inherited HTTP client pools on fork would require tracking live `Connection` instances in `lancedb` core and is left for a follow-up PR. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>

## Summary - Update Lance Rust dependencies to `v7.0.0-beta.4` using `ci/set_lance_version.py`. - Update the Java `lance-core` dependency property to `7.0.0-beta.4`. - Align LanceDB with dependency updates required by Lance 7, including `object_store` 0.13 API compatibility. Triggering tag: https://github.com/lance-format/lance/releases/tag/v7.0.0-beta.4 ## Verification - `cargo clippy --workspace --tests --all-features -- -D warnings` - `cargo fmt --all`

…flow (lancedb#3313) Fixes lancedb#3299 ## Problem Two security issues exist in `.github/workflows/java-publish.yml`: 1. **`gpg-passphrase` input is misused**: `actions/setup-java`'s `gpg-passphrase` input expects the **name** of an environment variable (default: `GPG_PASSPHRASE`), not the secret value itself. The previous value `${{ secrets.GPG_PASSPHRASE }}` was setting the env var name to the actual secret, which is incorrect. 2. **Passphrase visible on the command line**: `-Dgpg.passphrase=${{ secrets.GPG_PASSPHRASE }}` passes the GPG passphrase as a Maven system property argument, making it visible in process listings and potentially echoed in debug logs — a supply-chain security risk for release workflows. ## Solution - Fix `gpg-passphrase: MAVEN_GPG_PASSPHRASE` — use the correct env var name so `actions/setup-java` generates a proper Maven `settings.xml` entry that reads from `MAVEN_GPG_PASSPHRASE`. - Remove `-Dgpg.passphrase=...` from the Maven CLI invocation. - Add `MAVEN_GPG_PASSPHRASE: ${{ secrets.GPG_PASSPHRASE }}` to the `env:` block of the Publish step, so the passphrase is available as an environment variable rather than a CLI argument. ## Testing The Java publish workflow only runs on tag pushes, so this cannot be exercised in a PR build. The logic change is straightforward: `actions/setup-java` is documented to write a `settings.xml` that reads `<gpg.passphrase>` from the named env var, and `maven-gpg-plugin` picks it up from there without any CLI argument. Co-authored-by: octo-patch <[email protected]>

## Summary - Update Lance Rust workspace dependencies to `7.0.0-beta.7` using `ci/set_lance_version.py`. - Update the Java `lance-core` Maven property to `7.0.0-beta.7`. - Refresh `Cargo.lock` for the new Lance tag: https://github.com/lance-format/lance/releases/tag/v7.0.0-beta.7 ## Verification - `cargo clippy --workspace --tests --all-features -- -D warnings` - `cargo fmt --all`

…ancedb#3357) ## Summary `url::Url::query_pairs_mut()` leaves the URL with `query=Some("")` after `.clear()` even when the input had no query string. The listing-database connect path then captured that empty query into `ListingDatabase::query_string`, and `table_uri()` blindly appended `?<query>` to every per-table URI — producing URIs like `s3://bucket/prefix/foo.lance?`. The trailing `?` is benign for normal table operations, but it breaks any caller that constructs a sub-path from the table URI. In particular, MemWAL flushes write to `<table_uri>/_mem_wal/<shard>/<rand>_gen_<n>`, which `url::Url::parse` then re-parses as `path=<base table>` + `query=/_mem_wal/...`. `Dataset::write` resolves the base table dataset, finds it already exists, and fails with `Dataset already exists: …_gen_1` on the very first MemTable flush (observed deterministically against S3 across all merge_insert LSM modes; tracked in [lance-format/lance#6713](lance-format/lance#6715)). ## Fix Treat `Some("")` query the same as no query when capturing `query_string`. A real `?foo=bar` query is still propagated unchanged. Adds a regression test covering both the empty-query and non-empty-query paths. ## Verification - `url::Url::parse("s3://bucket/prefix/").query()` → `None`, but after `query_pairs_mut().clear()` → `Some("")`. Confirmed in a standalone repro. - Without this fix, every `table_uri()` for an `s3://`-style connection ends with `?`, breaking MemWAL and any future sub-path consumer in the same way. - New unit test `test_table_uri_url_path_has_no_trailing_question_mark` exercises both code paths.

…ncedb#3469) ## What's broken `MRRReranker.rerank_multivector([])` raises `IndexError: list index out of range`. The crash happens on line 128 (the `all()` type-homogeneity check passes vacuously on an empty iterable) and on line 134 which accesses `vector_results[0]` unconditionally, with no prior guard for an empty list. ## Why it happens `all()` over an empty iterable returns `True`, so the type check silently passes and execution falls through to `vector_results[0]` which crashes. ## Fix Added a two-line guard at the top of `rerank_multivector` that raises a clear `ValueError("vector_results must not be empty")` before any indexing occurs. ## Test Added `test_mrr_reranker_empty_input` in `test_rerankers.py` which calls `rerank_multivector([])` and asserts that a `ValueError` with the message "must not be empty" is raised. Fixes lancedb#3468 Co-authored-by: Aegis Dev <[email protected]>

…ank_multivector (lancedb#3467) ## What's broken Calling `RRFReranker().rerank_multivector([])` crashes with `IndexError: list index out of range` because the method accesses `vector_results[0]` for the type-homogeneity check before verifying the list is non-empty. The `all()` call passes vacuously on an empty iterable so the crash hits the next lines. ```python from lancedb.rerankers import RRFReranker RRFReranker().rerank_multivector([]) # IndexError: list index out of range ``` ## Why it happens The type check uses `vector_results[0]` as the reference type but never guards against an empty list. `all(...)` short-circuits to `True` when the iterable is empty, so the bad index access on the lines that follow is never reached by the existing guard logic. ## Fix Add an explicit empty-list check before any indexing.

Adds a REVIEW.md at the repo root with cross-SDK parity guidance for automated code review. The Claude Code review feature automatically loads `REVIEW.md` as review-only context. This is intentionally a semantic nudge, not a deterministic check, it relies on the reviewer reading the sibling SDK, so it will catch most gaps.

…edb#2654) (lancedb#3483) ## Summary Regression test for [issue lancedb#2654](lancedb#2654) — a nullable struct column whose first batch contains only `None` values crashed in `_align_field_types` with `AttributeError: 'pyarrow.lib.DataType' object has no attribute 'fields'`. The actual fix landed in lancedb#3394, but no test was added. This PR adds the reproducer from the issue as a test. ## Test plan - `test_add_nullable_struct_with_none`: creates a table with a nullable struct column, adds a row with a non-null struct value, then a row with `None` for the struct field. Verifies both rows land correctly. - Uses Lance file format v2.1 (`new_table_data_storage_version="2.1"`) because nullable structs aren't supported on v2.0. ## Related - lancedb#3028 (the original fix attempt, now superseded)

Make sure we are getting security fixes in there regularly, and other useful bumps.

lancedb#3444) ### Description This PR exposes native DataFusion expression support in the Rust SDK's `MergeInsertBuilder` via two new builder methods: `when_matched_update_all_expr` and `when_not_matched_by_source_delete_expr`. For remote LanceDB tables (where operations are serialized over HTTP/JSON to the SaaS backend), native DataFusion expression trees cannot be executed directly. The SDK handles this gracefully by returning a `NotSupported` error. ### Key Changes - **`MergeFilter` Enum**: Introduced a helper enum to store either a SQL string or a native `datafusion_expr::Expr`. - **`MergeInsertBuilder`**: Updated `when_matched_update_all_filt` and `when_not_matched_by_source_delete_filt` fields to store the new enum, and added `when_matched_update_all_expr` and `when_not_matched_by_source_delete_expr` builder methods. - **Execution & Remote Dispatch**: Dispatched the filter variants during local execution, and rejected expression filters with a clean `NotSupported` error in remote table request conversion. - **Testing**: Added a `test_merge_insert_expr` unit test covering conditional updates and deletes with programmatically built DataFusion expressions. ### Verification - Added integration test `test_merge_insert_expr` which successfully compiles and passes. - Formatted and linted the code. Closes lancedb#3416

## Bug Fix ### What is the bug? `QueryBuilder.to_pandas(blob_mode="descriptions")` could still fall back to `self.to_arrow()` for query outputs with blob columns. Custom query subclasses or wrappers can have `to_arrow()` behavior that is not compatible with pandas blob-description conversion, which can surface as low-level Arrow/list-batch conversion failures. ### What issues or incorrect behavior does the bug cause? Callers need to carry local `to_pandas` or plain-scan adapter special casing for blob descriptions, and scanner-only kwargs such as row addresses and fragment selection are not represented in LanceDB query state. ### How does this PR fix the problem? This PR routes blob-output query `to_pandas()` through the Lance scanner path for `lazy`, `bytes`, and `descriptions` modes when the query is a scanner-backed plain scan. For `blob_mode="descriptions"` with `flatten`, it collects scanner Arrow/table output, applies LanceDB `flatten_columns`, and converts to pandas from there. Non-plain blob query shapes now fail with a clear unsupported error instead of falling into subclass `to_arrow()` behavior. It also adds Python query state and builder methods for scanner-only plain-scan parameters: - `with_row_address()` for `_rowaddr` - `with_fragments(...)` for Lance fragment objects - `fragment_ids([...])` as a convenience wrapper that resolves IDs to Lance fragments ## Validation - `cd python && uv run --no-sync ruff format --check python/lancedb/query.py python/tests/test_query.py` - `cd python && uv run --no-sync ruff check python/lancedb/query.py python/tests/test_query.py` Targeted pytest was intentionally not run locally per maintainer request.

…3499) Bumps the rust-minor-patch group with 3 updates: [log](https://github.com/rust-lang/log), [test-log](https://github.com/d-e-s-o/test-log) and [serial_test](https://github.com/palfrey/serial_test). Updates `log` from 0.4.30 to 0.4.31 <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/rust-lang/log/releases">log's releases</a>.</em></p> <blockquote> <h2>0.4.31</h2> <h2>What's Changed</h2> <ul> <li>fix typos in kv compile errors and log documentation by <a href="https://github.com/Isvane"><code>@Isvane</code></a> in <a href="https://redirect.github.com/rust-lang/log/pull/726">rust-lang/log#726</a></li> <li>Leverage static str key when possible by <a href="https://github.com/tisonkun"><code>@tisonkun</code></a> in <a href="https://redirect.github.com/rust-lang/log/pull/727">rust-lang/log#727</a></li> <li>Prepare for 0.4.31 release by <a href="https://github.com/KodrAus"><code>@KodrAus</code></a> in <a href="https://redirect.github.com/rust-lang/log/pull/728">rust-lang/log#728</a></li> </ul> <h2>New Contributors</h2> <ul> <li><a href="https://github.com/Isvane"><code>@Isvane</code></a> made their first contribution in <a href="https://redirect.github.com/rust-lang/log/pull/726">rust-lang/log#726</a></li> </ul> <p><strong>Full Changelog</strong>: <a href="https://github.com/rust-lang/log/compare/0.4.30...0.4.31">https://github.com/rust-lang/log/compare/0.4.30...0.4.31</a></p> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/rust-lang/log/blob/master/CHANGELOG.md">log's changelog</a>.</em></p> <blockquote> <h2>[0.4.31] - 2026-06-02</h2> <h2>What's Changed</h2> <ul> <li>Leverage static str key when possible by <a href="https://github.com/tisonkun"><code>@tisonkun</code></a> in <a href="https://redirect.github.com/rust-lang/log/pull/727">rust-lang/log#727</a></li> </ul> <h2>New Contributors</h2> <ul> <li><a href="https://github.com/Isvane"><code>@Isvane</code></a> made their first contribution in <a href="https://redirect.github.com/rust-lang/log/pull/726">rust-lang/log#726</a></li> </ul> <p><strong>Full Changelog</strong>: <a href="https://github.com/rust-lang/log/compare/0.4.30...0.4.31">https://github.com/rust-lang/log/compare/0.4.30...0.4.31</a></p> <h2>[Unreleased]</h2> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/rust-lang/log/commit/580839288e5f2babc17e6c36f7d56e60082a47ef"><code>5808392</code></a> Merge pull request <a href="https://redirect.github.com/rust-lang/log/issues/728">#728</a> from rust-lang/cargo/0.4.31</li> <li><a href="https://github.com/rust-lang/log/commit/86d739f51a9c59a3cb66a79e695639e6fb41465b"><code>86d739f</code></a> prepare for 0.4.31 release</li> <li><a href="https://github.com/rust-lang/log/commit/c906cfb02e351b59cfe35c0f0be22093086aabb1"><code>c906cfb</code></a> Merge pull request <a href="https://redirect.github.com/rust-lang/log/issues/727">#727</a> from tisonkun/leverage-static-str-key-when-possible</li> <li><a href="https://github.com/rust-lang/log/commit/756c279649f79ce0ef8dccf952c5df4017791d1c"><code>756c279</code></a> leverage str literal as well</li> <li><a href="https://github.com/rust-lang/log/commit/3dd250d1537fd7e5974e0802b1025cc3e4561503"><code>3dd250d</code></a> rename Key::from_static_str to from_str_static</li> <li><a href="https://github.com/rust-lang/log/commit/db145979e229549215300f2696fa89b215cb1cab"><code>db14597</code></a> Leverage static str key when possible</li> <li><a href="https://github.com/rust-lang/log/commit/761461a5d0c8ea3d483d79b1de0205c2897318d2"><code>761461a</code></a> Merge pull request <a href="https://redirect.github.com/rust-lang/log/issues/726">#726</a> from Isvane/fix/typos</li> <li><a href="https://github.com/rust-lang/log/commit/48ce372edd343179cb9f4837381bf34c7679db3e"><code>48ce372</code></a> fix typos in kv compile errors and log documentation</li> <li>See full diff in <a href="https://github.com/rust-lang/log/compare/0.4.30...0.4.31">compare view</a></li> </ul> </details> <br /> Updates `test-log` from 0.2.20 to 0.2.21 <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/d-e-s-o/test-log/releases">test-log's releases</a>.</em></p> <blockquote> <h2>v0.2.21</h2> <ul> <li>Fixed spans in generated code, improving <code>rust-analyzer</code> interaction</li> </ul> <h2>New Contributors</h2> <ul> <li><a href="https://github.com/jorendorff"><code>@jorendorff</code></a> made their first contribution in <a href="https://redirect.github.com/d-e-s-o/test-log/pull/68">d-e-s-o/test-log#68</a></li> </ul> <p><strong>Full Changelog</strong>: <a href="https://github.com/d-e-s-o/test-log/compare/v0.2.20...v0.2.21">https://github.com/d-e-s-o/test-log/compare/v0.2.20...v0.2.21</a></p> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/d-e-s-o/test-log/blob/main/CHANGELOG.md">test-log's changelog</a>.</em></p> <blockquote> <h2>0.2.21</h2> <ul> <li>Fixed spans in generated code, improving <code>rust-analyzer</code> interaction</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/d-e-s-o/test-log/commit/b7b9da034578877997e0cbfb59ea11507e0a3da8"><code>b7b9da0</code></a> Bump version to 0.2.21</li> <li><a href="https://github.com/d-e-s-o/test-log/commit/db522dc408e1ac6f04b2d0c89dda7e4b48be0584"><code>db522dc</code></a> Add CHANGELOG entry for <a href="https://redirect.github.com/d-e-s-o/test-log/issues/68">#68</a></li> <li><a href="https://github.com/d-e-s-o/test-log/commit/5e996d9ac66882e6258d1d86df91417336436d14"><code>5e996d9</code></a> Wrap the injected init code, not the original test body</li> <li><a href="https://github.com/d-e-s-o/test-log/commit/c78563c1ca76720571dc5ffe731217adc7e781ed"><code>c78563c</code></a> Retain existing spans for test code</li> <li>See full diff in <a href="https://github.com/d-e-s-o/test-log/compare/v0.2.20...v0.2.21">compare view</a></li> </ul> </details> <br /> Updates `serial_test` from 3.4.0 to 3.5.0 <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/palfrey/serial_test/releases">serial_test's releases</a>.</em></p> <blockquote> <h2>v3.5.0</h2> <h2>What's Changed</h2> <ul> <li>Replace scc/sdd with std::sync::Mutex for Miri strict provenance compatibility by <a href="https://github.com/justanotheranonymoususer"><code>@justanotheranonymoususer</code></a> in <a href="https://redirect.github.com/palfrey/serial_test/pull/157">palfrey/serial_test#157</a></li> </ul> <h2>New Contributors</h2> <ul> <li><a href="https://github.com/justanotheranonymoususer"><code>@justanotheranonymoususer</code></a> made their first contribution in <a href="https://redirect.github.com/palfrey/serial_test/pull/157">palfrey/serial_test#157</a></li> </ul> <p><strong>Full Changelog</strong>: <a href="https://github.com/palfrey/serial_test/compare/v3.4.0...v3.5.0">https://github.com/palfrey/serial_test/compare/v3.4.0...v3.5.0</a></p> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/palfrey/serial_test/commit/6181f64de942180231fdb9098dd0894bbd9e7472"><code>6181f64</code></a> 3.5.0</li> <li><a href="https://github.com/palfrey/serial_test/commit/480bead2f697707cd2e61214287d5f0b8518d44d"><code>480bead</code></a> Merge pull request <a href="https://redirect.github.com/palfrey/serial_test/issues/157">#157</a> from justanotheranonymoususer/remove-scc-dep</li> <li><a href="https://github.com/palfrey/serial_test/commit/e03019e3cdc79b65daeaa913c99376aed69d4101"><code>e03019e</code></a> Update ci.yml</li> <li><a href="https://github.com/palfrey/serial_test/commit/820c0f3de967d55c52c59cac9ae55345511bf468"><code>820c0f3</code></a> Update ci.yml</li> <li><a href="https://github.com/palfrey/serial_test/commit/62a89b055fb923159c428269c5be999509344cb1"><code>62a89b0</code></a> Only skip file_lock with filesystem access</li> <li><a href="https://github.com/palfrey/serial_test/commit/5ff550164ed6f149fc80230faa8d5b5ded234190"><code>5ff5501</code></a> Update ci.yml</li> <li><a href="https://github.com/palfrey/serial_test/commit/0bd996de9eb044293e149095465701175309942e"><code>0bd996d</code></a> Let's try --all-features</li> <li><a href="https://github.com/palfrey/serial_test/commit/338e4ed891a095e2bfda01572c150449e5f26e73"><code>338e4ed</code></a> Fix formatting</li> <li><a href="https://github.com/palfrey/serial_test/commit/a55cde5d1d1572db2a8e5930d361d95df9796ea0"><code>a55cde5</code></a> Cleanup code_lock.rs</li> <li><a href="https://github.com/palfrey/serial_test/commit/9ad7a8f18c9a598109df5197214669e8680ccb96"><code>9ad7a8f</code></a> Remove unnecessary test leftover changes</li> <li>Additional commits viewable in <a href="https://github.com/palfrey/serial_test/compare/v3.4.0...v3.5.0">compare view</a></li> </ul> </details> <br /> Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore <dependency name> major version` will close this group update PR and stop Dependabot creating any more for the specific dependency's major version (unless you unignore this specific dependency's major version or upgrade to it yourself) - `@dependabot ignore <dependency name> minor version` will close this group update PR and stop Dependabot creating any more for the specific dependency's minor version (unless you unignore this specific dependency's minor version or upgrade to it yourself) - `@dependabot ignore <dependency name>` will close this group update PR and stop Dependabot creating any more for the specific dependency (unless you unignore this specific dependency or upgrade to it yourself) - `@dependabot unignore <dependency name>` will remove all of the ignore conditions of the specified dependency - `@dependabot unignore <dependency name> <ignore condition>` will remove the ignore condition of the specified dependency and ignore conditions </details> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

## Summary - Add `__reduce__` methods to `LanceDBClientError` and `RetryError` so that instances can be pickled and unpickled correctly - `HttpError` inherits the fix from `LanceDBClientError` since it has no additional `__init__` parameters - Add tests verifying pickle roundtrip for all three exception classes Fixes lancedb#3447 ## Test plan - [x] Verified pickle roundtrip for `LanceDBClientError` with and without `status_code` - [x] Verified pickle roundtrip for `HttpError` (subclass, no extra init params) - [x] Verified pickle roundtrip for `RetryError` (subclass with many extra params) - [ ] CI tests pass 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.6 <[email protected]> Co-authored-by: Will Jones <[email protected]>

Updates Lance dependencies to v8.0.0-beta.2 across the Rust workspace and Java lance-core metadata. The update was generated with ci/update_lance_dependency.py and required no compatibility code changes. Lance tag: https://github.com/lance-format/lance/releases/tag/v8.0.0-beta.2 ## ⛔ Merge blocker: legal review required This bump pulls in a new transitive **dev/profiling** dependency chain `inferno v0.11.21` → `pprof v0.15.0` → `lance-testing`, and `inferno` is licensed **CDDL-1.0** (copyleft). To get `cargo-deny` green, `CDDL-1.0` was added to the `deny.toml` allow list. **Do not merge until legal has reviewed and signed off on allowing CDDL-1.0.** The dependency is dev/test-only and not distributed, but the allow-list addition still requires legal approval per our policy. --------- Co-authored-by: Daniel Rammer <[email protected]> Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>

…b#3501) ## Summary Wires `RemoteTable::set_lsm_write_spec` / `unset_lsm_write_spec` to the sophon REST endpoints added in [lancedb/sophon#6181](lancedb/sophon#6181), replacing the previous `NotSupported` stubs. - `set_lsm_write_spec` maps the `LsmWriteSpec` onto sophon's request DTO — mode-tagged `sharding` (`unsharded` / `bucket` / `identity`), `maintained_indexes`, and `writer_config_defaults` — and POSTs to `/v1/table/{name}/set_lsm_write_spec/`. - `unset_lsm_write_spec` POSTs to `/v1/table/{name}/unset_lsm_write_spec/`. - Both call `check_mutable` first, matching the other remote mutations. - `maintained_indexes` is sent verbatim (an empty list means "no maintained indexes", matching native semantics). ## Testing - Added mocked-endpoint unit tests for unsharded / bucket / identity set and for unset. - `cargo check --features remote --tests` passes. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>

…lancedb#3459) ## Summary `AsyncTable.search()` computes the query embedding with `loop.run_in_executor(None, ...)`, which uses asyncio's **default** `ThreadPoolExecutor`. That pool is shared with all other `run_in_executor(None, ...)` work, so a slow embedding call — a heavy local model or an HTTP request to an embeddings API — ties up those threads and starves unrelated async I/O under concurrent load. This moves the (potentially blocking) embedding call onto a **dedicated executor**, isolating it from the default pool. Closes lancedb#3310. ## Problem `python/lancedb/table.py`, `AsyncTable.search()`: ```python return ( await loop.run_in_executor( None, # asyncio's default executor, shared with other blocking I/O embedding.function.compute_query_embeddings_with_retry, query, ) )[0] ``` Under load, concurrent searches whose embeddings block (or any other code using the default executor) contend for the same small thread pool. ## Change - Add a dedicated `ThreadPoolExecutor(thread_name_prefix="lancedb-embedding")` in `background_loop.py`, exposed via `embedding_executor()`. - Use it in `AsyncTable.search()`'s `make_embedding` instead of the default executor. - Reset the executor in the existing `_reset_after_fork` hook — its worker threads don't survive `fork()`, same as the background event loop. It's recreated lazily, so this is cheap. ## Design notes The issue asked whether maintainers preferred a configurable executor, a dedicated internal one, or another approach (no response in the thread). I went with a **dedicated internal executor**: it fixes the starvation with no public API change and stays consistent with the existing `LOOP` singleton. Making the pool size configurable would be an easy follow-up if preferred. Scope is limited to `search()`. The broader "embedding functions need real async support" (including `add()`) is tracked separately in lancedb#3268. ## Testing - Added `test_async_search_runs_embedding_on_dedicated_executor`: patches the embedding function to record the executing thread during an async search and asserts it runs on a `lancedb-embedding` thread. Verified it **fails** against the previous `run_in_executor(None, ...)` and passes with the fix. - `ruff format`, `ruff check`, and `pyright` pass on the changed files.

Updates LanceDB Lance dependencies to Lance v8.0.0-beta.4. Includes the required compatibility fix for the new Lance file writer finish summary API. Lance tag: https://github.com/lance-format/lance/releases/tag/v8.0.0-beta.4

BREAKING CHANGE: direct Rust users lose the `IndexStatistics::loss` field. Python and Node.js consumers are unaffected in practice for remote tables (the value was always `None`/absent), but the attribute is gone for local tables too. `IndexStatistics::loss` was local-only — LanceDB Cloud never returned it, so `RemoteTable::index_stats` always set `loss: None`. It's vestigial; this removes it. - Remove `loss` from `IndexStatistics` and the internal `IndexMetadata` in `rust/lancedb/src/index.rs`, plus the summing logic in `NativeTable::index_stats`. - Drop `loss` from the Python and Node.js bindings (and their tests/docs). Fixes lancedb#3493 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>

Updates Lance dependencies from v8.0.0-beta.4 to v8.0.0-beta.5 across the Rust workspace and Java lance-core version. No compatibility code changes were required; clippy and rustfmt pass after installing the missing runner components. Lance tag: https://github.com/lance-format/lance/releases/tag/v8.0.0-beta.5

Updates LanceDB Lance dependencies from v8.0.0-beta.5 to v8.0.0-beta.6 and refreshes Cargo metadata. No compatibility fixes were required; Java lance-core was bumped to 8.0.0-beta.6 as well. Lance tag: https://github.com/lance-format/lance/releases/tag/v8.0.0-beta.6

## Bug Fix ### What is the bug? Namespace-backed `LanceTable.to_arrow()` full-table reads bypassed the existing `QueryTable` server-side query path and called the lower-level table `to_arrow()` implementation directly. In Geneva/Sophon this could fail while parsing the Arrow IPC response for `hist.get_table().to_arrow()` / `to_pandas()`, even though `hist.get_table().search().to_arrow()` worked. ### What issues or incorrect behavior does the bug cause? Full-table reads on namespace-backed tables with `QueryTable` pushdown could fail with Arrow IPC parse errors, while query/search reads on the same table succeeded. Since `to_pandas()` delegates through `to_arrow()` for non-blob/native cases, pandas export was affected too. ### How does this PR fix the problem? When `QueryTable` pushdown is enabled, sync and async table `to_arrow()` now construct a plain no-filter, no-limit, all-columns query and execute it through the table-level `_execute_query()` path. `AsyncTable` now preserves namespace context from async namespace connections so async full reads can make the same pushdown decision. Non-namespace tables and namespace tables without `QueryTable` pushdown keep their existing behavior. ### Tests - `uv run --extra tests --extra dev --no-sync ruff check python/lancedb/table.py python/lancedb/namespace.py python/tests/test_namespace.py` - `uv run --extra tests --extra dev --no-sync ruff format python/lancedb/table.py python/lancedb/namespace.py python/tests/test_namespace.py` - `uv run --extra tests --extra dev --no-sync pytest python/tests/test_namespace.py::TestPushdownOperations::test_lance_table_to_arrow_uses_query_pushdown python/tests/test_namespace.py::TestAsyncPushdownOperations::test_async_table_to_arrow_uses_query_pushdown python/tests/test_namespace.py::test_local_table_to_arrow_and_to_pandas_are_unchanged -q` - `uv run --extra tests --extra dev --no-sync pytest python/tests/test_namespace.py -q`

### Description Adds first-class support for table branches across the Rust core and the Python and TypeScript SDKs. Rust ```rust use lance::dataset::refs::Ref; // Create a branch from main and write to it — main is untouched. let exp = table.create_branch("exp", Ref::Version(None, None)).await?; exp.add(batches).await?; // Reopen the branch later: check out from a table, or open it directly. let exp = table.checkout_branch("exp").await?; let exp = db.open_table("items").branch("exp").execute().await?; let branches = table.list_branches().await?; table.delete_branch("exp").await?; ``` Python ```python # Create a branch from main and write to it branch = await table.branches.create("exp", from_ref="main") await branch.add(data) # Reopen the branch later: check out from a table, or open it directly. branch = await table.branches.checkout("exp") branch = await db.open_table("items", branch="exp") await table.branches.list() await table.branches.delete("exp") ``` TypeScript ```typescript const branches = await table.branches(); // Create a branch from main and write to it const branch = await branches.create("exp"); await branch.add(data); // Reopen the branch later: check out from a table, or open it directly. const checkedOut = await branches.checkout("exp"); const opened = await db.openTable("items", undefined, { branch: "exp" }); await branches.list(); await branches.delete("exp"); ``` ### Testing - Added unit tests - ran smoke tests against python and typescript sdks on local machine ### Next steps - Add RemoteTable support - Add Branch Comparison support - Merge Branching support

### Description Stacked on lancedb#3490. Adds an optional version to branch checkout across the Rust core and the Python and TypeScript SDKs, so you can open a specific version on a branch ("version V of branch B"), not just the branch's latest version Rust ```rust // Open version 3 of branch "exp" (a read-only view): check out from an // existing table, or open it directly from the connection. let exp_v3 = table.checkout_branch("exp", Some(3)).await?; let exp_v3 = db.open_table("items").branch("exp").version(3).execute().await?; // checkout_latest re-attaches to the branch's writable HEAD. exp_v3.checkout_latest().await?; // With no branch, a version opens main at that version. let main_v3 = db.open_table("items").version(3).execute().await?; ``` Python ```python # Open version 3 of branch "exp" (a read-only view): check out from an # existing table, or open it directly from the connection. branch_v3 = await table.branches.checkout("exp", version=3) branch_v3 = await db.open_table("items", branch="exp", version=3) # checkout_latest re-attaches to the branch's writable HEAD. await branch_v3.checkout_latest() # With no branch, a version opens main at that version. main_v3 = await db.open_table("items", version=3) ``` TypeScript ```typescript // Open version 3 of branch "exp" (a read-only view): check out from an // existing table, or open it directly from the connection. const branchV3 = await (await table.branches()).checkout("exp", 3); const opened = await db.openTable("items", undefined, { branch: "exp", version: 3 }); // checkoutLatest re-attaches to the branch's writable HEAD. await branchV3.checkoutLatest(); // With no branch, a version opens main at that version. const mainV3 = await db.openTable("items", undefined, { version: 3 }); ``` ### Testing - Added unit tests (Rust, Python sync + async, TypeScript): branch-scoped resolution at a version number shared with `main` and with another branch, read-only enforcement on a pinned handle, `checkout_latest` recovery to the branch's HEAD, fork-point reads, and the nonexistent-version/branch error paths. - Ran smoke tests against the Python and TypeScript SDKs on local machine.

## What's broken `Table.update(values={...})` raises `NotImplementedError: SQL conversion is not implemented for this type` when a value is a numpy scalar such as `np.int64`, `np.int32`, `np.float32`, or `np.bool_`. These arise naturally from indexing an ndarray or a pandas int/bool column. `np.float64` happens to work (it subclasses `float`), which makes the failure inconsistent and surprising. ```python df = pd.DataFrame({"id": np.array([10, 20], dtype="int32")}) t.update(where="id = 1", values={"id": df["id"].iloc[0]}) # np.int32 # -> NotImplementedError: SQL conversion is not implemented for this type ``` ## Why it happens `value_to_sql` is a `singledispatch` with handlers only for native Python types and `np.ndarray`; numpy `integer`/`floating`/`bool_` scalars aren't Python subclasses, so they fall through to the `NotImplementedError` base. ## Fix Register handlers for `np.bool_`, `np.integer`, and `np.floating` that delegate to the existing native handlers. ## Test `value_to_sql` on `np.int32/int64/float32/float64/bool_` all convert; `np.int32` raised before. Co-authored-by: Ishaan Samantray <[email protected]>

…e' (lancedb#3519) ## Summary Fixes the `NAPI_RS_FORCE_WASI=false` issue by upgrading `@napi-rs/cli` from `3.5.1` to `3.7.0`. Closes lancedb#3267 ## Root Cause In the `native.js` loader generated by `napi build`, the check was: ```js if (!nativeBinding || process.env.NAPI_RS_FORCE_WASI) { ``` In JavaScript, any non-empty string is truthy, so `NAPI_RS_FORCE_WASI=false` (a non-empty string) inadvertently triggered the WASI fallback path. This caused an `ENOENT` error when `lancedb.wasi.cjs` was not present. ## Fix `@napi-rs/[email protected]` ([napi-rs/napi-rs#3236](napi-rs/napi-rs#3236)) introduced a tri-state check in the template that generates `native.js`: **Before (generated by @napi-rs/[email protected]):** ```js if (!nativeBinding || process.env.NAPI_RS_FORCE_WASI) { ``` **After (generated by @napi-rs/[email protected]):** ```js const forceWasi = process.env.NAPI_RS_FORCE_WASI === 'true' || process.env.NAPI_RS_FORCE_WASI === 'error' if (!nativeBinding || forceWasi) { ``` Only the literal string `'true'` (or `'error'` for strict mode) now activates the WASI path. All other values, including `'false'`, `'0'`, or an unset variable, behave as if WASI is not forced. ## Changes - `nodejs/package.json`: bump `@napi-rs/cli` from `3.5.1` to `3.7.0` - `nodejs/package-lock.json` / `nodejs/pnpm-lock.yaml`: update lock files to match The fix is in the upstream napi-rs tool; the generated `native.js` is not committed to this repository and is produced at build time by `napi build`.

## Summary This PR extends nested-field regression coverage across Rust local/remote, Python sync/async, and Node so canonical escaped paths stay consistent across scalar, vector, and FTS index lifecycle behavior. It also aligns LanceDB's LabelList type gate with Lance by accepting `LargeList<primitive>` columns while keeping `List<Struct<...>>` unsupported until Lance defines stable membership semantics for struct labels. Part of lancedb#3406.

…de for pre-Haswell Companion to the lance-side runtime SIMD dispatch landing in tobocop2/lance:fix/runtime-simd-multiversion (will swap to an upstream lance-format/lance release once that PR merges). Per westonpace's review on lancedb#3324 (fast by default, extra steps to work on legacy), the wheel build .cargo/config.toml baselines stay at target-cpu=haswell + +avx2,+fma,+f16c for both linux-gnu and linux-musl. Pre-Haswell users build from source with RUSTFLAGS=-C target-cpu=x86-64-v2 to opt into the runtime-dispatch path the embedded lance crate now provides. Changes: - Bump 14 lance crate deps from lance-format/lance.git tag=v6.0.0-beta.1 to tobocop2/lance.git rev=026346c25892fbeb91a4144ecf67d739f987c9d2; version bumped to =6.0.0-beta.3 so a cargo update against upstream's tag doesn't silently overwrite the fork-rev pin. - Add net.git-fetch-with-cli = true so cargo honors git auth when fetching the fork dep over SSH (comes out once we depend on a release). - python/README.md: document the from-source build path (RUSTFLAGS=-C target-cpu=x86-64-v2 maturin build --release). Verified end-to-end on Sandy Bridge Xeon E5-2609: pre-PR `pip install lancedb` SIGILLs at import; post-PR a from-source build with the RUSTFLAGS override produces a wheel where import + table create + vector search all work at the AVX tier. Closes #1.

…-source builds (#6630) Tracks #6618. On x86_64 CPUs without AVX2 — Sandy Bridge / Ivy Bridge / Westmere on Intel, Bulldozer / Piledriver / Steamroller on AMD — `import lancedb` SIGILLs because the wheel bakes AVX2 + FMA into every compiled function with no runtime guard. numpy and pyarrow handle the same hardware via runtime CPU dispatch. ## Summary - Adds 5-tier runtime SIMD dispatch (scalar / AVX / AVX+FMA / AVX2+FMA / AVX-512) to the f32/f64 hot kernels in `lance-linalg::distance::{cosine, dot, l2, norm_l2}`. Same `match *SIMD_SUPPORT` + `mod x86 { #[target_feature] pub unsafe fn ... }` shape as `dot_u8.rs` / `cosine_u8.rs` / `l2_u8.rs`. Where the AVX2 and AVX+FMA kernel bodies use no AVX2-specific intrinsics, the dispatch matches `Avx2 | AvxFma` to a shared kernel. - Adds `lance.simd_info()` Python introspection mirroring `pyarrow.runtime_info()` so users can verify which tier the runtime selected. - Adds a `qemu-pre-haswell` CI job that builds with `RUSTFLAGS="-C target-cpu=x86-64-v2"` (env-var-scoped to that one job — workspace `.cargo/config.toml` is unchanged) and runs `lance-linalg` lib tests under `qemu-x86_64 -cpu Nehalem`. - Documents the legacy build path in `CONTRIBUTING.md`: `RUSTFLAGS="-C target-cpu=x86-64-v2" cargo build --release`. Per [westonpace's review on lancedb/lancedb#3324](lancedb/lancedb#3324 (comment)), the workspace baseline stays at `target-cpu=haswell`. Modern wheels are unchanged; legacy users opt into the lower baseline at build time. ## Benchmark The AVX2 path on modern hardware is preserved as one of the per-tier kernels and the workspace baseline still bakes AVX2 into surrounding code, so by construction the modern compile is unchanged. Numbers still pending — Codespace's 30-min idle timeout killed my last full `cargo bench -p lance-linalg --bench {cosine,dot,l2,norm_l2}` run mid-suite (even with `nohup` — the VM itself sleeps). If anyone can recommend a free resource that holds a benchmark for ~1 hour, or a maintainer-preferred narrower bench shape, I'd appreciate the pointer. Pre-Haswell verification on Sandy Bridge Xeon E5-2609 (the hardware the published wheel SIGILLs on) via the companion lancedb wheel build: pre-PR `pip install lancedb` SIGILLs at import; post-PR a from-source build with the documented `RUSTFLAGS` override produces a wheel where import + table-create + vector-search all work at the AVX tier. PASS output in [`tobocop2/lancedb#2`](tobocop2/lancedb#2). ## Incidental fix: FMA not verified before the AVX2 tier While auditing the dispatch table for this PR I found a latent soundness bug that predates it. On `main` the tier is chosen with `is_x86_feature_detected!("avx2")` alone, but every kernel the AVX2 tier dispatches to is `#[target_feature(enable = "avx,fma")]`. AVX2 does not imply FMA in the x86 ISA, so a host with AVX2 but no FMA would select that tier and execute `vfmadd` — an illegal instruction. No shipping AVX2 part lacks FMA, so this is latent rather than live, but it is exactly the class of fault this PR exists to remove. The [fix](bb8af46) checks FMA explicitly, documents the tier's contract, and adds a test that fails if any tier is ever selected on a host that cannot run its kernels. Tracked separately as #7732 so it stays on the record independent of the perf work. ## Test plan - [x] `cargo test -p lance-linalg --lib` — 83/83 on aarch64 dev box - [x] `cargo clippy --all-targets -- -D warnings` clean; `cargo fmt --check` clean; `Cargo.lock` unchanged - [x] 11 proptest cases verifying scalar↔SIMD bit-for-bit equivalence per tier per kernel; gated on `is_x86_feature_detected!()` so each runs on hosts that can execute its tier - [x] SIGILL repro confirmed gone on Sandy Bridge Xeon E5-2609 with the documented `RUSTFLAGS` override - [ ] qemu-pre-haswell CI gate green (lights up when this PR runs CI) - [x] Modern-hardware bench delta posted --- To be transparent: this isn't my domain of expertise and the implementation is AI-generated — I stuck to the existing `mod x86 { #[target_feature] }` precedent and verified end-to-end on the failing hardware, but wanted to be upfront. Happy to roll in feedback.  ## Summary by CodeRabbit * **New Features** * Added `lance.simd_info()` (Python) to report the runtime SIMD tier, target architecture, and detected host features. * **Bug Fixes** * Improved runtime SIMD dispatch and scalar fallback behavior for cosine, dot, L2 distance, and L2 norms, including more accurate tier handling across mixed AVX/AVX+FMA capability. * **Documentation** * Added instructions for building on legacy x86_64 (pre-Haswell) hosts and verifying SIMD selection. * **Tests** * Added `simd_info()` tests and expanded dispatched-vs-scalar parity coverage, plus a new regression seed. * **Chores** * Enhanced CI with QEMU-based SIMD-tier validation on a lower x86-64 baseline.  --------- Co-authored-by: Trenton Holmes <[email protected]> Co-authored-by: Claude Opus 4.8 <[email protected]>

…3599) ## What `MRRReranker.rerank_multivector` averages each document's reciprocal ranks over the wrong denominator. It divides by the number of rankings the document *happens to appear in*, instead of the total number of rankings being fused. ```python # python/python/lancedb/rerankers/mrr.py for result_id, reciprocal_ranks in mrr_score_map.items(): mean_rr = np.mean(reciprocal_ranks) # divides by len(present systems) ``` `mrr_score_map[doc]` only accumulates a reciprocal rank for the systems in which the document was returned, so `np.mean` never accounts for the systems that missed it. ## Why it's wrong Mean Reciprocal Rank fusion treats a system that didn't return a document as a reciprocal rank of `0` and averages across **all** systems. That's the exact mechanism by which it rewards cross-system consensus. Dividing by the appearance count removes that, so a document liked by a single ranking can beat one ranked highly by every ranking. Concretely, fusing 3 vector rankings: | Doc | Ranks | Current score | Correct score | |-----|-------|---------------|---------------| | A | #1 in 1 system only | `mean([1.0]) = 1.000` | `1.0 / 3 = 0.333` | | B | #1, #1, #2 across all 3 | `mean([1, 1, .5]) = 0.833` | `2.5 / 3 = 0.833` | The current code ranks **A above B** - a document two of three rankings ignored outranks one all three ranked at or near the top. This also makes `rerank_multivector` inconsistent with `rerank_hybrid` in the same file, which already treats a missing system as `0` (`vector_rr = 0.0` / `fts_rr = 0.0`), and with the class docstring ("average of reciprocal ranks across different search results"). ## Fix Divide the summed reciprocal ranks by the total number of rankings: ```python num_systems = len(vector_results) ... mean_rr = float(np.sum(reciprocal_ranks)) / num_systems ``` ## Tests Adds `test_mrr_multivector_rewards_consensus`, which asserts the exact MRR scores and that the consensus document ranks first. It fails on `main` and passes with this change. Existing reranker tests are unaffected.

) Tracks #3324. On x86_64 CPUs without AVX2 (Sandy Bridge / Ivy Bridge / Westmere on Intel; Bulldozer / Piledriver / Steamroller on AMD), `import lancedb` SIGILLs because the wheel bakes AVX2 + FMA into every compiled function. Per [westonpace's review](#3324 (comment)), the default `lancedb` wheel stays fast; pre-Haswell users get a separately-published `lancedb-compat` wheel. ## Summary - Adds a `lancedb-compat` matrix entry to `pypi-publish.yml` that builds with `RUSTFLAGS="-C target-cpu=x86-64-v2"` (Nehalem-class baseline). Same Python API (`import lancedb` works) — files install to the same namespace, so the two wheels conflict at install time and users pick one. Same pattern as `psycopg2` / `psycopg2-binary` and `tensorflow` / `tensorflow-cpu`. - Generalizes `build_linux_wheel` and `upload_wheel` composites with optional `package-name` and `rustflags` inputs (defaults preserve the existing 4 `lancedb` matrix entries verbatim). - Documents the choice in `python/README.md`: `pip install lancedb-compat` for pre-Haswell hosts. The default `.cargo/config.toml` baseline is unchanged. ## Sequencing 1. ~~lance-format/lance#6630 merges → runtime SIMD dispatch lands in lance.~~ **Done — merged.** 2. lancedb's lance dep is bumped to a release that includes it (separate PR / normal cadence). 3. This PR's `lancedb-compat` wheel build path starts producing a wheel that runs on pre-Haswell hardware. **Maintainer setup**: register `lancedb-compat` on PyPI and configure trusted publishing. ## Verified end-to-end on Sandy Bridge Xeon E5-2609 Verification was done locally against a fork-pinned lance dep that includes the runtime dispatch implementation, using the same `RUSTFLAGS="-C target-cpu=x86-64-v2"` flags this PR uses in CI: ``` $ RUSTFLAGS="-C target-cpu=x86-64-v2" maturin build --release $ pip install ./target/wheels/lancedb-*.whl $ python verify.py PASS: import + simd dispatch + table create + vector search all work. ``` Pre-fix on the same CPU (default `pip install lancedb`): `Illegal instruction (core dumped)`. Full reproducer (deps + clone + build + verification): https://gist.github.com/tobocop2/2e341358b55c143527416edfdb1e37df. Fork-internal verification PR with the dep bump and full logs: [`tobocop2#2`](tobocop2#2). ## Benchmarks — no regressions on modern CPUs from the lance-side change These are the numbers I ran for the lance PR, confirming the runtime dispatch doesn't slow down the default (`target-cpu=haswell`) wheel that existing users install. Criterion, one machine, one session, base → PR, no `RUSTFLAGS` override. Full methodology, null experiments, and logs: [lance-format/lance#6630 benchmark comment](lance-format/lance#6630 (comment)) and the [logs gist](https://gist.github.com/tobocop2/3c6d0f449cbd736aa2501f89a7fe56a2). | benchmark | EPYC 7B13 (`avx2`, `fma`, no `avx512f`) | Xeon Cascade Lake (`avx512f`) | |---|---|---| | `Cosine(f32, scalar)` *(control)* | +0.04% | +0.09% | | `Cosine(f64, scalar)` | −0.34% | −1.94% | | `Cosine(u8, SIMD)` | +2.30% | +3.63% | | `Dot(f16, SIMD)` | −0.58% | +0.61% | | `Dot(f32, SIMD)` | +0.34% | **−6.08%** | | `Dot(f32, arrow_arity)` | +0.02% | −0.00% | | `L2(f32, scalar)` | −0.10% | −0.02% | | `L2(f32, simd)` (dim 1024) | +2.63% | −0.53% | | **`L2(simd,f32x8)` (dim 8)** | **−45.9%** | **−25.1%** | | `L2(u8, SIMD)` | +0.42% | −3.11% | | `NormL2(f32, SIMD)` | −1.02% | −4.17% | | `NormL2(f64, SIMD)` | +3.51% | −0.58% | Nothing regresses beyond the noise floor. Dim 8 — the PQ sub-vector width — improves 25–46%. --- To be transparent: this isn't my domain of expertise and the lance-side implementation is AI-generated. I verified it works end-to-end on the failing hardware. Happy to roll in feedback.

tobocop2 changed the title ~~Bump lance to runtime-dispatch fix + lower wheel baseline to x86-64-v2 (companion to tobocop2/lance#2)~~ build: lancedb wheel SIGILLs on pre-Haswell x86_64 — bump lance + lower baseline Apr 26, 2026

tobocop2 changed the title ~~build: lancedb wheel SIGILLs on pre-Haswell x86_64 — bump lance + lower baseline~~ build: lancedb wheel unusable on pre-Haswell x86_64 (import SIGILL) — bump lance + lower wheel baseline Apr 26, 2026

lancedb-robot and others added 2 commits April 27, 2026 15:13

This was referenced Apr 28, 2026

feat(lance-linalg): runtime SIMD dispatch for pre-Haswell x86_64 from-source builds lance-format/lance#6630

Merged

feat(wheels): publish lancedb-compat for pre-Haswell x86_64 hosts lancedb/lancedb#3327

Merged

tobocop2 force-pushed the fix/runtime-simd-pre-haswell branch from 9268796 to 83d6d15 Compare April 28, 2026 05:00

github-actions Bot added Python chore labels Apr 28, 2026

tobocop2 changed the title ~~build: lancedb wheel unusable on pre-Haswell x86_64 (import SIGILL) — bump lance + lower wheel baseline~~ build: bump lance to runtime-dispatch fix; document RUSTFLAGS override for pre-Haswell Apr 28, 2026

Lance Release and others added 14 commits April 28, 2026 13:29

Bump version: 0.31.0-beta.9 → 0.31.0-beta.10

2e36cd9

Bump version: 0.28.0-beta.9 → 0.28.0-beta.10

4dcd7f4

Bump version: 0.31.0-beta.10 → 0.31.0-beta.11

a2aea7b

Bump version: 0.28.0-beta.10 → 0.28.0-beta.11

c091243

devteamaegis and others added 25 commits June 3, 2026 14:05

ci: update python lockfile weekly (lancedb#3498)

a16676e

Make sure we are getting security fixes in there regularly, and other useful bumps.

Bump version: 0.33.1-beta.1 → 0.33.1-beta.2

952055d

Bump version: 0.30.1-beta.1 → 0.30.1-beta.2

39a9f3e

tobocop2 force-pushed the fix/runtime-simd-pre-haswell branch from 83d6d15 to dd4d03e Compare June 10, 2026 16:13

mbabic84 mentioned this pull request Jun 27, 2026

semantic_search crashes with SIGILL on pre-Haswell CPUs (LanceDB AVX2 requirement) mbabic84/kilo-docker#223

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

build: bump lance to runtime-dispatch fix; document RUSTFLAGS override for pre-Haswell#2

build: bump lance to runtime-dispatch fix; document RUSTFLAGS override for pre-Haswell#2
tobocop2 wants to merge 112 commits into
mainfrom
fix/runtime-simd-pre-haswell

tobocop2 commented Apr 26, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Apr 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

19 participants

Conversation

tobocop2 commented Apr 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Verified end-to-end on Sandy Bridge Xeon E5-2609

Test plan

Uh oh!

github-actions Bot commented Apr 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

19 participants

tobocop2 commented Apr 26, 2026 •

edited

Loading