feat: add Apache Solr integration - #3860
Open
julian-risch wants to merge 7 commits into
Open
Conversation
Add `solr-haystack`, providing `SolrDocumentStore` plus BM25, embedding and
hybrid retrievers. Solr has had `DenseVectorField` and the `{!knn}` query
parser since 9.0, so it can serve both halves of a RAG pipeline.
Talks to Solr's JSON APIs over `httpx` rather than a third-party client:
pysolr has no async support at all, aiosolr has no delete method, and solrpy
is beta with no license file. One httpx client pair gives sync and async from
a single code path.
Metadata is stored in fields whose names encode the value's Python type
(`meta_s_page` vs `meta_l_page`), because Solr fixes a field's type on
creation while Haystack metadata types are only known at write time. This
keeps the round-trip exact and keeps values that share a string form -- the
int 1, the str "1", the float 1.0 and True -- distinct.
Requires Solr 9.6, which is where dependable k-NN pre-filtering landed
(SOLR-16858); the store checks the server version on first use.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Contributor
Coverage report (solr)
Click to see where and how coverage changed
This report was generated by python-coverage-comment-action |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
github-actions
Bot
added
topic:CI
type:documentation
`_apply_fuzziness` appended the fuzzy suffix to every whitespace-delimited token, quoted phrases included, so `"Apache Solr"` at `fuzziness=1` went out as `"Apache~1 Solr"~1`. The `~1` inside the quotes is analysed into the term rather than read as syntax, so the phrase matched nothing at all: verified against Solr 10.0.0, the old form returned zero documents where the phrase should have matched one. Tokens containing a quote are now left alone, and `+"a b"` counts as a single token so a prefixed phrase is not split. Substituting rather than re-joining keeps every character of a malformed query. Also: - `SolrHybridRetriever` takes `document_store` positionally, matching the Elasticsearch and OpenSearch hybrid retrievers. The BM25 and embedding retrievers keep it keyword-only, which is what their counterparts in every other integration do. - Restore the unrelated `amazon_bedrock` row in the root README, picked up from the integration scaffolding script. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
9 tasks
`_existing_ids_payload` escaped the ids it handed `{!terms}`, but the terms
query parser takes its values verbatim - it never unescapes - so the
backslashes became part of the term and the ids stopped matching. A comma
broke it a second way, by colliding with the value separator.
Every character `escape_query_chars` touches was affected, plus whitespace and
the comma:
! " & ( ) * + - / : ; ? [ \ ] ^ { | } ~ space tab ,
Both duplicate policies then failed silently, and `SKIP` lost data. Verified
against Solr 10.0.0 with `id="a-b"`: `FAIL` did not raise and overwrote the
document, and `SKIP` replaced the stored content instead of leaving it alone.
Haystack's own hash ids contain none of these characters, which is why the
shared write suites did not catch it.
Asking the real-time get handler instead removes the class of bug rather than
the instance: the ids travel as repeated `id` parameters that nothing parses,
so there is no escaping to get wrong and no separator to collide with. The
parameters go in the request body, which a query string could not hold - 500
64-character ids overflow it.
That also fixes a second, independent bug in the same lookup. The check was a
search, so it could not see documents written but not yet committed: with
`commit=False` or `commit_within_ms` set, duplicate detection silently
degraded for every id, hash ids included. A real-time get sees them. On a core
configured without an update log it falls back to the committed index, which
is no worse than the search it replaces.
Tests: the payload builder and both response shapes - one id answers with a
bare `doc`, several with a `response` block - plus integration coverage for
each affected character, a mixed batch, and the uncommitted case, sync and
async. Confirmed the new tests fail against the old implementation, with the
`plain` id as a control, and confirmed the suite still passes on Solr 9.10.1.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Unit coverage was 73% overall and 47% for `document_store.py`. Every method there issues a request as its first act, so a unit test never got past that line and the whole surface - all the deletes, the updates, the four metadata introspection calls, and both `_async` twins of everything - was reachable only from the integration suite. Solr responses are dicts and each method reads only the keys it cares about, so one canned payload carrying all of them answers every endpoint. That turns the surface into a table: one row per call, giving the endpoint it has to reach and what it should make of the response. A second table repeats the exercise against a Solr holding nothing, which is where the early returns live. Coverage is now 99% overall, and `document_store.py` has no uncovered statement. More to the point, the tables pin two things worth pinning: which endpoint each call uses - the deduplicating write has to reach `/get`, and unique values has to reach `/select`, since only classic facets can filter buckets by substring - and the shape each one reads back. Also closes gaps the coverage numbers were hiding: the async bootstrap (untested until now, including core creation and turning schemaless guessing off), `cursorMark` paging past the first page in both paginating readers, and `from_`/`size` on unique values, whose total must count every matching value rather than the page. Verified with a mutation sweep rather than the coverage figure alone: sending the dedup lookup back to `/query`, parsing existing ids as always-empty, stopping pagination after one page, swapping min for max, dropping boolean facet decoding, ignoring the pagination window, and skipping either half of the async bootstrap are each caught. The pagination hole was found that way - the tables alone had missed it. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The two tables carried a sync and an async lambda per row, and a second pair of tables repeated the exercise for an empty core - four parametrized tests and a lambda for every call. Naming the method instead of wrapping it lets one table serve both halves, since the async twin is always the same name plus `_async`, and lets the populated and empty cases share it as a payload column. Sixteen calls plus nine early returns now read as twenty-five single lines. Coverage is unchanged at 99% with no uncovered statement in `document_store.py`, and the mutation sweep still catches all nine breakages. It also caught a tenth. `test_json_encoded_metadata_cannot_be_faceted` was vacuous: the fake offered no buckets for the JSON field, so a store that forgot to skip it returned an empty list and the test passed anyway. The fake now offers buckets it must refuse to read. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
julian-risch
commented
Aug 27, 2026
julian-risch
commented
Aug 27, 2026
It ran to 142 lines against the 15-20 every other document store keeps, so it now matches `opensearch/README.md`: title, badges, the two links, and a contributing note. Code examples are going into the documentation proper instead. The container note keeps `--wait`, since without it the tests race Solr's startup, and gains the Solr 9 image override that the old Development section carried. That override now says to recreate the volume first: the Solr image declares a volume for its data directory and Compose reuses it, so an older Solr meets a newer one's index and the core fails to load with an error that reads like a bug in the integration rather than stale state. `manage_schema`'s docstring pointed at the README for the list of fields a self-managed schema needs, which the trim would have left dangling. It now points at `schema.schema_payload`, which is the definitive list, is published by `hatch run docs`, and cannot drift from the code that builds the schema. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Co-authored-by: Julian Risch <[email protected]>
julian-risch
marked this pull request as ready for review
August 27, 2026 12:42
julian-risch
requested review from
bogdankostic and
davidsbatista
and removed request for
a team and
bogdankostic
August 27, 2026 12:42
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related Issues
Proposed Changes:
Adds
solr-haystack, an Apache Solr integration. Solr has shippedDenseVectorFieldand the{!knn}query parser since 9.0, so it can serve both halves of a RAGpipeline
SolrDocumentStoreSolrBM25Retriever,SolrEmbeddingRetriever,SolrHybridRetrieverscripts/create_new_integration.py, so.github/labeler.yml,CI_coverage_comment.ymlandREADME.mdare added/updatedHow did you test it?
DocumentStoreBaseExtendedTests,CountDocumentsByFilterTest,CountUniqueMetadataByFilterTest,GetMetadataFieldsInfoTest,GetMetadataFieldMinMaxTestandGetMetadataFieldUniqueValuesTest, plus async.-n 4safe.Notes for the reviewer
Design decisions to check in the review:
httpxrather than a Solr client.pysolr(the maintained one) has no async support at all,aiosolrhas no delete method, andsolrpyis 4-Beta with no license file. Solr's surface here issmall and entirely JSON, so one
httpx.Client/AsyncClientpair gives sync and async from asingle code path.
meta_s_pagefor"100",meta_l_pagefor
100). Solr fixes a field's type when the field is created, while Haystack metadata types areonly known at write time. This keeps the round-trip exact, and keeps values that merely share a
string form — the int
1, the str"1", the float1.0andTrue— as four distinct values.The type code is a prefix because Solr dynamic fields only allow a leading or trailing wildcard,
so
meta_*_sis not a legal pattern.field:-10is a Lucene syntax error, andfield:(-10 OR 100)silently parses as "100 but not10". Negative numerics are therefore escaped in
filters.py, with regression tests covering equality, ranges and groupedinclauses.Checklist
fix:,feat:,build:,chore:,ci:,docs:,style:,refactor:,perf:,test:.🤖 Generated with Claude Code