CacheChecker.run issues one filter_documents call per item (N+1 query pattern)

[Harsh23Kashyap]

CacheChecker.run issues one filter_documents call per item (N+1 query pattern)

Bug

CacheChecker.run (haystack/components/caching/cache_checker.py:75) loops over the input items list and calls self.document_store.filter_documents(...) once per item:

for item in items:
    filters = {"field": self.cache_field, "operator": "==", "value": item}
    found = self.document_store.filter_documents(filters=filters)
    if found:
        found_documents.extend(found)
    else:
        misses.append(item)

For an input of N items, this is N round-trips to the document store. For InMemoryDocumentStore the constant is small; for QdrantDocumentStore, WeaviateDocumentStore, OpenSearchDocumentStore, etc., every call is a network request. With a 10 ms per-call latency, 200 items takes ~2 s; 5 000 items (a realistic batch) takes ~50 s.

The same pattern is duplicated in CacheChecker.run_async (cache_checker.py:99).

The Haystack filter spec already supports the in operator (haystack/document_stores/types/protocol.py:70, haystack/utils/filters.py:262), so the whole check can be done in a single call.

Reproduction

from haystack import Document
from haystack.components.caching.cache_checker import CacheChecker
from haystack.document_stores.in_memory import InMemoryDocumentStore

store = InMemoryDocumentStore()
store.write_documents(
    [Document(content=f"d{i}", meta={"url": f"https://example.com/{i}"}) for i in range(200)]
)

calls = 0
orig = store.filter_documents
def counting(*a, **kw):
    global calls; calls += 1
    return orig(*a, **kw)
store.filter_documents = counting

CacheChecker(store, cache_field="url").run(
    items=[f"https://example.com/{i}" for i in range(200)]
)
print("filter_documents calls:", calls)   # -> 200

For any non-in-memory store, each call is a separate network round-trip.

Expected behavior

A single filter_documents call should be issued for the whole items list, using the in operator:

filters = {"field": self.cache_field, "operator": "in", "value": items}
found = self.document_store.filter_documents(filters=filters)

misses should be derived from the items whose value did not appear in the result set, preserving the current return contract.

Why this matters

• Cache-checking on URL- or ID-keyed lookups is a common pattern for Web RAG pipelines. Today a 1 000-URL crawl triggers 1 000 round-trips against the document store.
• The proposed change is a pure performance fix: semantics of the return value are preserved (verified locally on InMemoryDocumentStore: identical hits set and misses list for inputs with duplicates, missing values, and nested cache_field paths).

Proposed fix

@component.output_types(hits=list[Document], misses=list)
def run(self, items: list[Any]) -> dict[str, Any]:
    if not items:
        return {"hits": [], "misses": []}
    filters = {"field": self.cache_field, "operator": "in", "value": items}
    found = self.document_store.filter_documents(filters=filters) or []
    seen = {self._get_field_value(d) for d in found}
    misses = [i for i in items if i not in seen]
    return {"hits": list(found), "misses": misses}

(Plus a _get_field_value helper that mirrors the field-path handling in haystack/utils/filters.py:304, and the same change in run_async.)

Acceptance criteria

• CacheChecker.run issues exactly one filter_documents call regardless of len(items).
• Returned hits and misses are identical to the current implementation for: single item, many items, items with duplicates, items all missing, items partially missing, and a nested cache_field like "meta.foo.bar".
• run_async mirrors the change and still issues exactly one filter_documents_async call.
• Existing test_filters_syntax is updated to assert the new single-call filter shape.
• New unit tests cover the per-call count and the miss-derivation logic.

Backward compatibility

• Public API (run / run_async signatures, return type, return value semantics) is unchanged.
• The new filter shape is observable only through document_store.filter_documents calls and the test_filters_syntax assertion, both internal.

Risks

• Some document stores may have a practical limit on in-list size. If a back-end rejects large lists, falling back to the existing per-item loop is a straightforward mitigation.
• The miss-derivation assumes cache_field values are hashable. The current == filter requires the same, so no new constraint is introduced.

[Ayush-yadav11]

Hi, I would like to work on this issue.

I plan to update both CacheChecker.run and CacheChecker.run_async to use a single filter_documents call with the in operator. I will preserve the existing hits and misses behavior, including duplicate items and nested cache fields, and add tests for the call count and miss derivation.
Would you be happy for me to take this issue?

[nata2627]

@Ayush-yadav11 apologies — I had a finished branch for this before your message and only saw it as I was about to push, so I've opened it as #12539 rather than sit on it. It takes the approach you describe; the one thing that took the time was that deriving misses from the returned documents in Python changes behaviour in three ways, so the mapping is done with document_matches_filter instead.

If you would rather take this one, say so and I'll close mine — no version of this is worth two evenings.