feat(eval-harness): default retrieval to auto search via strategy module

Move context retrieval into retrieval_strategy.build_context_block so evals use scope=auto with a 10k character budget, prepend the user-node summary, and drop per-type search limit constants.

Co-authored-by: Cursor <[email protected]>

Author: jackaldenryan

zep-eval-harness/.claude/commands/zep-eval-harness/SKILL.md

@@ -15,7 +15,7 @@ The pipeline has four steps:
 
 ### Scope: Single-Shot Retrieval
 
-**The harness evaluates single-shot retrieval only.** Every test case issues one fixed batch of scoped searches from the raw test question (nodes + edges, optionally episodes, across the user graph and any document graph), then hands the resulting context block to the response model in a single turn — no second retrieval round, no query reformulation. This mirrors deterministic/programmatic retrieval, not the tool-based pattern where an agent is handed Zep search tools (e.g. `search_graph` from the Zep MCP server) and decides when and what to search.
+**The harness evaluates single-shot retrieval only.** Every test case issues one retrieval from the raw test question via auto search (`config/evaluation_config/retrieval_strategy.py`), then hands the resulting context block to the response model in a single turn — no second retrieval round, no query reformulation. This mirrors deterministic/programmatic retrieval, not the tool-based pattern where an agent is handed Zep search tools (e.g. `search_graph` from the Zep MCP server) and decides when and what to search.
 
 That makes the harness a clean instrument for the ingestion and search configuration, but it says nothing about agent tool-use behavior. A tool-based agent may do better (several targeted searches, reformulating after a weak result) or worse (never searching, poorly phrased queries, running out of turns). When reporting results, scope conclusions to the config under test and never present them as a prediction of production agent performance.
 
@@ -28,7 +28,7 @@ That makes the harness a clean instrument for the ingestion and search configura
 
 This is the metric that matters most. It directly measures Zep's retrieval quality — whether the knowledge graph and search surface the right facts, entities, and relationships.
 
-When completeness is low, the key diagnostic question is: **does the graph contain the right information but search failed to retrieve it, or is the information missing from the graph entirely?** Use `zep_graph_inspect.py` to examine what's actually in the graph. If the information is there but not retrieved, the issue is search configuration (limits, reranker, query phrasing). If the information is absent from the graph, the issue is upstream — ingestion, ontology, or custom instructions need adjustment.
+When completeness is low, the key diagnostic question is: **does the graph contain the right information but search failed to retrieve it, or is the information missing from the graph entirely?** Use `zep_graph_inspect.py` to examine what's actually in the graph. If the information is there but not retrieved, the issue is retrieval strategy (auto-search character budget, query phrasing). If the information is absent from the graph, the issue is upstream — ingestion, ontology, or custom instructions need adjustment.
 
 **Answer Accuracy (SECONDARY)** — Did the LLM produce a correct answer from the retrieved context?
 - **CORRECT**: Answer conveys the same key information as the golden answer
@@ -38,15 +38,13 @@ This measures whether the response model uses the context well. It depends on th
 
 Metrics are calculated in aggregate, per-category (based on test case `category` field), and per-user.
 
-### Context Block: Edges vs Nodes vs Episodes
+### Context Block: Auto Search
 
-The evaluation script constructs a context block from graph search results. Understanding what each component contributes:
+The evaluation script retrieves context via `build_context_block()` in `config/evaluation_config/retrieval_strategy.py`. The default strategy uses `scope="auto"` with `MAX_CHARACTERS = 10000`, and prepends the user-node summary (fetched separately — auto search does not include it).
 
-- **Edges (facts)**: Relationships extracted by Zep between entities — e.g., "Sarah WORKS_FOR TechCorp", "Biscuit IS_OWNED_BY Sarah". These are the primary source of structured knowledge. Controlled by `USER_FACTS_LIMIT` / `DOC_FACTS_LIMIT`.
-- **Nodes (entities)**: Entity summaries — e.g., a Person node with name, relationship type, and a description synthesized from all conversations mentioning them. Controlled by `USER_ENTITIES_LIMIT` / `DOC_ENTITIES_LIMIT`.
-- **Episodes (raw data)**: The original messages, document chunks, or JSON data that was ingested. These provide verbatim source text but are bulkier. Disabled by default (`*_EPISODES_LIMIT = 0`). Enable by setting limits > 0 in `config/evaluation_config/constants.py`.
+Auto search packs relevant edges (facts), nodes (entities), episodes, observations, and thread summaries into a pre-assembled context string. ``limit`` and ``reranker`` do not apply under auto — volume is controlled by the character budget.
 
-Most evaluations work best with edges + nodes (structured, concise). Enable episodes when verbatim source text is needed for answering questions that require exact quotes or details not captured in the graph extraction.
+To change retrieval (e.g. manual multi-scope searches with per-type limits and a reranker), edit `build_context_block` in that module. That function is the source of truth for the strategy.
 
 All commands run from `zep-eval-harness/` using `uv run`. Depending on the user's request, either run the scripts directly or provide the terminal commands for the user.
 
@@ -76,7 +74,8 @@ config/
 ├── document_chunking_config/
 │   └── constants.py                    # CHUNK_SIZE, CHUNK_OVERLAP, LLM_CONTEXTUALIZATION_MODEL
 └── evaluation_config/
-    ├── constants.py                    # Search limits, LLM_RESPONSE_MODEL, LLM_JUDGE_MODEL
+    ├── constants.py                    # LLM_RESPONSE_MODEL, LLM_JUDGE_MODEL
+    ├── retrieval_strategy.py           # build_context_block() — retrieval + context assembly
     └── response_prompt.py              # get_response_system_prompt() — AI persona for eval
 ```
 

zep-eval-harness/README.md

@@ -174,20 +174,20 @@ Rate limits are handled automatically — if you hit limits, the retry backoff w
 
 ### Pipeline Steps (automated in zep_evaluate.py)
 
-1. **Search**: Query Zep's knowledge graph (nodes, edges) using cross-encoder reranker
+1. **Search**: Query Zep's knowledge graph with ``scope="auto"`` (character-budgeted context block)
 2. **Evaluate Context**: Assess whether retrieved context contains sufficient information (PRIMARY METRIC)
 3. **Generate Response**: Use LLM with retrieved context to answer questions
 4. **Grade Answer**: Evaluate answers against golden answers using LLM judge (SECONDARY METRIC)
 
 ### Scope: Single-Shot Retrieval
 
-**This harness evaluates single-shot retrieval only.** Each test case issues one fixed batch of searches — the raw test question against nodes and edges (plus episodes if enabled), across the user graph and any document graph, all in parallel — assembles the results into a context block, and hands it to the response model in a single turn. There is no second retrieval round and no query reformulation. That mirrors *deterministic/programmatic* retrieval, where your application searches on every turn and injects the context itself.
+**This harness evaluates single-shot retrieval only.** Each test case issues one retrieval — the raw test question via auto search on the user graph (and any document graph), using the strategy in `config/evaluation_config/retrieval_strategy.py` — and hands the resulting context block to the response model in a single turn. There is no second retrieval round and no query reformulation. That mirrors *deterministic/programmatic* retrieval, where your application searches on every turn and injects the context itself.
 
 Production agents are frequently built the other way: Zep is exposed to the model as **tools** — for example `search_graph` from the [Zep MCP server](../mcp/zep-mcp-server/), or your own tool definitions — and the LLM decides when to search, how to phrase each query, and whether to search again after seeing results.
 
 Read the scores accordingly:
 
-- **What they measure**: whether your ingestion and search configuration (ontology, custom instructions, chunking, search limits, reranker) puts the right facts within reach of one well-formed query. Pinning retrieval to a single deterministic search is what keeps runs comparable — the config stays the only variable.
+- **What they measure**: whether your ingestion and retrieval strategy (ontology, custom instructions, chunking, auto-search character budget) puts the right facts within reach of one well-formed query. Pinning retrieval to a single deterministic search is what keeps runs comparable — the config stays the only variable.
 - **What they do not measure**: agent behavior. A tool-based agent can beat these numbers by issuing several targeted searches and reformulating after a weak result, or fall short of them by not searching at all, phrasing a query poorly, or running out of turns. Tool choice, query formulation, and multi-turn dynamics are untested here.
 
 If your production path exposes Zep through tools, treat a strong result here as a prerequisite rather than a verdict, and evaluate the agent end-to-end as well. See [Evaluate Zep for your use case](https://help.getzep.com/evaluate-zep-for-your-use-case).
@@ -210,11 +210,12 @@ config/
 ├── document_chunking_config/
 │   └── constants.py                          # CHUNK_SIZE, CHUNK_OVERLAP, LLM_CONTEXTUALIZATION_MODEL
 └── evaluation_config/
-    ├── constants.py                          # Search limits, LLM_RESPONSE_MODEL, LLM_JUDGE_MODEL
+    ├── constants.py                          # LLM_RESPONSE_MODEL, LLM_JUDGE_MODEL
+    ├── retrieval_strategy.py                 # build_context_block() — retrieval + context assembly
     └── response_prompt.py                    # get_response_system_prompt() — the system prompt for AI responses
 ```
 
-Each script imports only from its relevant config subfolder. The response prompt used during evaluation is defined in `config/evaluation_config/response_prompt.py` and can be customized independently from the evaluation logic.
+Each script imports only from its relevant config subfolder. The response prompt used during evaluation is defined in `config/evaluation_config/response_prompt.py` and can be customized independently from the evaluation logic. Retrieval behavior lives entirely in `retrieval_strategy.py`.
 
 ## Run Tracking
 
@@ -390,15 +391,16 @@ To add more users:
 
 ## Advanced Evaluation
 
-### Tune Zep Search Parameters
+### Tune Zep Retrieval Strategy
 
-The evaluation script uses `cross_encoder` reranker by default for best accuracy. Search parameters and LLM models are configured in `config/evaluation_config/constants.py`:
-- `USER_FACTS_LIMIT = 20`: Number of facts (edges) from user graph
-- `USER_ENTITIES_LIMIT = 10`: Number of entities (nodes) from user graph
-- `USER_EPISODES_LIMIT = 0`: User episodes disabled by default (set >0 to enable)
-- `DOC_FACTS_LIMIT = 10`: Number of facts from document graph
-- `DOC_ENTITIES_LIMIT = 5`: Number of entities from document graph
-- `DOC_EPISODES_LIMIT = 0`: Document episodes disabled by default
+Retrieval is defined by a single module: `config/evaluation_config/retrieval_strategy.py`. Its `build_context_block()` function is the source of truth — edit that function (and the constants it uses) to change how context is retrieved and assembled.
+
+Default strategy:
+- `SCOPE = "auto"`: Zep packs edges, nodes, episodes, observations, and thread summaries into a pre-assembled context string
+- `MAX_CHARACTERS = 10000`: character budget for auto search (``limit`` and ``reranker`` do not apply under auto)
+- User-node summary is fetched via `user.get_node()` and prepended (auto search does not include it)
+
+LLM models remain in `config/evaluation_config/constants.py`:
 - `LLM_RESPONSE_MODEL`: Model for generating responses
 - `LLM_JUDGE_MODEL`: Model for grading answers
 
@@ -407,20 +409,15 @@ Chunking-specific constants are in `config/document_chunking_config/constants.py
 - `CHUNK_OVERLAP`: Characters of overlap between consecutive chunks (default: 100)
 - `LLM_CONTEXTUALIZATION_MODEL`: Model for document chunk contextualization
 
-You can experiment with different rerankers by modifying the `reranker` parameter in `perform_graph_search()`:
-- `cross_encoder`: Best accuracy, slower (default)
-- `rrf`: Reciprocal Rank Fusion, balanced
-- `mmr`: Maximal Marginal Relevance, diversity-focused
-
-For guidance, check out the [Searching the Graph documentation](https://help.getzep.com/searching-the-graph).
+For guidance, check out the [Searching the Graph documentation](https://help.getzep.com/searching-the-graph) (including Auto Search).
 
 ### Customize Response Prompt
 
 The system prompt used when generating AI responses during evaluation is defined in `config/evaluation_config/response_prompt.py`. Edit the `get_response_system_prompt()` function to customize the AI's persona, response style, or instructions for your use case. This is snapshotted into each evaluation run for reproducibility.
 
-### Customize Context Block
+### Customize Context Block / Retrieval
 
-The harness constructs a custom context block from graph search results. You can modify the `construct_context_block()` function in `zep_evaluate.py` to format results differently. See the [Customize Your Context Block documentation](https://help.getzep.com/cookbook/customize-your-context-block) for best practices.
+Change retrieval by editing `build_context_block()` in `config/evaluation_config/retrieval_strategy.py`. For example, you can switch to manual multi-scope searches with per-type limits and a chosen reranker, or adjust the auto-search character budget. See the [Customize Your Context Block documentation](https://help.getzep.com/cookbook/customize-your-context-block) and [Searching the Graph](https://help.getzep.com/searching-the-graph) for best practices.
 
 ### Add JSON/Text Data
 
@@ -485,12 +482,9 @@ Results are saved to `runs/evaluations/{run_number}_{timestamp}/results.json` wi
     "document_run": { "run_number": 1, "run_dir": "runs/documents/1_20260331T222500", "graph_id": "zep_eval_shared_documents_1d4d9a28" }
   },
   "search_configuration": {
-    "user_facts_limit": 20,
-    "user_entities_limit": 10,
-    "user_episodes_limit": 0,
-    "doc_facts_limit": 10,
-    "doc_entities_limit": 5,
-    "doc_episodes_limit": 0
+    "strategy": "auto_search",
+    "scope": "auto",
+    "max_characters": 10000
   },
   "model_configuration": {
     "response_model": "gemini-2.5-flash-lite",

zep-eval-harness/config/evaluation_config/constants.py

@@ -1,13 +1,3 @@
-# Search configuration — user graphs
-USER_FACTS_LIMIT = 20  # Number of facts (edges) to return
-USER_ENTITIES_LIMIT = 10  # Number of entities (nodes) to return
-USER_EPISODES_LIMIT = 0  # Number of episodes to return (when enabled)
-
-# Search configuration — standalone document graph
-DOC_FACTS_LIMIT = 10  # Number of facts (edges) to return
-DOC_ENTITIES_LIMIT = 5  # Number of entities (nodes) to return
-DOC_EPISODES_LIMIT = 0  # Number of episodes to return (when enabled)
-
 # LLM models for evaluation
 LLM_RESPONSE_MODEL = "gemini-3-flash-preview"  # Model used for generating responses
 LLM_JUDGE_MODEL = "gemini-3-flash-preview"  # Model used for grading responses

zep-eval-harness/config/evaluation_config/retrieval_strategy.py

@@ -0,0 +1,123 @@
+"""
+Retrieval strategy for evaluation context blocks.
+
+This module is the source of truth for how the harness retrieves and assembles
+context. Edit ``build_context_block`` (and the constants it closes over) to
+change search behavior — there is no separate per-scope limit/reranker config.
+
+Default: ``scope="auto"`` with a 10k character budget. Auto search packs edges,
+nodes, episodes, observations, and thread summaries into a pre-assembled
+``result.context`` string. ``limit`` and ``reranker`` do not apply under auto.
+The user-node summary is fetched separately and prepended (auto search does
+not include it).
+"""
+
+from __future__ import annotations
+
+import asyncio
+from typing import TYPE_CHECKING
+
+from retry import retry_with_backoff
+
+if TYPE_CHECKING:
+    from zep_cloud.client import AsyncZep
+
+STRATEGY_NAME = "auto_search"
+SCOPE = "auto"
+MAX_CHARACTERS = 10_000
+
+
+def get_search_configuration() -> dict:
+    """Snapshot of the active retrieval strategy for evaluation result files."""
+    return {
+        "strategy": STRATEGY_NAME,
+        "scope": SCOPE,
+        "max_characters": MAX_CHARACTERS,
+    }
+
+
+async def _fetch_user_summary(zep_client: AsyncZep, user_id: str) -> str | None:
+    """Fetch the user-node summary, or None if unavailable."""
+    try:
+        user_node_response = await retry_with_backoff(
+            zep_client.user.get_node,
+            user_id=user_id,
+            description=f"get user node [{user_id}]",
+        )
+        node = getattr(user_node_response, "node", None)
+        summary = getattr(node, "summary", None) if node else None
+        if summary and str(summary).strip():
+            return str(summary).strip()
+    except Exception as e:
+        print(f"  Could not retrieve user summary for [{user_id}]: {e}")
+    return None
+
+
+async def build_context_block(
+    zep_client: AsyncZep,
+    *,
+    user_id: str,
+    query: str,
+    doc_graph_id: str | None = None,
+) -> str:
+    """
+    Retrieve a context block for ``query`` using the configured strategy.
+
+    Fetches the user-node summary, runs auto search on the user graph, and
+    optionally auto-searches a standalone document graph in parallel. Assembles
+    summary + Zep's materialized ``context`` string(s).
+    """
+    print(f"Searching [{user_id}]: '{query}' (scope={SCOPE}, max_characters={MAX_CHARACTERS})")
+
+    summary_task = _fetch_user_summary(zep_client, user_id)
+    user_task = retry_with_backoff(
+        zep_client.graph.search,
+        user_id=user_id,
+        query=query,
+        scope=SCOPE,
+        max_characters=MAX_CHARACTERS,
+        description=f"auto search user [{user_id}]",
+    )
+
+    if doc_graph_id:
+        doc_task = retry_with_backoff(
+            zep_client.graph.search,
+            graph_id=doc_graph_id,
+            query=query,
+            scope=SCOPE,
+            max_characters=MAX_CHARACTERS,
+            description=f"auto search doc [{doc_graph_id}]",
+        )
+        user_summary, user_result, doc_result = await asyncio.gather(
+            summary_task, user_task, doc_task
+        )
+    else:
+        user_summary, user_result = await asyncio.gather(summary_task, user_task)
+        doc_result = None
+
+    parts: list[str] = []
+
+    if user_summary:
+        parts.append(
+            "# High-level summary of the user\n"
+            "<USER_SUMMARY>\n"
+            f"{user_summary}\n"
+            "</USER_SUMMARY>"
+        )
+
+    user_context = getattr(user_result, "context", None) or ""
+    if user_context.strip():
+        parts.append(user_context.strip())
+
+    if doc_result is not None:
+        doc_context = getattr(doc_result, "context", None) or ""
+        if doc_context.strip():
+            parts.append(
+                "The following context is from shared reference documents.\n\n"
+                + doc_context.strip()
+            )
+
+    if not parts:
+        return "No relevant context found."
+
+    return "\n\n".join(parts)