Investigate derived semantic retrieval for bounded context packets

[greygoody]

Problem

CodeCtx currently exposes explicit, inspectable task context such as the task goal, allowed scope, Git/worktree state, runtime metadata, required evidence, and explained net moves.

Exact links, identifiers, and structured state should remain authoritative. However, agents may also need to recover conceptually related prior constraints, decisions, failures, or closure evidence when terminology has changed and no direct link exists.

The question is whether a derived semantic retrieval capability can improve bounded context packets without turning similarity into domain truth or expanding CodeCtx into a general memory platform.

Current hypothesis

A semantic layer may be useful as a disposable projection over explicit CodeCtx records.

Possible shape:

• export selected context records with stable source references and epistemic/status metadata;
• generate embeddings through a provider-neutral interface;
• support hybrid retrieval: hard metadata filters, explicit graph/direct links, exact keyword matching, then semantic similarity;
• return attributed results with source, score, status, and selection reason;
• keep all vectors and indexes rebuildable and non-canonical;
• expose retrieval through a narrow CLI or tool-facing contract only after evaluation proves value.

This is a hypothesis, not accepted architecture.

Questions to answer

1. Which CodeCtx records are legitimate semantic inputs: task intent, constraints, decisions, evidence, unknowns, rejected moves, failures, or closure summaries?
2. Can semantic retrieval materially outperform direct links plus keyword search on realistic CodeCtx tasks?
3. How should results preserve distinctions between accepted state, proposal, unknown, rejected material, and stale context?
4. Should this capability live in CodeCtx core, an optional adapter, or a separate private projection service?
5. What provider and model identity must be recorded so incompatible vectors are never silently compared?
6. What bounded packet contract would let an agent inspect why each result was selected?

Required experiment

Build or simulate a small representative corpus from completed and active tasks, then hand-label realistic queries and compare:

• keyword-only retrieval;
• embedding-only retrieval;
• hybrid retrieval;
• explicit graph/direct-link retrieval plus hybrid fallback.

Measure at minimum:

• relevant-record recall within a small result budget;
• stale or superseded context retrieval rate;
• contradictory/rejected-context coverage;
• provenance completeness;
• relevant tokens added per packet;
• whether retrieved context changes or improves an actual task decision.

Constraints

• Explicit CodeCtx state, Git evidence, tests, and human decisions remain authoritative.
• Similarity must never promote, accept, or mutate state.
• Retrieval must preserve source references and epistemic/status metadata.
• The semantic index must be disposable and reproducible from canonical records.
• No required hosted vector database or embedding provider in CodeCtx core.
• No silent partial context packet when the declared retrieval contract requires complete coverage.

Non-goals

• exposing model hidden states, activations, attention data, reasoning tokens, or KV caches;
• building a universal personal-memory system;
• replacing branch/task links, Git traversal, or exact identifiers;
• introducing a general ontology, relation graph, or event framework;
• selecting pgvector, a hosted vector store, or OpenAI embeddings before the experiment earns that dependency;
• adding semantic retrieval to the public product before a concrete consumer proves it useful.

Acceptance criteria

This issue may close with one of three valid outcomes:

1. Promote a minimal contract: evidence shows a bounded semantic fallback materially improves context retrieval, and a follow-up implementation issue defines the smallest public boundary.
2. Keep it external/private: the mechanism is useful but belongs in Orb/private workflow infrastructure rather than CodeCtx core.
3. Reject or defer: direct links, graph traversal, and keyword search are sufficient, or semantic retrieval adds too much stale/noisy context for its cost.

Any promoted result must include an inspectable evaluation corpus, retrieval witnesses, provider/model identity handling, and explicit kill or demotion conditions.

[greygoody]

@codex Take ownership of the first bounded investigation slice for this issue.

Mission

Inspect the current greygoody/codectx repository and issue #5, then determine whether derived semantic retrieval is justified for CodeCtx and where the smallest legitimate boundary belongs.

This is an evidence-gathering and design-validation mission, not permission to add a speculative vector subsystem.

Required work

1. Anchor the run by reporting the repository, branch/default branch, commit, working-tree state, toolchain, and relevant existing public contracts.
2. Inspect the current task/context/session models, CLI surface, event/proof structures, README.md, AGENTS.md if present, and tests.
3. Identify the exact observed retrieval problem that current direct links, Git traversal, structured fields, and keyword search cannot solve. Preserve it as unknown if repository evidence does not establish one.
4. Compare three boundaries:
   • no CodeCtx change; keep semantic retrieval in an external/private context service;
   • provider-neutral export/projection contract only;
   • optional bounded semantic retrieval adapter inside CodeCtx.
5. Identify a concrete consumer workflow for each viable boundary. Do not treat hypothetical future agents as sufficient consumer evidence.
6. Design the smallest reversible evaluation capable of changing the decision. Compare direct/keyword retrieval against semantic and hybrid retrieval using a representative corpus or fixtures that do not expose private user data.
7. Define witnesses, failure-detection checks, stale/superseded-context handling, provider/model identity rules, provenance requirements, and kill/demotion criteria.
8. Conclude with one explicit verdict:
   • promote a minimal follow-up implementation contract;
   • keep the mechanism external/private;
   • reject/defer due to insufficient observed need.

Mutation authority

You may add a narrowly scoped, public-safe investigation artifact or evaluation fixture/harness only when it is necessary to produce inspectable evidence. Do not implement production semantic retrieval, add a vector database, select OpenAI embeddings as a required dependency, or alter canonical CodeCtx state semantics.

If you make repository changes:

• keep them in one reversible branch;
• run the repository's declared proof commands;
• create a PR linked to issue Investigate derived semantic retrieval for bounded context packets #5;
• clearly distinguish observed evidence, deductions, proposals, and unknowns in the PR description.

If no repository change is justified, report the complete investigation and verdict on this issue instead of manufacturing a PR.

Stop conditions

Stop and report rather than expanding scope if:

• no real consumer workflow is observable;
• representative evaluation data would require exposing private conversation/project content;
• a second source of truth would be introduced;
• semantic similarity would influence authority or mutation;
• the useful mechanism clearly belongs outside CodeCtx;
• the next meaningful evidence requires user-provided private workflow data.

The issue body is the governing problem statement and acceptance boundary.

[chatgpt-codex-connector]

You have reached your Codex usage limits. You can see your limits in the Codex usage dashboard.

[greygoody]

Codectx HUD direction marker

status: compatible-existing-investigation
vocabulary_generation: repository-context-v1
target_position: optional-derived-lens
contract_changes: deferred
canonical_direction_pr: "#9"

This investigation remains valid. Semantic retrieval may become an optional derived lens, but it must not become Codectx memory authority, compositor truth, or an implicit replacement for exact links and structured state. PR #9 does not authorize implementation, indexing, RAG, persistence, or changes to ContextPacket v1.