Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ Any code change must either adhere to our spec files perfectly or you should ask
| `src/jarvis/reply/prompts/prompts.spec.md` | System/user prompt templates | — |
| `src/jarvis/tools/builtin/web_search.spec.md` | webSearch tool: cascade fetch, SSRF guard, prompt-injection fence, links-only envelope | Untrusted web content is fenced as data, not instructions; rank preference over speed; honest failure over confabulation |
| `src/jarvis/tools/builtin/nutrition/log_meal.spec.md` | logMeal tool: single-property schema for planner fast-path, internal nutrition extraction, untrusted-data fence, follow-ups | Public schema is a single optional `meal` string; nutrition fields are internal; user text is fenced as data |
| `src/jarvis/tools/builtin/project_intake.spec.md` | Multi-turn project intake with deterministic gate | Gate runs pre-planner; one question per turn by construction; template match is keyword-based, never LLM |
| `src/jarvis/utils/location.spec.md` | GeoIP location detection | Privacy-first; local GeoLite2 DB only |
| `src/jarvis/memory/graph.spec.md` | Node graph memory (v2), self-organising tree, UI explorer | Dynamic structure; access-aware; auto-split/merge (future) |
| `src/jarvis/memory/summariser.spec.md` | Diary summariser prompt contract, hygiene rules (deflection, attribution, topic separation), post-process scrub, and bulk-sweep clean button | Two-layer defence: prompt + deterministic scrub; corrupted summaries poison every downstream consumer |
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -165,7 +165,7 @@ Jarvis starts listening automatically — just say "Jarvis" and talk!
- **Unlimited Memory** - Never forgets. Searches across all your conversation history. Memory Viewer GUI included.
- **Adaptive Tone** - Automatically surgical for code, pragmatic for business, encouraging for wellbeing — no manual mode switching
- **Smart Tool Selection** - Embedding-based relevance filtering picks only the tools needed per query — add unlimited MCP tools without performance degradation
- **Built-in Tools** - Screenshot OCR, web search (DuckDuckGo → Brave → Wikipedia fallback chain with auto-fetch), weather, file access, nutrition tracking, location awareness, plus a tool-discovery escape hatch the agent uses to widen its own toolset mid-reply
- **Built-in Tools** - Screenshot OCR, web search (DuckDuckGo → Brave → Wikipedia fallback chain with auto-fetch), weather, file access, nutrition tracking, location awareness, guided project intake (multi-turn interview that builds a project brief and saves it to Obsidian), plus a tool-discovery escape hatch the agent uses to widen its own toolset mid-reply
- **Knowledge Graph Memory** - Self-organising memory that learns from conversations, auto-splits by topic, and surfaces relevant knowledge automatically
- **Natural Voice** - Say "Jarvis" anywhere in your sentence, interrupt with "stop", follow up without repeating the wake word
- **Dictation Mode** - Free, offline alternative to WisprFlow — hold a hotkey, speak, release to paste text into any app
Expand Down
14 changes: 13 additions & 1 deletion docs/llm_contexts.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,16 @@ Every distinct LLM call in Jarvis, what feeds it, what consumes it, and how it i
- **Planner precedence**: when the planner explicitly emitted a `searchMemory` step, the gate is bypassed — the planner has more signal than coverage and overriding it would silently drop intent. The gate only short-circuits the fail-open empty-plan path.
- **Rationale**: prevents re-running diary/graph lookups when the hot window already grounds the follow-up (e.g. "his most famous song" after a Bieber webSearch).

## 3c. Project Intake Gate (pre-planner force-route)

- **File**: [src/jarvis/tools/builtin/project_intake.py](src/jarvis/tools/builtin/project_intake.py) — `get_gated_session()`; wired into [src/jarvis/reply/engine.py](src/jarvis/reply/engine.py) right after redaction, before recent-dialogue lookup, MCP refresh, the tool router (#7), and the planner (#12).
- **Trigger**: once per reply, before anything else in `run_reply_engine`. Gated on `cfg.project_intake_enabled` (default `True`).
- **Model / gating**: NO LLM — deterministic SQLite lookup (`Database.get_active_intake_session()`).
- **Inputs**: none beyond DB state — no query text is inspected.
- **Output**: if a non-completed `project_intake_sessions` row exists, forces `run_tool_with_retries(tool_name="projectIntake", tool_args={"input": redacted})` directly, prints/speaks/records the tool's `reply_text` verbatim, and returns immediately — the router, planner, memory enrichment, and the entire agentic LLM loop are all skipped for this turn. Fail-open on any DB error or malformed row shape (`get_gated_session` catches and treats it as "no session", falling through to normal routing).
- **Rationale**: multi-turn interview state must survive turns deterministically; a fresh planner/router call every turn has no guarantee of recognising "mid-interview" the way a small local model might miss it. See `project_intake.spec.md` "Why a gate, not just a tool".
- **Data-flow edge**: this is also the first builtin tool invocation in the app that calls out to an MCP server (Obsidian "Jarvis Brain") directly from tool code rather than via the LLM's own tool-call loop — see `ProjectIntakeTool.run()` → `write_brief_to_obsidian()` on interview completion, and the separate `StartProjectDevelopmentTool` for the later Antigravity hand-off trigger.

## 4. Memory Digest (optional, SMALL models)

- **File**: [src/jarvis/reply/enrichment.py](src/jarvis/reply/enrichment.py) — `digest_memory_for_query()` + `_distil_batch()`.
Expand Down Expand Up @@ -222,12 +232,14 @@ Driven by `detect_model_size(model_name) → SMALL (≤7B) | LARGE (8B+)`:
- Flags: `memory_digest_enabled`, `tool_result_digest_enabled`, `llm_thinking_enabled`, `intent_judge_thinking_enabled`, `tool_selection_strategy`
- Timeouts: `llm_chat_timeout_sec` (45s), `llm_digest_timeout_sec` (8s, shared across #4/#5/#6), `llm_tools_timeout_sec`, `intent_judge_timeout_sec` (6s), `planner_timeout_sec` (3s)
- Caps: `agentic_max_turns` (8), `tool_search_max_calls` (3), `_LLM_MAX_SELECTED` (5), `_DIGEST_MAX_CHARS` (400), `_TOOL_DIGEST_MAX_CHARS` (600)
- Non-LLM gates: `project_intake_enabled` (default `True`, see #3c), `project_templates_path`

## Flow

```
user input
└─▶ [2] Intent Judge (voice only, SMALL)
└─▶ [3c] Project Intake Gate (no LLM — active session? force projectIntake tool, return, done)
└─▶ [2] Intent Judge (voice only, SMALL)
└─▶ [7] Tool router (narrows catalogue for the planner)
└─▶ [12] Planner (gates memory; advisory for the router allow-list)
├─ plan requests searchMemory → [3] Enrichment extract → [4] Memory digest (optional)
Expand Down
2 changes: 2 additions & 0 deletions evals/helpers.py
Original file line number Diff line number Diff line change
Expand Up @@ -281,6 +281,8 @@ class MockConfig:
dialogue_memory_timeout: int = 300
mcps: Dict[str, Any] = field(default_factory=dict)
use_stdin: bool = True
project_intake_enabled: bool = True
project_templates_path: str = ""


@dataclass
Expand Down
97 changes: 97 additions & 0 deletions evals/test_project_intake_flow.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,97 @@
"""End-to-end eval — the project intake trigger phrase must be routed to
the projectIntake tool by normal (LLM) tool selection, since starting a
new intake is the one point in the flow where tool selection depends on
the router/planner rather than the deterministic gate (see
project_intake.spec.md "Trigger detection").

This complements the deterministic unit/integration tests in
tests/tools/builtin/test_project_intake.py, which cover the gate,
template matching, full flow, abandon phrase, and Obsidian write without
needing a live LLM. This eval is the one case that genuinely needs the
router+planner to reliably pick the tool from natural language.

Run: EVAL_JUDGE_MODEL=gemma4:e2b ./scripts/run_evals.sh project_intake_flow
"""

import os

import pytest

from conftest import requires_judge_llm
from helpers import assert_not_fallback_reply, JUDGE_MODEL

_TEMPLATES_PATH = os.path.join(
os.path.dirname(os.path.dirname(os.path.abspath(__file__))),
"project_templates.json",
)


@pytest.mark.eval
@requires_judge_llm
class TestProjectIntakeFlow:
"""Router must select projectIntake when the user says the trigger
phrase with no active intake session, and the reply must be exactly
the type question — not a paraphrase, not extra chatter."""

def test_trigger_phrase_starts_intake_and_asks_type(
self, mock_config, eval_db, eval_dialogue_memory,
):
from jarvis.reply.engine import run_reply_engine

mock_config.ollama_base_url = "http://localhost:11434"
mock_config.ollama_chat_model = JUDGE_MODEL
mock_config.project_templates_path = _TEMPLATES_PATH

response = run_reply_engine(
db=eval_db, cfg=mock_config, tts=None,
text="vamos começar um novo projeto",
dialogue_memory=eval_dialogue_memory,
)

print(f"\n Project Intake Trigger ({JUDGE_MODEL}):")
print(f" Response: {(response or '')[:300]}")

assert_not_fallback_reply(response, context="project-intake-trigger")

session = eval_db.get_active_intake_session()
assert session is not None, (
"No project_intake_sessions row was created — the router did "
f"not select projectIntake for the trigger phrase. Response: "
f"{(response or '')[:400]}"
)
assert session["status"] == "awaiting_type"

response_lower = (response or "").lower()
assert "tipo de projeto" in response_lower, (
"Reply does not ask which type of project this is, as required "
f"by the intake flow. Response: {(response or '')[:400]}"
)

def test_second_turn_is_gated_deterministically_not_via_llm(
self, mock_config, eval_db, eval_dialogue_memory,
):
"""Once a session is active, the follow-up turn must be forced to
projectIntake by the gate — this should hold even with a judge
model in the loop, proving the gate really does short-circuit the
router/planner rather than relying on the model picking the tool
again on its own."""
from jarvis.reply.engine import run_reply_engine

mock_config.ollama_base_url = "http://localhost:11434"
mock_config.ollama_chat_model = JUDGE_MODEL
mock_config.project_templates_path = _TEMPLATES_PATH

eval_db.insert_intake_session()

response = run_reply_engine(
db=eval_db, cfg=mock_config, tts=None,
text="branding",
dialogue_memory=eval_dialogue_memory,
)

session = eval_db.get_active_intake_session()
assert session is not None
assert session["status"] == "in_progress"
assert session["project_type"] == "branding"
# Exactly one question relayed verbatim, no LLM paraphrase.
assert response == response.strip()
69 changes: 69 additions & 0 deletions project_templates.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
{
"website": {
"label": "Site / Loja Online",
"keywords": ["site", "website", "web site", "loja online", "landing page", "ecommerce", "e-commerce", "pagina web", "página web"],
"questions": [
"Qual é o objetivo principal do site? (vender, apresentar a marca, captar contactos/leads, agendar marcações...)",
"Quem é o público-alvo? (idade, zona geográfica, tipo de cliente)",
"Que produtos ou serviços tens de destacar, e quantos aproximadamente?",
"Precisas de loja online com pagamentos, ou só um catálogo/apresentação sem compra direta?",
"Já tens marca, logo e cores definidas, ou também precisas disso incluído?",
"Tens algum site (teu ou de concorrentes) que gostes como referência, ou algo que queiras claramente evitar?",
"Precisas de o site estar em mais do que uma língua?",
"Qual é o prazo desejado?",
"Qual é o orçamento aproximado?"
]
},
"app": {
"label": "Aplicação Móvel ou Web",
"keywords": ["app", "aplicacao", "aplicação", "aplicativo", "mobile", "ios", "android"],
"questions": [
"Qual é o problema principal que a aplicação resolve para quem a usa?",
"Quem são os utilizadores-alvo?",
"Deve funcionar em telemóvel, computador, ou ambos?",
"Precisa de contas de utilizador e login?",
"Vai envolver pagamentos ou subscrições dentro da app?",
"Há alguma app parecida que conheças e queiras usar como referência (ou evitar)?",
"Qual é o prazo desejado?",
"Qual é o orçamento aproximado?"
]
},
"marketing": {
"label": "Campanha de Marketing",
"keywords": ["campanha", "marketing", "anuncio", "anúncio", "publicidade", "ads", "promocao", "promoção"],
"questions": [
"Qual é o objetivo da campanha? (vendas, visibilidade de marca, lançamento de um produto...)",
"Quem é o público-alvo?",
"Que canais queres usar? (redes sociais, Google Ads, email, físico...)",
"Há uma data ou evento associado a esta campanha?",
"Já tens materiais criativos (fotos, vídeos, textos) ou é preciso criar de raiz?",
"Qual é o orçamento disponível para investimento em anúncios?",
"Como vais medir se a campanha correu bem?"
]
},
"branding": {
"label": "Marca / Identidade Visual",
"keywords": ["marca", "branding", "logo", "identidade visual", "naming"],
"questions": [
"O negócio já tem nome definido, ou também precisas de ajuda a escolher um?",
"Como descreverias a personalidade da marca em 3 palavras?",
"Quem é o público-alvo desta marca?",
"Há marcas (de qualquer setor) cujo estilo visual admiras?",
"Há cores ou estilos que queres claramente evitar?",
"Onde vai ser usada a identidade visual? (site, redes sociais, loja física, embalagens...)",
"Qual é o prazo desejado?"
]
},
"other": {
"label": "Outro / Genérico",
"keywords": [],
"questions": [
"Descreve em uma frase o que queres construir ou alcançar com este projeto.",
"Quem é o público-alvo ou principal beneficiário?",
"Qual é o resultado que consideras sucesso?",
"Há restrições importantes a considerar? (técnicas, legais, de recursos)",
"Qual é o prazo desejado?",
"Qual é o orçamento aproximado?"
]
}
}
Loading