Skip to content

Onboarding + discoverability: interactive lens picker, full toolbox, exhaustive mining #19

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
0xadvait merged 1 commit into from
Jun 6, 2026
Merged

Onboarding + discoverability: interactive lens picker, full toolbox, exhaustive mining #19

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
49 changes: 45 additions & 4 deletions AGENTS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,21 @@

This guide is for an LLM agent that has landed in this repo and needs to help a
user set up or use doxa interactively. Be command-first and keep the user in the
loop for provider, model, and source choices.
loop for provider, model, and source choices. Surface capabilities proactively --
users often don't know what's available.

## Capabilities at a glance

| Goal | Command |
| --- | --- |
| State | `doxa status` · smoke test `doxa demo` · diagnose `doxa doctor` |
| Start a base | `doxa init --lens-template <name>` -- browse `doxa lenses list` |
| Lenses | `doxa lenses list` / `show <name>` / `add <name> --from <name>` |
| Ingest | `doxa ingest <file\|url\|pdf\|youtube\|->` |
| Hard pages | `doxa ingest <url> --via jina\|firecrawl\|brightdata\|claude\|codex\|hermes` `[--mode browser\|extract --prompt ...] [--yolo]` |
| Answer | `doxa query "<q>" --answer` / `--json` / `--domain <slug>` / `--search hybrid` |
| Honest | `doxa eval` · `doxa sources list\|remove <id>` · `doxa domains set <slug> <0-10>` |
| Agent skill | `doxa skill install --harness claude-code\|codex\|hermes\|openclaw` |

## Detect

Expand Down Expand Up @@ -87,7 +101,25 @@ doxa init ./doxa.yaml --yes --provider codex-cli
doxa init ./fireworks.yaml --yes --provider openai-compatible --model accounts/fireworks/models/<slug>
```

Help the user write a lens with three fields:
### Pick a lens (do this first -- don't make the user invent one)

A lens is the question doxa asks of every source; it shapes every belief. doxa
ships an opinionated library -- show it and choose together rather than starting
from a blank lens:

```bash
doxa lenses list # founder-strategy, investment-memo, research-literature, ...
doxa lenses show investment-memo # inspect one first
doxa init --lens-template founder-strategy # seed a base from it
doxa lenses add my-lens --from founder-strategy # fork + tailor it to the user
```

`doxa init` with no `--lens-template` prompts the user to pick from the same
library interactively. Built-ins: `durable-beliefs`, `founder-strategy`,
`investment-memo`, `technical-design`, `research-literature`, `policy-analysis`,
`personal-principles`, `customer-discovery`.

If none fit, write a custom lens with three fields:

- `name`: short identifier.
- `description`: what kind of beliefs to extract.
Expand All @@ -107,10 +139,11 @@ For pre-fetched text or copied notes, pipe stdin and attach metadata:
printf '%s' "$SOURCE_TEXT" | doxa ingest - --title "Title" --author "Author" --url "https://source.example"
```

Show the verbatim guarantee:
Show the verbatim guarantee, and diagnose setup if anything looks off:

```bash
doxa eval
doxa eval # every quote still verbatim, every belief still linked
doxa doctor # config, storage, provider, and semantic-index readiness
```

Query it (keyword is the zero-setup default):
Expand Down Expand Up @@ -142,6 +175,14 @@ doxa ingest <url> --via hermes --mode browser
doxa ingest <url> --via hermes --mode extract --prompt "title, author, date as JSON"
```

Agent fetchers run safely by default. For unattended runs on sources the user
trusts, `--yolo` enables codex `--dangerously-bypass-approvals-and-sandbox`,
hermes `--yolo`, and the `command` shell for that ingest:

```bash
doxa ingest <url> --via codex --yolo
```

```bash
doxa ingest https://target.example --via jina # free, good first try for bot-walled pages
doxa ingest https://target.example --via brightdata # needs BrightData env vars
Expand Down
131 changes: 72 additions & 59 deletions doxa/_assets/skill/SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,103 +1,116 @@
---
name: doxa
description: Use when answering from or growing a custom knowledge base where every belief must be anchored to exact source quotes.
description: Use when answering from or building a custom knowledge base where every claim must be anchored to an exact source quote. Covers grounded retrieval, choosing a lens, ingesting sources (incl. hard/bot-walled pages), and keeping the base honest.
---

# doxa Skill

Use the installed `doxa` CLI as the source of truth for grounded beliefs in a
custom knowledge base. No quote, no claim.
doxa is a verbatim-grounded belief base: it mines sources into linked
(belief, exact-quote) records and answers only from them. The rule, always:
**No quote, no claim.** Use the installed `doxa` CLI as the source of truth, and
proactively offer the capabilities below -- users often don't know they exist.

## Check state first
## Capabilities at a glance

```bash
doxa status
```

`doxa status` reports the active config, data location, and belief/quote counts.
If it shows `beliefs: 0`, the base is empty -- there is nothing to ground on, so
offer to `doxa ingest` a source instead of answering. Use `doxa sources list` to
see what has been ingested and `doxa sources remove <id>` to undo a mistake.
| Goal | Command |
| --- | --- |
| See where things stand | `doxa status` (config, counts, provider, semantic on/off) |
| Answer, grounded | `doxa query "<q>" --answer` (or `--json` for tools) |
| Start a base | `doxa init --lens-template <name>` -- browse with `doxa lenses list` |
| Ingest a source | `doxa ingest <file\|url\|pdf\|youtube\|->` |
| Fetch a hard page | `doxa ingest <url> --via jina\|firecrawl\|brightdata\|claude\|codex\|hermes` |
| Verify / diagnose | `doxa eval` (quotes still verbatim) · `doxa doctor` (setup) |
| Manage sources | `doxa sources list` · `doxa sources remove <id>` |
| Semantic search | `doxa index`, then `doxa query "<q>" --search hybrid` |

## Answer from the belief base
Run any command with `-h` for full options.

Run:
## 1. Check state first

```bash
doxa query "<user question>" --search keyword
doxa status
```

For a final user-facing response, prefer the local answer renderer:

```bash
doxa query "<user question>" --search keyword --answer
```
Reports the active config, data location, and belief/quote counts. If it shows
`beliefs: 0`, the base is empty -- there is nothing to ground on, so offer to
**start a base** (below) rather than answering.

Use domain focus only when it is clearly relevant to the user's question:
## 2. Answer from the belief base

```bash
doxa query "<user question>" --search keyword --domain technical
doxa query "<user question>" --search keyword --domains policy,finance
doxa query "<user question>" --answer # readable, grounded brief
doxa query "<user question>" --json # structured records for tooling
doxa query "<user question>" --domain technical --top 10
```

Keyword search is the zero-setup default. If the project has configured
semantic search, `--search hybrid` may be used; if hybrid reports that the
semantic leg is unavailable and falls back to keyword, continue with the
returned keyword-grounded results.
Keyword search is the zero-setup default and covers both belief and quote text.
If the project configured semantic search, `--search hybrid` fuses semantic +
keyword; if it reports the semantic leg is unavailable and falls back to keyword,
continue with the returned keyword results.

Read the returned beliefs and verbatim quotes as the only ground truth. Answer
from those returned records. Cite or include the exact quotes when they matter.
Read the returned beliefs and verbatim quotes as the **only** ground truth.
Humanize only the surrounding prose. Never alter the bytes, punctuation,
capitalization, or whitespace inside a returned quote span, and never invent a
quote, attribution, source, or belief that was not returned by the CLI.
capitalization, or whitespace inside a returned quote, and never invent a quote,
attribution, source, or belief the CLI did not return.

## Grow the belief base
## 3. Start or grow a base

When the user wants to add a trusted source, run:
### Pick a lens first (don't make the user invent one)

A lens is the question doxa asks of every source -- it shapes every belief
extracted. doxa ships an opinionated library; show it and pick together:

```bash
doxa ingest <source>
doxa lenses list # founder-strategy, investment-memo, research-literature, ...
doxa lenses show investment-memo # see one before using it
doxa init --lens-template founder-strategy # seed a base from it (or `doxa init` to pick interactively)
doxa lenses add my-lens --from founder-strategy # fork + customize for this user
```

`<source>` may be a text file, PDF, article URL, or YouTube URL depending on the
installed doxa extras. Ingestion mines beliefs through the configured provider
and drops any quote that cannot be verified as a real substring of the source.

If the source text has already been fetched by the harness, pipe it through
stdin and attach metadata:
### Ingest sources

```bash
printf '%s' "$SOURCE_TEXT" | doxa ingest - --title "Title" --url "https://source.example"
doxa ingest <file|url|pdf|youtube|->
```

For bot-protected sources, prefer a harness BrightData MCP fetch followed by
`doxa ingest -`. Direct CLI use can route through BrightData with
`doxa ingest <url> --via brightdata` when `BRIGHTDATA_API_TOKEN` and
`BRIGHTDATA_ZONE` are set.
Quotes that aren't a verbatim substring of the source are dropped automatically.
For a hard or bot-walled page, choose a fetcher with `--via` (jina is a good free
first try), or let a browsing agent fetch it:

## Domain preferences
```bash
doxa ingest <url> --via jina # free clean markdown
doxa ingest <url> --via hermes --mode browser # agent renders JS, then mines
doxa ingest <url> --via codex --mode extract --prompt "name, price, date as JSON"
```

To inspect domain weights:
If you (the harness) already fetched the text -- e.g. via a BrightData MCP --
pipe it in, no fetcher needed:

```bash
doxa domains
printf '%s' "$SOURCE_TEXT" | doxa ingest - --title "Title" --url "https://source.example"
```

To adjust them when the user asks:
`--yolo` runs agent fetchers unattended (codex bypass-approvals, hermes `--yolo`,
command shell) -- use only on sources the user trusts.

## 4. Keep the base honest

After ingesting, verify and report what happened:

```bash
doxa domains set technical 8
doxa domains add finance 6
doxa domains remove creative
doxa eval # every quote still verbatim, every belief still linked
doxa doctor # config, storage, provider, and semantic-index readiness
doxa sources list # what's ingested, with belief/quote counts
doxa sources remove <id> # undo a bad source
doxa domains set technical 8 # bias retrieval toward a topic when the user asks
```

Domains are stored as `domain:<slug>` tags on beliefs and quotes. Retrieval also
uses `preferences.domain_aliases` so legacy plain tags such as `founders`,
`token-economics`, `trust`, or `taste` can boost and help discover the requested
domain. Domain boosts do not replace quote grounding.
Domains are stored as `domain:<slug>` tags; retrieval also matches legacy plain
tags via `preferences.domain_aliases`. Domain boosts never replace quote grounding.

## Grounding rule

A doxa quote is verbatim source text. A doxa belief is usable only when linked
to at least one returned quote. If the CLI does not return enough evidence,
state that the belief base does not contain enough grounded evidence.
A doxa quote is verbatim source text. A doxa belief is usable only when linked to
at least one returned quote. If the CLI does not return enough evidence, say the
belief base does not contain enough grounded evidence, and offer to ingest more
trusted sources -- never fill the gap from your own memory.
40 changes: 37 additions & 3 deletions doxa/cli.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -188,6 +188,39 @@ def _prompt_provider(default: str) -> str:
return _normalize_provider(raw)


def _lens_choice(raw: str, names: list[str]) -> str | None:
"""Map a picker input (number, name, or blank/'custom') to a template name, or None for custom."""
raw = raw.strip()
if not raw or raw.lower() in {"custom", "c"}:
return None
if raw.isdigit():
index = int(raw) - 1
return names[index] if 0 <= index < len(names) else None
return raw if raw in names else None


def _prompt_lens() -> dict[str, Any]:
"""Interactive lens picker. Returns a chosen template dict, or {} for a custom lens."""
catalog = lens_catalog()
if not catalog:
return {}
names = [row["name"] for row in catalog]
width = max(len(name) for name in names)
print("\nPick a lens -- the question doxa asks of every source (this shapes every belief):")
for index, row in enumerate(catalog, start=1):
suffix = " (yours)" if row["origin"] == "user" else ""
print(f" {index}. {row['name'].ljust(width)} {row['summary']}{suffix}")
print(f" {len(catalog) + 1}. {'custom'.ljust(width)} define your own from scratch")
default_name = "durable-beliefs" if "durable-beliefs" in names else names[0]
name = _lens_choice(_prompt_text("Lens number or name", default_name), names)
if name is None:
return {}
try:
return get_lens_template(name)
except DoxaError:
return {}


def _init_answers(args: argparse.Namespace, *, interactive: bool) -> dict[str, str]:
provider_default = _normalize_provider(args.provider)
provider = _prompt_provider(provider_default) if interactive else provider_default
Expand All @@ -211,17 +244,18 @@ def _init_answers(args: argparse.Namespace, *, interactive: bool) -> dict[str, s
base_url = _prompt_text("OpenAI-compatible base_url", base_url)

template = get_lens_template(args.lens_template) if getattr(args, "lens_template", None) else {}
if interactive and not template:
template = _prompt_lens() # let the user pick from the library instead of inventing one
lens_name_default = args.lens_name or template.get("name") or DEFAULT_LENS["name"]
lens_description_default = args.lens or template.get("description") or DEFAULT_LENS["description"]
lens_question_default = args.lens_question or template.get("question") or DEFAULT_LENS["question"]
lens_stances = template.get("stances") or ["supports", "questions", "rejects", "complicates"]
lens_tags = template.get("tags") or []
if interactive:
if template:
print(f"Starting from the '{args.lens_template}' lens template (edit any field below).")
print(f"\nUsing the '{template.get('name')}' lens -- press Enter to keep each field, or edit it:")
else:
print("Now define the lens: what kind of beliefs should doxa mine?")
print(" (tip: start from a template -- see `doxa lenses list`.)")
print("\nDefine a custom lens: what kind of beliefs should doxa mine?")
lens_name = _prompt_text("Lens name", lens_name_default)
lens_description = _prompt_text("Lens description", lens_description_default)
lens_question = _prompt_text("Guiding question", lens_question_default)
Expand Down
35 changes: 35 additions & 0 deletions doxa/guide.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,13 @@ def d(text: str) -> str:
h("Driving doxa from an AI agent? (Claude Code, Codex, Hermes ...)"),
" doxa skill install --harness claude-code " + d("then just ask the agent"),
"",
h("There's more than the basics -- a quick tour:"),
" doxa lenses list " + d("opinionated lenses: founder, investment, research ..."),
" doxa ingest <url> --via jina " + d("fetch hard pages (jina/firecrawl/brightdata/agent)"),
" doxa query --answer | --json | --search hybrid " + d("brief | machine | semantic"),
" doxa eval | doxa doctor " + d("verify every quote | check your setup"),
" " + d("full tour: doxa guide"),
"",
h("More"),
" doxa guide " + d("guided walkthrough") + " doxa status " + d("where things stand"),
" doxa --help " + d("every command") + " doxa <cmd> -h " + d("help for one command"),
Expand Down Expand Up @@ -116,6 +123,34 @@ def d(text: str) -> str:
" doxa.yaml -- readable, diffable, yours. Run " + t("doxa status") + " to see counts.",
"",
"",
h("THE FULL TOOLBOX"),
"",
" " + t("Lenses") + " -- you don't have to invent one; start from the library:",
" doxa lenses list " + d("8 opinionated lenses + your own"),
" doxa lenses show investment-memo " + d("see one before you use it"),
" doxa init --lens-template founder-strategy",
" doxa lenses add my-lens --from founder-strategy " + d("fork + make it yours"),
"",
" " + t("Ingest anything") + " -- the URL fetcher is pluggable (--via):",
" doxa ingest <url> --via jina " + d("free clean markdown (firecrawl/brightdata too)"),
" doxa ingest <url> --via hermes --mode browser " + d("agent renders JS, then mines"),
" doxa ingest <url> --via codex --mode extract --prompt \"...\"",
" doxa ingest <url> --via codex --yolo " + d("unattended agent fetch (trusted sources)"),
" doxa ingest f.pdf | \"https://youtu.be/...\" | - " + d("PDFs, video transcripts, stdin"),
"",
" " + t("Ask, your way:"),
" doxa query \"...\" --answer " + d("readable evidence brief"),
" doxa query \"...\" --json " + d("structured, for tools/agents"),
" doxa query \"...\" --domain founders --top 10",
" doxa query \"...\" --search hybrid " + d("semantic + keyword (after doxa index)"),
"",
" " + t("Keep it honest:"),
" doxa eval " + d("every quote still verbatim, every belief still linked"),
" doxa doctor " + d("config, storage, provider, semantic readiness"),
" doxa sources list | remove <id> " + d("see / undo what you ingested"),
" doxa domains set founders 8 " + d("bias retrieval toward a topic"),
"",
"",
h("FOR AI AGENTS"),
"",
" doxa is built to be driven by an agent. Install the skill and your agent",
Expand Down
8 changes: 7 additions & 1 deletion doxa/lens.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -76,9 +76,15 @@ def build_extraction_prompt(config: dict[str, Any], source_meta: dict[str, str],
{source_block}

Rules:
- Extract EVERY distinct belief this lens surfaces in the source text. There is no
target number -- never cap the list to a round count and never pad it. Scale to the
source: a rich passage yields many beliefs, a thin one few. Mine the whole chunk and
omit nothing that fits the lens; do not stop early to keep the output short.
- Likewise mine every verbatim quote that grounds a belief -- as many as the text supports.
- Every quote.quote must be copied verbatim from Source text.
- Each belief must be linked by at least one quote.belief_ids entry.
- Prefer concise beliefs that state a stance, not a summary paragraph.
- Prefer concise beliefs that each state one stance, not a summary paragraph -- but split
a passage into multiple beliefs rather than dropping any; one belief per distinct claim.
- Use domain:<slug> tags only when the source text clearly supports that domain classification.
- conviction is a number from 0 to 1 based only on how directly the quote supports the belief.
- Return strict JSON with top-level keys "beliefs" and "quotes".
Expand Down
Loading
Loading