quickbeam

This repo contains infrastructure for building and serving vector search over on-chain data sources registered with Fangorn. The core script pulls manifests from The Graph, resolves payloads from IPFS, walks the typed graph they describe, and then builds embeddings via fastembed/ONNX.

Two meanings of "bundle". This doc uses the word in two unrelated ways:

Schema bundle (--bundle) — a registered subgraph schema whose v3 manifests carry typed node chunks plus an edge chunk. The builder walks those edges to join records.

Snapshot bundle (--bundle-cid, /bundle/*) — an exported NDJSON copy of the populated Qdrant collection, used to seed new instances without a GPU.

How it works

quickbeam build: offline (local, trusted) embeddings builder. Pulls from subgraph, resolves IPFS, joins schemas, embeds, writes to Qdrant.
quickbeam watch: live daemon that polls for new dataset tips and embeds them automatically as they arrive. Keeps the GPU model loaded between cycles. A dataset tip is now a git-native commit (a small IPFS object wrapping the manifest, with a parent link and an embedding contract); the watcher resolves the commit, inherits model/dim/distance from it (sizes the collection to the data, not a CLI default), and diffs the new tip against the last-built commit so it embeds only what changed and tombstones entities the commit removed. Legacy raw-manifest tips still work unchanged. See docs/NEW_QUICKSTART.md.
quickbeam serve: read-only API server. Connects to Qdrant and serves search, browse, and catalog endpoints. It does not ingest on startup, but instead expects the collection to already be populated, either by the builder or by seeding from a snapshot. Can optionally run the watcher alongside it (serve --watch) so one process both ingests and serves, and can gate the search routes behind x402 payments.
quickbeam mcp: a Model Context Protocol layer for agents. A self-contained local pull-client of the Semantic CDN — it pulls a dataset's shards and searches them locally (the query never leaves the process), exposing semantic search and typed-edge graph traversal over raw records, with on-chain provenance on every result, and can optionally charge the calling agent per tool call via x402.
quickbeam cdn + quickbeam pull: the Semantic CDN — instead of running queries on the server (where the node sees every query = intent), the operator bakes the embedded graph into immutable, content-addressed shard files (a "domain") and serves them as static, resumable downloads. A user pulls a domain into their own local Qdrant and queries it offline. Knowledge moves to the user; the network never sees a query. See docs/SEMANTIC_CDN.md.

The builder produces the record shape { track_id, fields, meta } by walking a typed graph, through one of two data sources:

Schema bundle (--bundle) — a single bundle schema publishes manifests carrying typed node chunks ({id, type, fields}) and edge chunks ({rel, from, to}). The builder walks one publisher's graph.
Composed view (--view) — fuses several publishers' bundles into one graph, joining on global identity (Entity URI + aliases + sameAs linksets) before projecting.

Both are projected the same way: one or more root profiles (--root-profile, see ROOT_PROFILES) each walk the graph from a chosen root type and emit a distinct document. Everything downstream (role inference, embedding text, Qdrant payload) is identical.

Ingest engine layout

The offline ingestion engine (shared by build and watch) lives in the quickbeam.ingest package:

quickbeam/ingest/
  build.py             the `quickbeam build` CLI (parse_args + main)
  identity.py          deterministic point ids + the matryoshka vector transform
  checkpoint.py        resumable-build state (ingest checkpoint + role map)
  embed.py             fastembed engine (GPU-OOM resilient), doc-text composition, Qdrant indexes + upload
  umap.py              2-D UMAP projection → catalog-map artifact / px-py payloads
  commits.py           git-native tips: unwrap commits, diff, tombstone removed entities
  sources/subgraph.py  The Graph event queries
  sources/ipfs.py      IPFS CID resolution
  graph/projection.py  ROOT_PROFILES, the graph walk, and the shared join helpers
  graph/bundle.py      single-publisher bundle join
  graph/view.py        multi-source view fusion (union-find over global identity)

quickbeam/embeddings.py is now a thin back-compat facade that re-exports this package, so existing from quickbeam.embeddings import ... imports keep working; new code should import from the specific quickbeam.ingest.* module.

Installation

# From the repo root
python -m venv venv
source venv/bin/activate

pip install -e ".[gpu]"   # CUDA-accelerated embeddings (recommended for build)
pip install -e ".[cpu]"   # CPU-only fallback

This installs the quickbeam CLI entry point. Run quickbeam --help to see all commands.

quickbeam build    Build embeddings from subgraph / IPFS data into Qdrant
quickbeam watch    Live daemon: poll subgraph for new events and embed automatically
quickbeam serve    Start the Fangorn search API server (optionally with --watch + x402)
quickbeam mcp      Run the MCP server exposing search as agent tools (x402-aware)
quickbeam cdn      Semantic CDN: bake the embedded graph into static, pullable domain shards
quickbeam pull     Pull a domain from a Semantic CDN into a local Qdrant collection
quickbeam export   Export the Qdrant collection as an NDJSON bundle
quickbeam migrate  Migrate a local Qdrant collection to Qdrant Cloud
quickbeam data     An ETL pipeline to generate seed / test data from public data sources

The mcp and x402 layers need extra dependencies (FastMCP + EIP-712 signing):

pip install -e ".[agent]"   # fastmcp + eth-account + httpx
pip install -e ".[dev]"     # pytest + fastmcp + eth-account (to run the test-suite)

Quickstart

New: the git-native flow. Datasets are now versioned repos you commit and push (git for data), and watch/build embed off the commit diff. For the end-to-end runbook in that model — including delete propagation and inherited embedding contracts — see docs/NEW_QUICKSTART.md. The classic subgraph-event runbook below still works.

1. Run Qdrant

docker run -d -p 6333:6333 -p 6334:6334 \
  -v "$(pwd)/python/qdrant_storage:/qdrant/storage:z" \
  --name qdrant-demo \
  qdrant/qdrant

2. Build embeddings

Link the NVIDIA libraries if using CUDA:

export LD_LIBRARY_PATH=\
$VIRTUAL_ENV/lib/python3.12/site-packages/nvidia/cudnn/lib:\
$VIRTUAL_ENV/lib/python3.12/site-packages/nvidia/cublas/lib:\
$LD_LIBRARY_PATH
# verify Cuda is available
python -c "import onnxruntime as ort; print('Available Providers:', ort.get_available_providers())"

From a schema bundle

quickbeam build \
  --bundle fangorn.mb.creativecore.v1=0xac92db425c174e4301cd41e81e16d99fd2c5f4e2f13b739004996e95875e990d \
  --root-profile track \
  --graph-api-key <> \
  --ipfs-gateway https://green-reasonable-heron-957.mypinata.cloud/ipfs \
  --dim 256 \
  --umap \
  --reset

Pass --root-profile at least once (repeatable) to choose which projection(s) to emit. Use --view NAME=0x... instead of --bundle to fuse several publishers' bundles into one graph before projecting.

Resuming a build

quickbeam build is fully resumable. Progress is saved to --checkpoint-file (default ./db/ingest_checkpoint.json) at the granularity of individual bundle manifests. On re-run without --reset, already-completed manifests are skipped before any IPFS data is fetched — RAM usage stays flat regardless of how many records have already been embedded.

The checkpoint tracks:

completed_manifest_cids — manifests that have been fully embedded (skipped on re-run).
processed_track_ids — records within the currently in-flight manifest, used only for crash-recovery mid-manifest. Cleared when the manifest completes.
last_tip — per-schema, the last-built commit CID. The watcher diffs the new tip against it to embed only the delta and tombstone removed entities (git-native flow).

UMAP only (reproject existing collection)

quickbeam build --umap-only

3. Watch for new events (optional)

After an initial build, run quickbeam watch to keep the collection up to date as new manifests are published on-chain.

quickbeam watch \
  --bundle "fangorn.mb.bundle.v1=0xabc123..." \
  --graph-api-key <key> \
  --ipfs-gateway https://your-gateway.mypinata.cloud/ipfs \
  --ipfs-gateway-key <pinata-jwt> \
  --poll-interval 120

The watcher uses the same checkpoint file as the builder. On startup it reads last_block from the checkpoint and only queries subgraph events with blockNumber_gt: last_block, so it never re-scans the full history. The GPU model is loaded once and kept alive across poll cycles.

Git-native tips (commit diff). When a tip is a commit, the watcher: (1) at startup, resolves the tip commit for the watched schema and inherits its embedding contract (model/dim/distance) to size the collection — the CLI --embedding-model/--dim/distance flags are only a fallback when the commit carries no contract; (2) each cycle, diffs the new tip against last_tip and deletes the points of any entities the commit dropped (delete propagation) before embedding the additions. Raw-manifest tips (pre-commit publishes) are handled exactly as before.

Filter hierarchy

All filters are optional and combinable. Narrower filters reduce both subgraph load and embedding work.

# Watch all publishers, all dataset names:
quickbeam watch --bundle fangorn.mb.bundle.v1=0xabc...

# Only a specific publisher:
quickbeam watch --bundle fangorn.mb.bundle.v1=0xabc... \
  --owner 0xdeadbeef

# Only certain dataset names (any publisher):
quickbeam watch --bundle fangorn.mb.bundle.v1=0xabc... \
  --dataset Track Recording

# Most specific — one publisher's named datasets:
quickbeam watch --bundle fangorn.mb.bundle.v1=0xabc... \
  --owner 0xdeadbeef --dataset Track

--owner and --dataset are both repeatable. --dataset filters on the name field of the ManifestPublished event — the name the publisher assigned when registering the dataset.

4. Start the server

quickbeam serve \
  --collection fangorn \
  --qdrant-host localhost --qdrant-port 6333

The server starts immediately and serves whatever is already in Qdrant. Use POST /reingest to pull new subgraph data without restarting.

Serve + watch in one process

Pass --watch to run the live embedding daemon alongside the server, so one deployment both ingests and serves. Everything before --watch configures the server; everything after it is forwarded verbatim to quickbeam watch. The watcher writes to Qdrant; the server reads from it; the watcher is a child process that is terminated when the server exits.

quickbeam serve \
  --collection fangorn \
  --watch \
    --bundle "fangorn.mb.bundle.v1=0xabc123..." \
    --graph-api-key <key> \
    --ipfs-gateway https://your-gateway.mypinata.cloud/ipfs \
    --poll-interval 120

Note this loads the embedding model twice (once in each process), so plan VRAM accordingly — or run the two commands as separate services against the same Qdrant.

Snapshots

A snapshot is a portable copy of the populated Qdrant collection. It lets you seed a new instance — including one without a GPU — from a pinned IPFS artifact.

curl -X POST localhost:6333/collections/fangorn/snapshots
# grab the latest snapshot from qdrant
docker exec qdrant-core find /qdrant -name "*.snapshot"
# exfiltrate the latest snapshot from docker and store locally
docker cp qdrant-core:/qdrant/snapshots/fangorn/fangorn-8009660693873684-2026-06-16-22-11-57.snapshot ~/.snapshot
# zip the snapshot
gzip -k ~/.snapshot
# pin to ipfs (from the root)
node src/pinata.mjs upload ~/.snapshot.gz "fangorn-8009660693873684-2026-06-16-22-11-57.snapshot.gz"
# note the sha256 sum of the snapshot before cleanup
sha256sum ~/.snapshot 
rm -rf ~/.snapshot ~/.snapshot.gz

Export

# Full bundle (fields + embeddings — use this to seed a complete server)
quickbeam export --src http://localhost:8080 --out bundle.ndjson

# Embeddings only (track_id + vector — minimal artifact for vector-space clients)
quickbeam export --src http://localhost:8080 --out embeddings.ndjson --embeddings-only

Pin to IPFS

gzip -k bundle.ndjson
node src/pinata.mjs upload bundle.ndjson.gz "quickbeam-bundle-v1"

See Managing Pinata data for listing and bulk-deleting pinned files.

Export a Qdrant snapshot

# Write snapshot to Qdrant storage
curl -X POST localhost:6333/collections/fangorn/snapshots

# Find the file
docker exec qdrant-core find /qdrant -name "*.snapshot"

# Copy out and compress
docker cp qdrant-core:/qdrant/snapshots/fangorn/<snapshot-file> ~/.snapshot
gzip -k ~/.snapshot

# Pin to IPFS
node src/pinata.mjs upload ~/.snapshot.gz "<snapshot-file>.gz"

# Record the sha256 before cleanup
sha256sum ~/.snapshot
rm ~/.snapshot ~/.snapshot.gz

Seed on startup

quickbeam serve \
  -s test.sond3r.track.invariants.3=0x... \
  --bundle-cid QmYourBundleCIDHere

If the collection is empty and --bundle-cid is provided, the server fetches the NDJSON from IPFS and upserts it in the background. The server is live immediately — results populate as the seed progresses. If the collection already has points, the seed is skipped.

Manual import

# From a local file
cat bundle.ndjson | curl -X POST http://localhost:8080/bundle/import \
  -H "Content-Type: application/x-ndjson" \
  --data-binary @-

# Stream directly between two instances
curl -N http://host-a:8080/bundle/export \
  | curl -X POST http://host-b:8080/bundle/import \
       -H "Content-Type: application/x-ndjson" \
       --data-binary @-

Semantic CDN

The search server runs queries server-side — which means the node observes every query vector, and a semantic query is intent. The Semantic CDN inverts this: the operator distributes the public embeddings as static, content-addressed artifacts; the user pulls a slice into their own local Qdrant and queries it offline. Knowledge moves to the user, the network never sees a query. Full walkthrough in docs/SEMANTIC_CDN.md; the short version:

# (operator) declare domains as filters over the collection
cat > domains.json <<'JSON'
{ "domains": {
  "music":  { "description": "Recordings & artists", "filter": { "entityType": ["Recording","Artist"] } },
  "venues": { "description": "Places & events",       "filter": { "entityType": ["Place","Event"] } }
} }
JSON

# (operator) bake immutable shards from Qdrant, then serve them statically
quickbeam cdn bake  --config domains.json --cdn-dir ./cdn --collection fangorn
quickbeam cdn serve --cdn-dir ./cdn --port 8090

# (user) pull a domain into a LOCAL collection, then query it offline
quickbeam pull music --cdn-url http://localhost:8090 --collection music_local
quickbeam serve --collection music_local      # local search — CDN sees nothing

A domain is operator-declared (a named entityType/owner filter, in domains.json). bake writes cdn/<domain>/shard-NNNN.ndjson.gz (reusing the /bundle/export row shape) plus a manifest.json carrying a sha256 per shard, and a top-level catalog.json.

Each manifest.json is also self-describing so a pulled domain drives a generic, schema-agnostic client with no hardcoding: an inferred role_map (title/subtitle/tags/spatial/…) and an entity_types vocabulary with per-type counts are always baked in. Two optional per-domain keys in domains.json add more — bundle_schema (path to a Fangorn bundle schema → copies its type + relationship vocabulary into manifest.bundle) and presentation (an overlay of icons / accent colors / fieldLabels / externalUrl templates, passed through verbatim for UI polish). See docs/SEMANTIC_CDN.md. serve is a separate minimal FastAPI app exposing only static reads (/catalog, /domains/{name}/manifest, /domains/{name}/edges, /domains/{name}/shards/{file}) with HTTP Range support, so shards are cacheable and downloads resume. pull verifies every shard against its sha256 and loads it into the local collection with deterministic point ids, so an interrupted or repeated pull is safe.

Alongside the semantic axis (record shards), a domain can carry a relational axis — its linkset of typed edges ({rel, from, to, fromType, toType}, see linkgen). Edges live at cdn/<name>/edges.json, served at /domains/{name}/edges, so a pull-client (the MCP server) can walk the knowledge graph offline. Edge endpoints are the same node ids as records' track_id, so the two axes join by id. Two ways to populate it:

Live — quickbeam watch ships the typed edges it fetches on-chain each cycle, merging them into edges.json (deduped, incremental — the relational counterpart to the record delta shards). The relational axis stays fresh with the stream, no manual step.
One-shot — quickbeam cdn edges --domain <name> --source <linkset.json> installs a linkset from a file (e.g. a staged stage_volumes/*_edges.json).

Re-run cdn serve after first attaching edges so the running app picks up the /edges route.

Managing Pinata data

src/pinata.mjs is a small CLI for the Pinata account that backs your IPFS pins (snapshots, bundles). It needs PINATA_JWT in the environment (or a .env at the repo root).

# Upload / pin a file (replaces the old pin.mjs)
node src/pinata.mjs upload ~/.snapshot.gz "sond3r.snapshot.2026-06-14.gz"

# List pins (optionally filter by name substring)
node src/pinata.mjs list
node src/pinata.mjs list --name sond3r

# Delete by file ID(s)
node src/pinata.mjs delete <id1> <id2>

# Bulk-delete every file whose name matches a substring (prompts unless --yes)
node src/pinata.mjs delete-pattern "sond3r.snapshot"

# Delete everything in the account (prompts unless --yes)
node src/pinata.mjs delete-all

Pinata's name filter is a contains match, not a strict prefix — name files with a consistent prefix (e.g. sond3r.snapshot.*) for clean targeting. Also available via npm run pinata -- <args>.

Migrating to Qdrant Cloud

quickbeam migrate

migrate.py contains hardcoded source/destination credentials — edit it before running.

Then point the server at the cloud cluster:

quickbeam serve \
  -s ... \
  --qdrant-url https://your-cluster.cloud.qdrant.io:6334 \
  --qdrant-api-key <key>

x402 payment gating

x402 is the HTTP 402 Payment Required protocol for paid APIs. quickbeam implements the exact scheme over an EVM stablecoin (USDC by default) using EIP-3009 transferWithAuthorization signatures. It lives in quickbeam/x402.py and is used in two independent places:

HTTP gating — quickbeam serve --x402-pay-to 0x... installs middleware that gates the search routes (/search, /search/vector, /search/text).
Per-tool gating — quickbeam mcp --x402-pay-to 0x... charges the calling agent per MCP tool call (see MCP server).

The flow

Client calls a gated route with no X-PAYMENT header → server replies 402 with a JSON body { x402Version, accepts: [requirements], error }.
Client signs an EIP-3009 authorization for the quoted price, base64-encodes the payment into X-PAYMENT, and retries.
Server verifies the signature, settles, and serves the response with an X-PAYMENT-RESPONSE header describing settlement.

Verification is pluggable. By default a local verifier recovers the EIP-712 signer and checks the authorization terms without broadcasting — suitable for testnets, demos, and tests. Point --x402-facilitator <url> at a real facilitator for on-chain verify + settle.

# Gate the HTTP search routes at 0.001 USDC per request on Base Sepolia:
quickbeam serve \
  -s test.sond3r.track.invariants.3=0x... \
  --x402-pay-to 0xYourReceivingAddress \
  --x402-price 0.001 \
  --x402-network base-sepolia

Supported networks: base-sepolia (default), base, avalanche-fuji. Each has a default USDC contract; override with --x402-asset.

Agent-side helper

quickbeam/x402.py also ships PayingClient, an httpx.AsyncClient wrapper that transparently pays any 402 it receives (sign → retry → record settlement). This is the agent side, used by the test-suite and available for any Python client.

MCP server

quickbeam mcp is a Model Context Protocol server (quickbeam/mcp_server.py) that exposes on-chain-published knowledge to agents. It is a self-contained, local pull-client of the Semantic CDN: it pulls a dataset's immutable shards into an in-process index and searches them locally — the agent's query vector never leaves the process. That is the "intent is private" half of the Fangorn thesis, applied to the agent path (no query hits a central server, and there is no dependency on a live quickbeam serve).

Agents get back the raw record fields (not a lossy title/subtitle/tags role-map projection — an LLM reasons over JSON fine) and navigate two axes: semantic (vector similarity) and relational (typed linkset edges — the knowledge-mesh axis).

New here? docs/MCP_QUICKSTART.md walks an agent from zero to querying a live dataset (serve → watch → MCP → the five tools → registering with Claude Code).

# Phase 1 — free tools, remote streamable-http transport:
quickbeam mcp --transport http --host 0.0.0.0 --port 8765 \
  --cdn-url http://localhost:8090

# local stdio (MCP Inspector / Claude Desktop):
quickbeam mcp --transport stdio --cdn-url http://localhost:8090

Tools

list_datasets() — the CDN catalog: what knowledge exists (name, description, count, entity types, embedding dim). Free.
describe(dataset) — a dataset's entity types, real field vocabulary, relationship types (for neighbors), and embedding contract (model + dim). Free.
search(dataset, query, limit=10, entity_type=None, owner=None) — meaning-based search. Embeds the query locally, returns records as { id, entityType, fields, score, provenance } with the raw fields. Optional structured pre-filters by entity_type / owner.
get(dataset, id) — one record by its exact id (which is also its graph node endpoint, e.g. rh:asset:NVDA). Free.
neighbors(dataset, id, rel=None, direction="both", limit=25) — walk the linkset edges from a node ("what is connected to NVDA, and how"). Neighbors inside the dataset resolve to full fields; those outside it come back as { id, entityType } endpoints.

Relational-axis delivery. neighbors sources edges from the CDN's /domains/{name}/edges endpoint, which quickbeam watch keeps fresh live (or cdn edges one-shot). If a domain has no CDN linkset yet, it falls back to a local one via --edges <file-or-dir> (a JSON list of {rel, from, to, fromType, toType}, the shape linkgen/robinhood stage), and reports relational_axis: "not delivered" if neither is present.

Provenance

Every result carries on-chain provenance as a first-class field, sourced from each Qdrant point's meta:

"provenance": {
  "source_cid": "Qm…",          // manifest CID the record was published in
  "published":  "2026-06-14T…",  // ISO8601 from the block timestamp
  "version":    1,
  "publisher":  "0x…"            // publisher address
}

Phase 2 — charge agents per call

x402 gating for the MCP is phased and isolated in quickbeam/mcp_payments.py; with no --x402-pay-to, none of it runs and the tools are free. When enabled, each gated tool gains an optional payment argument:

quickbeam mcp --transport http \
  --x402-pay-to 0xYourReceivingAddress \
  --x402-price 0.001 --x402-network base-sepolia

Since MCP has no HTTP headers, payment rides on a tool argument instead of X-PAYMENT:

Agent calls search(dataset, query) with no payment → the tool returns the x402 requirements: { payment_required: true, accepts: [...] }.
Agent signs the quoted requirement and calls again with payment=<base64> → the tool returns results plus a payment settlement receipt.

The gated tools are the compute-bearing ones (search, neighbors); discovery (list_datasets, describe, get) stays free. The verify/settle primitives are reused verbatim from x402.py; only the transport (tool argument vs HTTP header) differs.

Embedding quality note. nomic-embed-text-v1.5 is asymmetric — documents are embedded with a search_document: prefix and queries with search_query:. The pull-client embeds queries locally with the search_query: prefix and applies the same matryoshka transform (LayerNorm → slice-to-dim → L2-normalize) the builder applied to documents, so query and document vectors share one space. Reusing that single transform (quickbeam.embeddings.matryoshka) is what keeps local retrieval correct.

Data pipelines

The quickbeam data subcommands generate seed data for testing. quickbeam data fetch outputs flat { name, fields } JSONL consumed by the ingest server's flat-schema path. quickbeam data mb outputs v3 bundle chunk files (node chunks + edge chunk) consumed by quickbeam build --bundle.

Last.fm + MusicBrainz

Scrapes artist discographies via the Last.fm API and optionally enriches with ISRC codes and contributors from MusicBrainz.

export LASTFM_API_KEY=your_key

quickbeam data fetch --volume 1 --max-gb 9.5
# Resumes automatically if interrupted — re-run the same command.
# When the volume ceiling is hit, upload and increment --volume.

quickbeam data fetch --volume 2 --max-gb 9.5
quickbeam data fetch --volume 1 --artists-file artists.txt   # custom seed list
quickbeam data fetch --volume 1 --no-mb                      # skip MusicBrainz lookups

Outputs volume_<N>_core.jsonl (structural) and volume_<N>_taxonomy.jsonl (genres/moods/themes/contexts).

MusicBrainz JSON dump

Downloads the full MusicBrainz release.tar.xz dump (~23 GB) and extracts up to --target-count tracks with tag data. Resumable at every stage — re-run to pick up where it left off. The latest dump URL is discovered automatically; pass --dump-url to pin a specific one.

quickbeam data mb --volume 1 --target-count 50000 --output-dir ./data
quickbeam data mb --volume 1 --target-count 50000 --connections 8   # faster download
quickbeam data mb --help

The download uses --connections (default 4) parallel HTTP range requests, each writing to a non-overlapping slice of a pre-allocated file. A .parts sidecar tracks completed chunks so interrupted runs skip them on restart. Pass --connections 1 to fall back to single-connection streaming.

Outputs three v3 bundle chunk files — ready to upload to IPFS and register as a bundle schema:

File	Contents
`volume_<N>_tracks.json`	`[{ id, type: "Track", fields: { trackId, isrcCode, title, byArtist, albumName, datePublished, durationMs, contributors, _mbid } }, ...]`
`volume_<N>_taxonomies.json`	`[{ id: "taxonomy:<trackId>", type: "TrackTaxonomy", fields: { trackId, genres, moods, themes, contexts } }, ...]`
`volume_<N>_edges.json`	`[{ rel: "hasTaxonomy", from: "<trackId>", to: "taxonomy:<trackId>" }, ...]`

These three files are the raw v3 bundle chunks — Track + TrackTaxonomy node files plus an edge file. Use src/publish_mb_bundle.ts (see End-to-end workflow below) to register schemas and publish them to Fangorn, then run quickbeam build --bundle to embed.

OpenStreetMap changesets

Fetches recent changesets within a bounding box from the public OSM API. Demonstrates that adding a new domain is a schema change, not an architecture change — the same ingest server handles OSM data automatically via role inference (title←comment, subtitle←user_id, spatial←bbox, etc.).

# Edit BBOX, TARGET_COUNT, DAYS_BACK in quickbeam/pipelines/osm.py first, then:
quickbeam data osm

Outputs stage_volumes/osm_changesets.json.

End-to-end workflow (MusicBrainz → Fangorn → Qdrant)

Step 1 — Generate the bundle chunk files

quickbeam data mb --volume 1 --target-count 50000 --output-dir ./data
# produces: data/volume_1_tracks.json
#           data/volume_1_taxonomies.json
#           data/volume_1_edges.json

Step 2 — Register schemas and publish to Fangorn

Publishing is moving to the git-native flow. The publish_*.ts scripts below still work, but they use the older raw-manifest publish path — no commit history, no structural sharing, no embed contract. The target is fangorn commit --bundle/--view

fangorn push (the same primitives record-set repos already use today), with the dataset-shaping/sharding half of these scripts folding into quickbeam data publish. See docs/NEW_QUICKSTART.md for the flow and what's live vs. planned.

src/publish_mb_bundle.ts must be placed in the fangorn-sdk src/ directory alongside setup-embeddings-testdata.ts (it imports TestBed and the SDK type system from there).

# from the fangorn-sdk root:
cp /path/to/quickbeam/embeddings/src/publish_mb_bundle.ts src/

pnpm dotenvx run -f .env -- tsx src/publish_mb_bundle.ts \
  --tracks      /path/to/data/volume_1_tracks.json \
  --taxonomies  /path/to/data/volume_1_taxonomies.json \
  --edges       /path/to/data/volume_1_edges.json \
  --dataset     ds.mb.v1

On first run this registers three schemas (all idempotent — safe to re-run):

Schema	Name	Description
Track	`fangorn.mb.track.v1`	Invariant metadata per recording
TrackTaxonomy	`fangorn.mb.track.taxonomy.v1`	Genre / mood tags
Bundle	`fangorn.mb.bundle.v1`	Track `—hasTaxonomy→` TrackTaxonomy

Large volumes are published in batches (--batch-size, default 2000). Progress is saved to tmp/mb-publish-ledger.json — re-run the same command to resume after a failure.

When done the script prints the bundle name and ID:

  bundle name : fangorn.mb.bundle.v1
  bundle id   : 0xabc123...

Step 3 — Build embeddings

quickbeam build \
  --bundle "fangorn.mb.bundle.v1=0xabc123..." \
  --root-profile track \
  --graph-api-key <key> \
  --ipfs-gateway https://your-gateway.mypinata.cloud/ipfs \
  --ipfs-gateway-key <pinata-jwt> \
  --dim 256 \
  --umap \
  --reset

Configuration reference

All config is via CLI flags. Run quickbeam build --help or quickbeam serve --help for the full list.

`quickbeam build`

Flag	Default	Description
`--bundle`		`NAME=0x...` bundle schema — walks one publisher's typed graph.
`--view`		`NAME=0x...` composed view — fuses several publishers' bundles into one graph before projecting. Mutually exclusive with `--bundle`.
`--root-profile`	required	Named projection to emit, repeatable (see `ROOT_PROFILES`). e.g. `--root-profile track`
`--profiles-file`		JSON file of custom/override root profiles, merged over the built-ins
`--max-depth`	`2`	Graph-walk depth for profiles that don't set one
`--subgraph-url`	Fangorn studio URL	The Graph subgraph endpoint
`--graph-api-key`	`""`	The Graph gateway API key
`--ipfs-gateway`	`https://gateway.pinata.cloud/ipfs`	IPFS gateway
`--qdrant-host`	`localhost`	Qdrant host
`--qdrant-port`	`6333`	Qdrant HTTP port
`--qdrant-grpc-port`	`6334`	Qdrant gRPC port
`--collection`	`quickbeam`	Qdrant collection name
`--checkpoint-file`	`./db/ingest_checkpoint.json`	Resume state file
`--embedding-model`	`nomic-ai/nomic-embed-text-v1.5`	fastembed model name
`--dim`	`256`	Matryoshka output dimensions: 256, 512, or 768
`--embed-batch`	`16`	GPU embed batch size — lower for small VRAM
`--searchable-fields`	`auto`	Comma-separated field allowlist, or `auto`
`--page-size`	`100`	Subgraph pagination page size
`--ipfs-timeout`	`20`	IPFS request timeout in seconds
`--concurrency`	`16`	Max concurrent IPFS fetches
`--umap`	`false`	Compute and store UMAP px/py after ingest
`--umap-only`	`false`	Skip ingest; only (re)compute UMAP on existing collection
`--umap-neighbors`	`15`	UMAP n_neighbors parameter
`--umap-min-dist`	`0.05`	UMAP min_dist parameter
`--reset`	`false`	Delete and recreate the Qdrant collection on startup

`quickbeam watch`

Flag	Default	Description
`--bundle`	required	`NAME=0x...` bundle schema to watch
`--root-profile`	required	Named projection to emit, repeatable. e.g. `--root-profile asset --root-profile transfer`
`--owner`		Filter to this publisher address. Repeatable.
`--dataset`		Filter to these dataset names. Accepts multiple values.
`--poll-interval`	`60`	Seconds between subgraph polls
`--subgraph-url`	Fangorn studio URL	The Graph subgraph endpoint
`--graph-api-key`	`""`	The Graph gateway API key
`--ipfs-gateway`	`https://gateway.pinata.cloud/ipfs`	IPFS gateway
`--ipfs-gateway-key`		Bearer token for authenticated IPFS gateways
`--qdrant-host`	`localhost`	Qdrant host
`--qdrant-port`	`6333`	Qdrant HTTP port
`--qdrant-grpc-port`	`6334`	Qdrant gRPC port
`--collection`	`fangorn`	Qdrant collection name
`--checkpoint-file`	`./db/ingest_checkpoint.json`	Shared with `build` — tracks `last_block` and completed manifests
`--embedding-model`	`nomic-ai/nomic-embed-text-v1.5`	fastembed model name
`--dim`	`256`	Matryoshka output dimensions
`--embed-batch`	`16`	GPU embed batch size
`--role-map-file`	`./db/role_map.json`	Role map path — loaded if present, inferred on first cycle otherwise
`--searchable-fields`	`auto`	Field allowlist or `auto`
`--page-size`	`100`	Subgraph pagination page size
`--ipfs-timeout`	`20`	IPFS request timeout in seconds
`--concurrency`	`16`	Max concurrent IPFS fetches

`quickbeam serve`

Flag	Default	Description
`--schema` / `-s`		`NAME=0x...` schema ID pair. Repeatable.
`--primary` / `-p`	first schema	Join key schema
`--subgraph-url`	Fangorn studio URL	The Graph endpoint
`--graph-api-key`	`""`	The Graph gateway API key
`--ipfs-gateway`	`https://gateway.pinata.cloud/ipfs`	IPFS gateway
`--qdrant-url`	`None`	Qdrant Cloud URL — overrides `--qdrant-host`/`--qdrant-port`
`--qdrant-api-key`	`None`	Qdrant Cloud API key
`--qdrant-host`	`localhost`	Qdrant host (local)
`--qdrant-port`	`6333`	Qdrant HTTP port
`--qdrant-grpc-port`	`6334`	Qdrant gRPC port
`--collection`	`quickbeam`	Qdrant collection name
`--embedding-model`	`nomic-ai/nomic-embed-text-v1.5`	Must match the builder
`--bundle-cid`	`None`	IPFS CID of an NDJSON bundle to seed from on first startup
`--searchable-fields`	`auto`	Field allowlist or `auto`
`--host`	`0.0.0.0`	Bind host
`--port`	`8080`	Bind port
`--reset`	`false`	Drop and recreate collection on startup
`--x402-pay-to`	`None`	Recipient address. Enables x402 gating on the search routes when set.
`--x402-price`	`0.001`	Price per gated request in whole token units
`--x402-network`	`base-sepolia`	EVM network: `base-sepolia`, `base`, `avalanche-fuji`
`--x402-asset`	network USDC	Token contract address
`--x402-decimals`	`6`	Token decimals for the price → atomic conversion
`--x402-facilitator`	`None`	Facilitator URL for on-chain verify+settle (omit for local verification)

Plus --watch <watch args...> to run the live daemon alongside the server.

`quickbeam mcp`

Flag	Default	Description
`--cdn-url`	`http://localhost:8090`	Base URL of the Semantic CDN it pulls datasets from
`--edges`	`None`	Local linkset JSON file or directory (relational axis), until the CDN delivers edges
`--transport`	`http`	`http` (streamable-http), `stdio`, or `sse`
`--host`	`0.0.0.0`	Bind host (http/sse)
`--port`	`8765`	Bind port (http/sse)
`--x402-pay-to`	`None`	Recipient address. Enables per-tool payment (Phase 2) when set.
`--x402-price`	`0.001`	Price per gated tool call in whole token units
`--x402-network`	`base-sepolia`	EVM network
`--x402-asset`	network USDC	Token contract address
`--x402-decimals`	`6`	Token decimals
`--x402-facilitator`	`None`	Facilitator URL (omit for local verification)

Env equivalents: QUICKBEAM_CDN_URL, QUICKBEAM_EDGES.

`quickbeam export`

Flag	Default	Description
`--src`	required	Source server URL, e.g. `http://localhost:8080`
`--out`	`bundle.ndjson`	Output file path
`--owner`	`None`	Filter export to a single owner address
`--embeddings-only`	`false`	Export only `track_id` + `embedding`, omit fields and metadata

`quickbeam cdn bake`

Flag	Default	Description
`--config`	`domains.json`	Operator domain config: `name → { description, filter }`
`--cdn-dir`	`./cdn`	Output directory for baked shards
`--collection`	`fangorn`	Source Qdrant collection to bake from
`--domain`	all	Bake only this one domain from the config
`--shard-size`	`50000`	Points per shard file
`--limit`	`0`	Cap total points baked per domain (0 = all). Use a small value for a lightweight in-browser snapshot.
`--scroll-batch`	`2000`	Qdrant scroll page size
`--embedding-model`	`nomic-ai/nomic-embed-text-v1.5`	Recorded in the manifest (Qdrant doesn't store it)
`--qdrant-url` / `--qdrant-api-key`	`None`	Qdrant Cloud (overrides host/port)
`--qdrant-host` / `--qdrant-port` / `--qdrant-grpc-port`	`localhost`/`6333`/`6334`	Local Qdrant

A domain's filter accepts entityType: [...] and owner: [...] (each a MatchAny); multiple keys are AND-ed. An empty/missing filter selects the whole collection.

`quickbeam cdn edges`

Flag	Default	Description
`--cdn-dir`	`./cdn`	Baked CDN directory (the domain must already be baked)
`--domain`	required	Domain to attach the linkset to
`--source`	required	Linkset JSON — a list of `{rel, from, to, fromType, toType}` edges, or `{edges:[...]}`

Installs the relational axis as cdn/<domain>/edges.json (served at /domains/{name}/edges) and records the edge count + relation types in the catalog.

`quickbeam cdn serve`

Flag	Default	Description
`--cdn-dir`	`./cdn`	Directory of baked shards to serve
`--host`	`0.0.0.0`	Bind host
`--port`	`8090`	Bind port
`--cors`	`false`	Enable permissive CORS (for browser-based pulls)

`quickbeam pull`

Flag	Default	Description
`domain`	required	Positional — domain name to pull (see the CDN's `/catalog`)
`--cdn-url`	`http://localhost:8090`	Base URL of the Semantic CDN
`--collection`	domain name	Local Qdrant collection to load into
`--cache-dir`	`./db/cdn_cache`	Where downloaded shards are cached
`--concurrency`	`4`	Parallel shard downloads
`--batch`	`500`	Upsert batch size
`--reset`	`false`	Recreate the local collection before loading
`--download-only`	`false`	Fetch + verify shards but don't load into Qdrant
`--qdrant-url` / `--qdrant-api-key` / `--qdrant-host` / `--qdrant-port` / `--qdrant-grpc-port`	local	Target Qdrant for the local collection

API

All endpoints return JSON. Hits are shaped as { id, fields, owner, meta, score?, embedding? }, where meta carries on-chain provenance { manifestCid, blockTimestamp, version, owner }.

When --x402-pay-to is set, /search, /search/vector, and /search/text require an X-PAYMENT header — see x402 payment gating.

`GET /browse`

Paginated browse. ?limit=20&offset=0

`GET /search`

Semantic search by text — embeds the query server-side. ?q=late+night+driving&n_results=10&owner=0x...

`POST /search/vector`

Query by raw embedding vector.

{ "embedding": [...], "n_results": 20, "owner": "0x..." }

`POST /search/text`

Lexical search over an in-memory index of title, subtitle, and tag fields. Faster than semantic search for exact name lookups.

{ "q": "arctic monkeys", "limit": 20, "owner": "0x..." }

`POST /embed`

Embed text using the same model as ingestion — keeps client and server embedding spaces aligned.

{ "text": "late night melancholic indie" }   → { "embedding": [...] }
{ "texts": ["track one", "track two"] }       → { "embeddings": [[...], [...]] }

`GET /schema`

Inferred semantic role map (title, subtitle, tags, etc.) and facet vocabularies for the active dataset.

`GET /catalog/map`

2D UMAP projection of the full collection for a galaxy/map view. Computed on first request and cached. Returns { "computing": true } while still running.

`POST /catalog/map/refresh`

Invalidates the map cache and recomputes in the background.

`GET /bundle/export`

Streams the full collection as NDJSON — one point per line. ?owner=0x...&limit=1000&offset=0

`POST /bundle/import`

Streaming NDJSON import — reads line by line, upserts in batches of 500.

`POST /bundle/upsert`

JSON body upsert for pre-embedded points (smaller programmatic use).

`GET /health`

Collection count, schema map, role map, cache state, checkpoint info.

`POST /reingest`

Triggers a background re-ingestion from the subgraph. Only re-embeds changed documents.

`POST /reingest/full`

Clears the checkpoint and re-ingests everything from scratch. Does not drop the Qdrant collection.

`GET /debug`

Join diagnostics — matched/unmatched track IDs across primary and secondary schemas.

Join semantics

Graph projection

One record is emitted per node whose type matches a --root-profile's root_type; the root node's stable, publisher-assigned id (or its Entity URI in a view) is the join key.
The profile walks the undirected graph up to max_depth hops from the root, folding the neighbor types it lists in include into grouped, deduped, capped label lists.
Nodes not reachable from any root contribute no fields.
Manifests that are not valid bundles (missing kind: "bundle" or edgeChunks) are skipped.

The semantic role map (title, subtitle, tags, spatial, etc.) is inferred automatically from field names and value shapes across the merged dataset. This is what makes the same server and app work for music tracks, OSM changesets, or any other Fangorn schema without per-domain configuration.

Name		Name	Last commit message	Last commit date
Latest commit History 32 Commits
example-robinhood-source		example-robinhood-source
examples		examples
playground		playground
quickbeam.egg-info		quickbeam.egg-info
quickbeam		quickbeam
src		src
tests		tests
.gitignore		.gitignore
README.md		README.md
package-lock.json		package-lock.json
package.json		package.json
pyproject.toml		pyproject.toml
tsconfig.json		tsconfig.json
watch_robinhood.sh		watch_robinhood.sh

Uh oh!

Folders and files

Latest commit

History

Repository files navigation

quickbeam

How it works

Ingest engine layout

Installation

Quickstart

1. Run Qdrant

2. Build embeddings

From a schema bundle

Resuming a build

UMAP only (reproject existing collection)

3. Watch for new events (optional)

Filter hierarchy

4. Start the server

Serve + watch in one process

Snapshots

Export

Pin to IPFS

Export a Qdrant snapshot

Seed on startup

Manual import

Semantic CDN

Managing Pinata data

Migrating to Qdrant Cloud

x402 payment gating

The flow

Agent-side helper

MCP server

Tools

Provenance

Phase 2 — charge agents per call

Data pipelines

Last.fm + MusicBrainz

MusicBrainz JSON dump

OpenStreetMap changesets

End-to-end workflow (MusicBrainz → Fangorn → Qdrant)

Step 1 — Generate the bundle chunk files

Step 2 — Register schemas and publish to Fangorn

Step 3 — Build embeddings

Configuration reference

quickbeam build

quickbeam watch

quickbeam serve

quickbeam mcp

quickbeam export

quickbeam cdn bake

quickbeam cdn edges

quickbeam cdn serve

quickbeam pull

API

GET /browse

GET /search

POST /search/vector

POST /search/text

POST /embed

GET /schema

GET /catalog/map

POST /catalog/map/refresh

GET /bundle/export

POST /bundle/import

POST /bundle/upsert

GET /health

POST /reingest

POST /reingest/full

GET /debug

Join semantics

Graph projection

About

Resources

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Uh oh!

`quickbeam build`

`quickbeam watch`

`quickbeam serve`

`quickbeam mcp`

`quickbeam export`

`quickbeam cdn bake`

`quickbeam cdn edges`

`quickbeam cdn serve`

`quickbeam pull`

`GET /browse`

`GET /search`

`POST /search/vector`

`POST /search/text`

`POST /embed`

`GET /schema`

`GET /catalog/map`

`POST /catalog/map/refresh`

`GET /bundle/export`

`POST /bundle/import`

`POST /bundle/upsert`

`GET /health`

`POST /reingest`

`POST /reingest/full`

`GET /debug`

Packages