Document ingestion for retrieval systems: get text out of a document, split it into units worth retrieving, and turn those into vectors.
Extracted from a production Thai government document system, where the hard part was never the embedding call — it was everything around it.
Chunking legal documents by their own structure instead of by character count is worth 5–10× in retrieval precision. Each statute uploaded as a single file, scored on whether the best-matching chunk identifies the clause that answers the question:
| Document | char-window | structure-aware | |
|---|---|---|---|
| Thai | กฎกระทรวง on state property (54 ข้อ) | 8% | 83% |
| Vietnamese | Luật nghĩa vụ quân sự 2015 (62 Điều) | 16% | 75% |
| EU English | GDPR — Regulation (EU) 2016/679 (99 Articles) | 33% | 91% |
| Japanese | 個人情報保護法 / APPI (185 条) | 0% | 83% |
Four languages, four legal domains, four drafting traditions — and the baselines
are predictable from clause density, which was tested by predicting the Japanese
result before running it. Reproduce any of them in one command —
cargo run --release --example eval_legal_chunking [-- vi|eu|ja] — see
eval/ for the setup, every miss, and why the window collapses: its
chunks retrieve relevant text but cannot be cited, which for legal work makes
them worthless.
use docpipe::{ai::{AiClient, AiConfig}, pipeline, profile::ChunkCfg};
let ai = AiClient::new(AiConfig::from_env());
let out = pipeline::prepare(&ai, text, &ChunkCfg::default(), true).await?;
for (chunk, vector) in out.chunks.iter().zip(out.vectors.iter()) {
// index chunk.content at chunk.char_start..chunk.char_end
}Structure-aware chunking for legal documents. Statutes and regulations are split by their own units, with the chapter breadcrumb and clause label prepended, so each chunk is a self-contained, citable clause instead of an arbitrary window that cuts a clause in half:
หมวด ๒ การจัดหาประโยชน์ › ข้อ ๙
กำหนดระยะเวลาเช่าไม่เกินสามสิบปี
เว้นแต่มีเหตุจำเป็น
Two drafting traditions ship today and the right one is detected automatically:
| Jurisdiction | Citable unit | Headings | Benchmarked |
|---|---|---|---|
| Thai | ข้อ / มาตรา | หมวด / ส่วนที่ | yes — 8% → 83% |
| Vietnamese | Điều | Chương / Mục (Roman numerals) | yes — 16% → 75% |
| EU English | Article | CHAPTER / Section (Roman numerals) | yes — 33% → 91% |
| Japanese | 第N条 | 第N章 / 第N節 (kanji numerals) | yes — 0% → 83% |
Adding one is a Grammar — a handful of keywords — plus a fixture pair under
eval/. The grammar is the cheap part; the measurement is the point,
and a jurisdiction isn't really done until it has numbers.
Ordinary prose falls back to a sliding character window, so you can point this at a mixed corpus. The strategy that actually ran is returned, not re-derived — your telemetry can't disagree with reality.
Text extraction, including from scans. extract::text() reads a PDF's text
layer and falls back to OCR when there isn't one — which is what a scanned
document is, and you can't tell until you try. TextSource reports which
answered, because OCR'd text carries recognition error:
cargo run --features pdf --example extract_text -- contract.pdfPDF text-layer reading is behind the pdf feature (pdf-extract is a heavy
dependency; callers who already have text shouldn't pay for it). The scanned
path works without it.
OCR with a vision-model upgrade path. Tesseract locally (with pdftoppm for
scanned PDFs), and if you point TyphoonConfig::base_url at any OpenAI-compatible
vision endpoint, that becomes the preferred engine with Tesseract as the
per-page fallback. Both are optional: with neither installed, OCR cleanly returns
None rather than failing your pipeline.
Per-document-kind configuration. profile::Stages describes how a class of
document should be processed — OCR mode and language, chunk strategy and sizes,
summariser window, which stages run at all — as plain serde data you can store
wherever you keep configuration.
An OpenAI-compatible client that talks to Ollama, vLLM, LiteLLM, OpenAI or Azure without a code change, and reports token usage on every call so you can meter it.
- No database, no storage, no HTTP. This crate computes; persisting the result is yours.
- No authorization. It never decides who may see a result, only how to produce one.
- No Office formats. Extracting .docx/.xlsx reliably means shelling out to LibreOffice — a 400 MB dependency and a subprocess, which is a deployment decision, not a library one. Convert to PDF first, or hand in the text. PDFs and images are covered.
- No configuration discovery. Everything takes the config it needs. The one
exception is an explicit
Config::from_env()you can decline to call — so two callers in one process can be configured differently, and tests are hermetic.
pipeline::prepare does chunk-then-embed in one call. If you report per-stage
progress, use pipeline::chunk and pipeline::embed_chunks separately so you can
write your own telemetry between them — there's no observer/callback abstraction,
because a sync callback can't await a host that persists progress, and a
channel drained by a spawned task reorders events against the caller's own.
Deployment-wide switches arrive as arguments, not as configuration this crate
reads: chunk_text_labelled(.., legal_allowed), Stages::citations_on(env_allows).
A per-document profile can narrow what the operator permits, never widen it.
Extracted from a running system and used by it, but young as a standalone crate: the API may still move before 1.0.
CI runs tests (--all-features, so the pdf path is compiled too), clippy with
warnings denied, the offline chunking example, and a docs build with broken
intra-doc links treated as errors. The retrieval benchmark is deliberately not
in CI: it needs an embedding endpoint, and a benchmark that quietly depends on
whichever model a runner happens to have is worse than one you run on purpose.
MIT.