Reconstructs the structure of a scanned book — chapters, titles, footnotes, page offsets — from output produced by Marker.
A folio is the page number printed on the page itself. Recovering the mapping from those printed folios to actual PDF page indices is the core problem this library solves, and the one everything else depends on.
Marker turns a PDF into Markdown/JSON, and on a clean digital PDF that is enough. On a scanned book it is not:
- there is no PDF outline to read chapters from
- running headers and footers repeat on every page and pollute the body text
- the printed folios do not line up with PDF page indices, and the offset is not constant — it drifts across front matter
- OCR mangles the contents page itself, which is exactly the page you need
- some scans are two-up: one landscape image holding two book pages, which scrambles reading order unless the spread is split first
This package is the layer that sits on top of Marker's output and reconstructs
the book's structure anyway. Pure Python, standard library plus pypdfium2.
Not affiliated with Datalab or the Marker project. This reads Marker's JSON output; it contains none of Marker's code and none of its model weights.
pip install -e .The import name is folio; the distribution name is folio-book, because the
PyPI name folio is held by an unrelated package last released in 2013.
from folio import convert
result = convert(marker_json, embedded_page_texts=embedded)
result["chapters"] # reconstructed chapters
result["metadata"]["chapterDetection"] # which tier produced them
result["metadata"]["totalChapters"]Each chapter carries title, startPage, wordCount, plainText and
separated footnotes.
Detection runs as a tiered cascade, strongest signal first, degrading to a safe fallback rather than guessing:
| Tier | Signal |
|---|---|
clean-toc |
OCR'd contents page, cleaned and anchored |
embedded-clean-toc |
the PDF's embedded text layer, when OCR mangled the contents page but the text layer is intact |
clean-body |
chapter openers detected in the body when no usable contents page exists |
full-text |
safe fallback — one unit, no fabricated structure |
Around that cascade:
clean_toc— finds contents pages and reconstructs the printed-page → PDF-page offset, which drifts across front matter and cannot be assumed constant.header_footer— strips running headers and footers using layout-only heuristics (position and repetition), not a model, so it works on any book.chapter_titles— recovers titles from opener pages and rejects OCR garbage.body_chapters— detects chapter openers directly from body text.classifier— front matter vs body vs back matter.
Two-up scans (one landscape image holding two book pages) are split into portrait pages before any of this runs, so reading order comes out right.
The hard part of this problem is not making one book work — it is making one
book work without breaking the last one. corpus_check.py turns that from a
feeling into a number:
python -m folio.corpus_check # score every book, table + verdict
python -m folio.corpus_check --book janus # one book, with its chapter list
python -m folio.corpus_check --json # machine-readableEach fixture records a real book's cached Marker output and its expected
outcome, and the harness enforces a safety property beyond the happy path: a
book may legitimately degrade to full-text, but it must never ship
confident garbage — no mega-chapter spanning 40%+ of the book, no large
uncovered leading gap, no mostly-generic titles.
corpus/cache/ is gitignored: those caches are OCR of real, copyrighted books.
corpus/fixtures.json (the expectations) and the harness are the shareable
part. See corpus/README.md for populating it with your own
books.
python -m pytest tests -q # 85 testsExtracted from the PDF pipeline of Velo, where it runs in production over user-uploaded scanned books. Its on-device sibling — PDF and EPUB extraction using Apple frameworks only — is Scribe.
Apache-2.0. See LICENSE.