Implement website.llms-txt: organized llms.txt, per-page markdown companions, llms-full.txt - #532
Merged
Merged
Conversation
…site.llms-txt support Symptom confirmed at HEAD (3ac596e): website.llms-txt accepted and silently dropped; no llms.txt, no .llms.md companions. Plan sketches an AST-based design (pampa qmd writer + website post-render assembly) and lists design questions on index organization, llms-full.txt, and naming. Co-Authored-By: Claude Fable 5 <[email protected]>
…ion index) Co-Authored-By: Claude Fable 5 <[email protected]>
…collision safety) Co-Authored-By: Claude Fable 5 <[email protected]>
…cklist llms-full.txt in scope; internal links rewrite to .md siblings; fidelity bar is readable markdown (no Q1 byte-parity); conditional content in PR scope (bd-stbdlesy); code-annotation preservation deferred (bd-to3vh0od, inert until q2 has code annotations). Co-Authored-By: Claude Fable 5 <[email protected]>
…llms-full.txt (bd-llms-txt-unimplemented-oih6z6j7)
Q2's website.llms-txt was accepted and silently dropped. This implements
the full feature, AST-first (no HTML scraping):
- LlmsCaptureTransform (Finalization tail): clones the page AST after
crossref-render, reduces it to a clean markdown view (unwraps section/
float/callout/copy-button chrome, reconstructs callouts, simplifies
footnotes, drops raw HTML, strips quarto-internal attrs, rewrites
same-site links to .md siblings, synthesizes the h1 title), serializes
via pampa's qmd writer, and deposits the string as a path-less
Project-scoped artifact (no RenderOutput change; no wasm ripple).
- Conditional content grows four-quadrant llms semantics: with the llms
view active, when/unless-format is evaluated for both the html target
and the llms view (llms token OR html-family match), one-view-only
content is marker-tagged, and the capture transform resolves the
markers for both views (bd-stbdlesy).
- New ProjectType::post_resources hook, called after the orchestrator's
resource-copy pass; WebsiteProjectType writes all llms artifacts there
so the output ledger sees every other producer first.
- llms_post_render: organized llms.txt (set-subtraction over declared
sidebars -> navbar -> pinned home -> Other; flat sites get ## Pages;
entries '- [title](href): description'), llms-full.txt in reading
order, and the overwrite guarantee: .quarto/llms-manifest.json records
generated paths; writing over anything else fails with new Q-5-28
(catalog entry + docs page included).
- Inert-key warning on non-website projects, next to the aliases one.
Tests: 17 e2e integration tests (llms_txt.rs) incl. both collision
cases, conditional content, drafts/404, incremental subset renders;
6 unit tests (config readers, href/retarget helpers, conditional
quadrants). Full workspace suite (11,949) and full cargo xtask verify
(incl. WASM leg) green.
Snapshot files: 1 added (integration__llms_txt__llms_companion_rich_content.snap)
- new snapshot for companion serialization quality on a content-rich
page; reviewed: clean markdown with reconstructed callout, ^1^
footnote + Footnotes section, minimal '::: {#tbl-nums}' crossref
anchor, resolved 'Table 1' ref text.
Discovered: bd-4vbd3b7g (prefix_caption misses Plain-block captions:
'Table N:' prefix lost in HTML and markdown; pre-existing).
Co-Authored-By: Claude Fable 5 <[email protected]>
docs/guides/projects/llms-txt.qmd (Guides sidebar): configuration, index organization rules, .md link mirroring, when-format="llms" conditional content, draft/404 exclusions, and the Q-5-28 overwrite guarantee. q2 render docs/ green (241/241). Plan records end-to-end evidence: investigation repro inspected; connect-docs port renders 348 companions (exactly Q1's count) with a sidebar-organized llms.txt and llms-full.txt. Co-Authored-By: Claude Fable 5 <[email protected]>
The q2 docs website now ships its own llms.txt (241 entries, organized by the Guides/Authoring/... sidebar sections), per-page .md companions, and llms-full.txt. Also adds a website.description for the index header. Verified with q2 render docs/ (241/241, no new warnings). Co-Authored-By: Claude Fable 5 <[email protected]>
✅ Snyk checks have passed. No issues have been found so far.
💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements
website.llms-txt(bd-llms-txt-unimplemented-oih6z6j7), which q2 previously accepted and silently dropped. Unlike Quarto 1's HTML-scrape-and-pandoc approach, this is AST-first, and the output deliberately improves on Q1's:What
llms-txt: trueproducesabout.html→about.md): the page AST captured at the Finalization tail (crossref numbers resolved), reduced to clean markdown — callouts reconstructed to::: {.callout-note}form, section/float/copy-button chrome unwrapped, footnote plumbing simplified, quarto-internal attributes stripped, same-site links rewritten to.mdsiblings — and serialized with pampa's qmd writer.llms.txt: an organized index per the llms.txt convention — site title,>description, one##section per sidebar section with- [title](href): descriptionentries (set-subtraction: sidebars → navbar → pinned home →## Other; flat sites get## Pages). Q1 emits a bare flat list.llms-full.txt: all companions concatenated in reading order (the de facto ecosystem companion file; Q1 doesn't emit it).Design highlights
<page>.md(ecosystem convention) rather than Q1's.llms.md, gated on an overwrite guarantee: a newProjectType::post_resourceshook runs after the orchestrator's resource-copy pass, and.quarto/llms-manifest.jsonrecords generated paths — writing over anything else fails the render with new Q-5-28 (catalog entry + docs page included), mirroring the alias-collision policy.when-format="llms"/unless-format="llms"work via four-quadrant evaluation insideConditionalContentTransform(the llms view matches thellmstoken or anything the html target matches, since the companion mirrors the html page). One-view-only content is marker-tagged;LlmsCaptureTransformis the sole marker consumer.RenderOutputis untouched. Drafts and the 404 page render normally but get no companion and no index entry. Non-website projects warn that the key is inert. Incremental (subset) renders regeneratellms.txt/llms-full.txtfrom the cached profile index and read skipped pages' companions back from disk.llms-txt: true(last commit) — 241 companions, sidebar-organized index. New user guide atdocs/guides/projects/llms-txt.qmd.Verification
crates/quarto-core/tests/integration/llms_txt.rs, written first and observed red), incl. both collision cases, conditional content, drafts/404, incremental subset renders, and an insta snapshot of a content-rich companion; 8 unit tests.cargo xtask verify(incl. WASM leg) green;cargo xtask lintgreen.llms.txt(absolute site-url links) and clean internal.mdlink rewriting throughout.Follow-up strands filed: bd-4vbd3b7g (pre-existing: "Table N:" caption prefix lost for Plain-block captions in HTML too), bd-6m1iyxl6 (raw HTML/shortcodes in
website.titleleak into the llms.txt H1), bd-5w81o2dh (listing-page companions vs. placeholder envelopes, uncharacterized), bd-to3vh0od (code-annotation preservation, inert until q2 has code annotations).🤖 Generated with Claude Code