Skip to content

docs: llms-full.txt link integrity, MCP entry, and package layering#600

Merged
ShawnChen-Sirius merged 2 commits into
chdb-io:mainfrom
ShawnChen-Sirius:docs/llms-txt-links
Jul 7, 2026
Merged

docs: llms-full.txt link integrity, MCP entry, and package layering#600
ShawnChen-Sirius merged 2 commits into
chdb-io:mainfrom
ShawnChen-Sirius:docs/llms-txt-links

Conversation

@ShawnChen-Sirius

@ShawnChen-Sirius ShawnChen-Sirius commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

Follow-up to #599 (these two commits were pushed after it merged, so they land here).

llms-full.txt: every link is now absolute and alive

A full sweep of all 132 unique link targets found four kinds of broken links, all fixed in the generator:

  • Site-absolute doc paths (/interfaces/formats) now carry the clickhouse.com/docs prefix.
  • File-relative links (guides/querying-pandas.md) resolve through a source-path → published-slug map, because filenames and slugs diverge (that file publishes at /chdb/guides/pandas).
  • Repo-relative links/images in ARCHITECTURE.md and the binding READMEs point at github.com blob / raw URLs.
  • YouTube iframes become plain [Watch the video](watch URL) links; dangling <Image> tags (their MDX imports are stripped) are removed; file:/data: connection-string examples are left untouched.

Two targets were verified genuinely dead and are unwrapped to plain text via a DEAD_LINK_PREFIXES list: the JeffSackmann/tennis_atp dataset repo (404) and jupysql.ploomber.io (domain no longer resolves). The six 403 targets (npmjs, crates.io, clickpy, badge.fury.io) are bot walls — packages verified alive via the npm registry and crates.io APIs.

Result: 130 unique links in the regenerated file, zero dead.

Known upstream issue, out of scope: the JupySQL guide's code sample still downloads from the removed tennis_atp repo, so that tutorial is broken in clickhouse-docs itself.

llms.txt: two additions

  • MCP entry: chDB is served through the official mcp-clickhouse server — pip install 'mcp-clickhouse[chdb]', CHDB_ENABLED=true, agents get a run_chdb_select_query tool.
  • Package layering note in the header: chdb-core builds the libchdb engine binaries; chdb (Python) and the node/bun/go/rust packages are thin bindings over the same C ABI — plus routing guidance for where issues belong.

CI: dead-link gate now covers both files

The link check walks llms.txt and llms-full.txt (markdown targets plus **URL:** provenance lines), and treats unreachable-after-retries (000) as dead — a vanished domain never returns 404. 403/429 stay warning-only.

🤖 Generated with Claude Code

Note

Fix link integrity and add MCP entry in llms.txt and llms-full.txt

  • Updates scripts/generate_llms_full.py to rewrite relative docs links to absolute https://clickhouse.com/docs/... URLs, convert iframe embeds to YouTube watch links, remove <Image /> tags, and unwrap known-dead external links to plain text.
  • Adds a two-pass slug map in the generator so intra-doc links are resolved before emission; GitHub-sourced relative links are rewritten to github.com blob or raw.githubusercontent.com URLs depending on whether they point to images.
  • Updates .github/workflows/llms-txt-check.yml to validate links in both llms.txt and llms-full.txt, treat HTTP 000 as dead alongside 404/410, and log 403/429 as warnings.
  • Adds a 'How the packages relate' bullet (chdb-core vs language bindings) and an MCP server entry to llms.txt.

Macroscope summarized 3fcefde.

ShawnChen-Sirius and others added 2 commits July 7, 2026 17:37
…ge layering

llms-full.txt / generator:
- Rewrite relative link targets to absolute URLs: site-absolute doc
  paths get the clickhouse.com/docs prefix, file-relative targets
  resolve through a source-path -> published-slug map (filenames and
  slugs diverge, e.g. guides/querying-pandas.md publishes at
  /chdb/guides/pandas), and repo-relative targets in ARCHITECTURE.md
  and binding READMEs point at github.com blob / raw URLs.
- Replace YouTube iframe embeds with plain watch links and drop
  <Image> tags whose imports were already stripped.
- Leave file:/data: connection-string examples untouched.
- Verified: all 59 clickhouse.com/docs links and all GitHub links in
  the regenerated file resolve (one upstream-content exception:
  JeffSackmann/tennis_atp, a third-party dataset repo referenced by
  the JupySQL guide, is gone from GitHub).

llms.txt:
- Add MCP entry: chDB is served through the official mcp-clickhouse
  server ('pip install mcp-clickhouse[chdb]', CHDB_ENABLED=true,
  run_chdb_select_query tool).
- Explain package layering: chdb-core builds the libchdb engine
  binaries; chdb (Python) and the node/bun/go/rust packages are thin
  bindings over the same C ABI.

Co-Authored-By: Claude Fable 5 <[email protected]>
- Full sweep of all 132 unique link targets in llms-full.txt found two
  genuinely dead ones: the JeffSackmann/tennis_atp dataset repo (404)
  and jupysql.ploomber.io (domain no longer resolves). The generator
  now unwraps links matching DEAD_LINK_PREFIXES to their anchor text;
  both are removed from the regenerated file.
- The six 403 targets (npmjs, crates.io, clickpy, badge.fury.io) are
  bot walls, not dead pages — the packages were verified alive via the
  npm registry and crates.io APIs — so 403 stays warning-only.
- The CI link check now covers llms-full.txt as well as llms.txt
  (markdown targets plus '**URL:**' provenance lines) and treats
  unreachable-after-retries (000) as dead, since a vanished domain
  never returns 404.

Known upstream content issue, out of scope here: the JupySQL guide's
code sample still downloads from the removed tennis_atp repo, so the
tutorial itself is broken in clickhouse-docs and needs a new dataset.

Co-Authored-By: Claude Fable 5 <[email protected]>
@ShawnChen-Sirius ShawnChen-Sirius merged commit 00f2d38 into chdb-io:main Jul 7, 2026
11 of 13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant