docs: llms.txt fixes, link-check CI, and generated llms-full.txt

[ShawnChen-Sirius]

What

Three follow-ups to the llms.txt added in #598:

1. llms.txt content fixes

• The Migrate from DuckDB entry pointed at a docs URL that is not published yet (returned 404). It now points at the runnable cookbook — migration-from-duckdb — with the static API analyzer and the 18-query benchmark.
• New Examples entry for otel-ingestion-buffer: a Node.js service moves OTEL/LLM-trace data from S3 into ClickHouse with a single SQL statement, without rows ever passing through the JS heap.
• New agent hint in the header: pages under clickhouse.com/docs serve clean Markdown via Accept: text/markdown, so agents can skip HTML parsing.

2. CI: .github/workflows/llms-txt-check.yml

Runs on PRs touching the llms files and weekly:

• Link check (hard gate): every link in llms.txt must resolve; 404/410 fail the job (with retries; 403/429 rate limits only warn).
• Coverage diff (informational): compares the curated links against the chDB entries in the auto-generated clickhouse.com/docs/llms.txt and writes both directions to the job summary — published docs pages missing from the curated index (curation candidates), and curated links missing from the docs index (rename/unpublish warnings).

3. llms-full.txt + generator

scripts/generate_llms_full.py (stdlib only) builds the full-corpus companion file that the llms.txt convention pairs with an index:

• all 44 docs/chdb pages from ClickHouse/clickhouse-docs, ordered getting-started → install → api → datastore → guides → configuration → debugging → reference,
• plus docs/ARCHITECTURE.md and the Node.js / Bun / Go / Rust binding READMEs,
• each page block carries a **URL:** provenance line so RAG chunks keep their source.

The generated llms-full.txt is committed: ~500 KB ≈ 125k tokens, so the whole chDB corpus fits in a single model context window.

Verification

• All 27 links in llms.txt checked: every one returns HTTP 200.
• Generator run end-to-end; output spot-checked for leftover MDX imports (0) and heading anchors (0).

🤖 Generated with Claude Code

Note

Add llms-full.txt generation script, CI link-check workflow, and llms.txt fixes

• Adds scripts/generate_llms_full.py, a CLI script that fetches chDB docs from the ClickHouse/clickhouse-docs GitHub API, cleans Markdown bodies, and writes a concatenated corpus to llms-full.txt with deterministic section ordering.
• Adds a CI workflow that fails if any link in llms.txt returns HTTP 404/410 after retries, and posts a coverage diff between curated links and the live docs index to the job summary without affecting pass/fail.
• Updates llms.txt with a new OTEL cookbook link, a revised DuckDB migration entry pointing to a cookbook path, and an agent note recommending Accept: text/markdown for docs pages.

^{Macroscope summarized 64e6ba6.}