ADFA-4739: Pipeline for producing template-based Kotlin documentation

.gitignore

@@ -12,3 +12,4 @@ __pycache__/
 *$py.class
 *.db
 *.sqlite
+run_e2e_pipeline_test.local.sh

ProcessDocs/ProcessKotlinDocs/ProcessKotlinWebsiteJSON/README.md

@@ -0,0 +1,148 @@
+# Process Kotlin Website JSON
+
+Scripts for converting a `kotlin-web-site/docs` checkout (JetBrains
+Writerside-flavored Markdown) into the JSON block schema this project's
+templating engine renders, and for loading that JSON, its navigation tree,
+and its media straight into a `documentation.db`-schema SQLite database.
+
+## Scripts
+
+| Script | Purpose |
+|---|---|
+| [`md_to_json.py`](md_to_json.py) | Converts every `topics/**/*.md` page into one JSON file (see schema below). Writes `theme.json` and copies `images/` into the output directory. |
+| [`build_nav.py`](build_nav.py) | Builds `nav.json`/`nav.html` sidebar navigation from `kr.tree`, resolving each `<toc-element topic="...">` against `md_to_json.py`'s output. |
+| [`find_missing_assets.py`](find_missing_assets.py) | QA pass: reports cross-page links, images, and `<include>` targets in the source tree that don't resolve to anything. Reuses `md_to_json.py`'s own resolution logic, so it flags exactly what would end up broken on the rendered site. |
+| [`populate_db.py`](populate_db.py) | The database path: converts the docs tree the same way `md_to_json.py` does, builds nav the same way `build_nav.py` does, and inserts pages + nav + images + CSS/JS directly into `documentation.db` (replacing everything under `k/html/` and `assets/`). Supports pruning whole `kr.tree` subtrees via `--blacklisted-element-titles`. |
+| [`optimize_media.py`](optimize_media.py) | Standalone media optimizer: downscales/recompresses a directory of images (pngquant, Pillow, Scour/cairosvg for SVG) into a mirrored output directory. |
+| [`insert_optimized_media.py`](insert_optimized_media.py) | Runs `optimize_media.py`'s pipeline over a directory of raw media, then replaces the corresponding `k/html/images/*` rows in an existing database, rewriting any page that referenced a renamed file and deleting anything left unreferenced. |
+
+## Requirements
+
+- Python 3.10+
+- `pip install markdown-it-py Pillow scour brotli`
+- `cairosvg` (only needed if an optimized SVG exceeds `--svg-rasterize-threshold`): `pip install cairosvg`
+- `pngquant` on `PATH` (e.g. `apt install pngquant`) — required by `optimize_media.py`/`insert_optimized_media.py`, and by `populate_db.py` for the images it inserts directly from the Writerside export.
+
+`populate_db.py` also expects, relative to its own location, and already
+included in this directory:
+
+- `templates/page.peb`, `templates/nav.peb` — Pebble templates upserted into the `Templates` table.
+- `assets/docs.css`, `assets/tabs.js`, `assets/sidebar.js` — static assets inserted at `assets/<name>`.
+
+## Inputs you need before starting
+
+- A checkout of `kotlin-web-site/docs` (the `<docs-root>` argument below) — contains `topics/`, `images/`, `v.list`, and `kr.tree`.
+- A config JSON with theming colors, e.g.:
+  ```json
+  {"broken-ext-link-color": "#cc0000", "menu-no-link-color": "#999999"}
+  ```
+- Writerside's own image export zip (e.g. `webHelpImages.zip`, found next to `kr.tree`) if you're using `populate_db.py`.
+
+## Workflow: generate JSON + nav for a static/templated preview
+
+Use this to produce standalone JSON pages and nav data (not the database)
+for local inspection or a different renderer.
+
+```bash
+# 1. Convert every topic .md into JSON, one file per page
+python3 md_to_json.py <docs-root> <output-dir> config.json
+
+# 2. Build the sidebar nav from kr.tree against that JSON output
+python3 build_nav.py <docs-root> <output-dir> <output-dir>
+
+# 3. (optional) Check for broken links/images/includes in the source tree
+python3 find_missing_assets.py <docs-root> missing-assets-report.md
+```
+
+`<output-dir>` ends up containing:
+- `topics/**/*.json` — one page per source `.md` file (schema below)
+- `theme.json` — the two theming colors, carried from `config.json`
+- `images/` — copied straight from `<docs-root>/images/`
+- `nav.json` / `nav.html` — sidebar tree and a pre-rendered static copy
+
+### Page JSON schema
+
+```json
+{
+  "id": "enum-classes",
+  "sourceFile": "topics/enum-classes.md",
+  "title": "Enum classes",
+  "blocks": [ { "type": "heading", "level": 2, "id": "...", "html": "..." }, "..." ]
+}
+```
+
+Block types: `heading`, `paragraph`, `code`, `blockquote`, `list`, `table`,
+`image`, `hr`, `tabs`, `note`/`tip`/`warning`, `html` (raw passthrough). See
+the module docstring in [`md_to_json.py`](md_to_json.py) for full shapes and
+known limitations (nested tabs, `<include>` resolution, variable
+substitution).
+
+## Workflow: generate + insert directly into the documentation database
+
+This is the path that actually populates `documentation.db`. It performs
+the same conversion as `md_to_json.py`/`build_nav.py` internally — you don't
+run those scripts first.
+
+```bash
+python3 populate_db.py <docs-root> config.json <webHelpImages.zip> [db-path]
+```
+
+- `db-path` defaults to `documentation.db` in the current directory, and must already exist with the expected schema (`Languages`, `ContentTypes`, `Templates` tables populated).
+- A timestamped backup (`<db-path>.backup-<timestamp>`) is written before any changes, via SQLite's `VACUUM INTO`.
+- Everything under `k/html/` and `assets/` is deleted and re-inserted in a single transaction (rolled back on error), then the database is `VACUUM`ed.
+
+### Pruning documentation you don't want (ADFA-4737)
+
+To leave a whole `kr.tree` subtree out of the database entirely — nav
+entry, converted pages, and all — pass `--blacklisted-element-titles` with
+the full `toc-title` path from a top-level element down to the one you want
+to drop. Levels are joined with `\/` (backslash-slash), not a bare `/`,
+since a bare `/` commonly appears inside a real title. The example below is
+illustrative only — open `<docs-root>/kr.tree` and copy the actual
+`toc-title` chain for whatever section you're dropping (e.g. Kotlin/Wasm):
+
+```bash
+python3 populate_db.py <docs-root> config.json <webHelpImages.zip> documentation.db \
+  --blacklisted-element-titles \
+    "<Top-Level Title>\/<Nested Title>"
+```
+
+Any other page's in-content link to a pruned topic renders as a styled
+"broken" link (via `broken-ext-link-color`) rather than a dead link with no
+indication anything changed. Run with `--blacklisted-element-titles` first
+against a scratch copy of the database and check the warnings on stderr for
+any path that didn't match — that usually means the toc-title or ancestor
+chain was copied wrong.
+
+## Workflow: optimizing and inserting media
+
+Two options, depending on whether the database already has pages loaded:
+
+**Standalone optimization only** (no database involved):
+
+```bash
+python3 optimize_media.py <input-dir> <output-dir> [--max-width 500] [--webp] [...]
+```
+
+**Optimize and update an existing database's images in place:**
+
+```bash
+python3 insert_optimized_media.py <media-dir> <db-path> [work-dir] [options]
+```
+
+This re-runs `optimize_media.py`'s pipeline, backs up the database first,
+replaces each `k/html/images/<name>` row with the optimized bytes, rewrites
+any page/nav reference to a file that got renamed during optimization (e.g.
+`--webp` conversion or SVG rasterization), and deletes any image no page
+references anymore. Both scripts share the same tuning flags
+(`--max-width`, `--jpeg-quality`, `--webp`, `--webp-quality`,
+`--pngquant-speed`, `--svg-precision`, `--svg-rasterize-threshold`,
+`--verbose`, `--log-file`), settable via `--config <file>` instead of the
+command line — see either script's module docstring for the full option
+reference.
+
+## Recommended order for a full refresh
+
+1. `find_missing_assets.py` against the new `<docs-root>` — fix anything broken in the source before converting it.
+2. `populate_db.py`, with `--blacklisted-element-titles` for anything you don't want documented (e.g. Kotlin/Wasm per ADFA-4737).
+3. `insert_optimized_media.py` against the raw media directory, if you want optimized (resized/compressed) images rather than Writerside's own export as-is.

ProcessDocs/ProcessKotlinDocs/ProcessKotlinWebsiteJSON/assets/docs.css

@@ -0,0 +1,227 @@
+html, body {
+  background: #ffffff;
+  font-family: Arial, Helvetica, sans-serif;
+}
+
+.docs-layout {
+  display: flex;
+  align-items: flex-start;
+}
+
+.docs-sidebar {
+  width: 280px;
+  flex-shrink: 0;
+  box-sizing: border-box;
+  overflow-y: auto;
+  max-height: 100vh;
+  position: sticky;
+  top: 0;
+}
+
+.docs-content {
+  flex: 1;
+  min-width: 0;
+  padding: 0 24px;
+}
+
+/* Source images/videos carry explicit width="..." attributes (from
+   Writerside's `{width="800"}` sizing hints, see md_to_json.py), which is an
+   intrinsic pixel size the img/video would otherwise render at even when
+   that's wider than the viewport. max-width: 100% lets it shrink to fit
+   instead of overflowing on narrow screens, while height: auto keeps its
+   aspect ratio as it scales down. */
+.docs-content img,
+.docs-content video,
+.docs-content iframe {
+  max-width: 100%;
+  height: auto;
+}
+
+/* Long code lines can't shrink like an image can without breaking the code's
+   formatting, so let the block itself scroll horizontally instead - without
+   this the unconstrained width pushes out past the viewport and the whole
+   page grows a horizontal scrollbar on narrow screens. */
+.docs-content pre.code-block {
+  overflow-x: auto;
+  box-sizing: border-box;
+  max-width: 100%;
+}
+
+/* Sidebar tree: <ul>/<li> are only used for their document structure here,
+   not as a bulleted list, so strip the browser's default marker/indent/
+   margin on every level (root .nav-tree and each nested .nav-subtree share
+   the class) and build indentation and disclosure icons ourselves below. */
+.docs-nav {
+  font-size: 14px;
+  line-height: 1.4;
+  color: #333333;
+}
+
+.docs-nav ul {
+  list-style: none;
+  margin: 0;
+  padding: 0;
+}
+
+.docs-nav .nav-subtree {
+  padding-left: 14px;
+}
+
+/* Per-topic collapse/expand (nav.peb: every node with children renders a
+   sibling .nav-toggle-group button next to its label/link, leaf nodes render
+   a same-sized .nav-spacer instead so labels still line up in a column).
+   Collapsed is the CSS default so the tree stays usable without JS;
+   assets/sidebar.js toggles .nav-expanded on click and auto-expands the
+   current page's ancestors. */
+.nav-row {
+  display: flex;
+  align-items: center;
+  gap: 4px;
+  padding: 4px 8px;
+  border-radius: 4px;
+  cursor: default;
+}
+
+.nav-row:hover {
+  background: #f0f1f2;
+}
+
+.nav-item > .nav-subtree {
+  display: none;
+}
+
+.nav-item.nav-expanded > .nav-subtree {
+  display: block;
+}
+
+.nav-toggle-group,
+.nav-spacer {
+  width: 14px;
+  height: 14px;
+  flex-shrink: 0;
+}
+
+.nav-toggle-group {
+  display: flex;
+  align-items: center;
+  justify-content: center;
+  background: none;
+  border: none;
+  padding: 0;
+  cursor: pointer;
+  color: #666666;
+}
+
+.nav-toggle-group::before {
+  content: "\25B8"; /* ▸ */
+  display: inline-block;
+  font-size: 10px;
+  transition: transform 0.15s ease;
+}
+
+.nav-item.nav-expanded > .nav-row > .nav-toggle-group::before {
+  transform: rotate(90deg);
+}
+
+.nav-link,
+.nav-group-title {
+  flex: 1;
+  min-width: 0;
+  padding: 2px 0;
+  color: inherit;
+  text-decoration: none;
+}
+
+.nav-link:hover {
+  text-decoration: underline;
+}
+
+.nav-group-title {
+  cursor: default;
+}
+
+.nav-link--active {
+  font-weight: 700;
+  color: #0b57d0;
+}
+
+.nav-item--current > .nav-row {
+  background: #e8f0fe;
+  border-left: 3px solid #0b57d0;
+  padding-left: 5px; /* 8px base padding - 3px border, so text doesn't shift */
+}
+
+/* Mobile off-canvas drawer (assets/sidebar.js: hamburger button, backdrop,
+   and edge-swipe open/close all toggle the classes below). Above the
+   breakpoint the sidebar stays a normal static column and these are inert. */
+.nav-toggle {
+  display: none;
+}
+
+.nav-backdrop {
+  display: none;
+}
+
+@media (max-width: 900px) {
+  .docs-layout {
+    display: block;
+  }
+
+  .nav-toggle {
+    display: block;
+    position: fixed;
+    top: 10px;
+    left: 10px;
+    z-index: 1001;
+    width: 40px;
+    height: 40px;
+    border: 1px solid #ccc;
+    border-radius: 4px;
+    background: #ffffff;
+  }
+
+  .nav-toggle::before {
+    content: "\2630"; /* ☰ */
+    font-size: 1.3em;
+  }
+
+  .docs-sidebar {
+    position: fixed;
+    top: 0;
+    left: 0;
+    height: 100%;
+    max-height: 100%;
+    width: 85vw;
+    max-width: 320px;
+    background: #ffffff;
+    z-index: 1000;
+    box-shadow: 2px 0 12px rgba(0, 0, 0, 0.25);
+    transform: translateX(-100%);
+    transition: transform 0.25s ease;
+    padding-top: 56px;
+  }
+
+  .docs-sidebar.sidebar-open {
+    transform: translateX(0);
+  }
+
+  .nav-backdrop {
+    display: block;
+    position: fixed;
+    inset: 0;
+    background: rgba(0, 0, 0, 0.4);
+    opacity: 0;
+    pointer-events: none;
+    transition: opacity 0.25s ease;
+    z-index: 999;
+  }
+
+  .nav-backdrop--visible {
+    opacity: 1;
+    pointer-events: auto;
+  }
+
+  .docs-content {
+    padding-top: 56px;
+  }
+}