Skip to content

Repair the rest of what Google's export does to frontmatter - #1045

Open
henry-dowling wants to merge 2 commits into
feat/skill-address-by-drive-filefrom
fix/docs-export-frontmatter
Open

Repair the rest of what Google's export does to frontmatter#1045
henry-dowling wants to merge 2 commits into
feat/skill-address-by-drive-filefrom
fix/docs-export-frontmatter

Conversation

@henry-dowling

@henry-dowling henry-dowling commented Aug 12, 2026

Copy link
Copy Markdown
Collaborator

our google drive to skills conversion uses the google drive "export to markdown" feature to convert docs to skills, which are markdown files by default. Unfortunately, this export mangles a bunch of things about the doc in the conversion, so we need to manually fix them. This is a surgical fix, we only go ahead with this manual un-mangling when the result will be a valid skill (we think in this case, it's clearly a good thing for us to do this). We also include a migration to repair already-broken already-imported markdown files from drive.

Thanks to @mholkesvik for the find

slop

Stacked on #1043. The escaped --- that #1040 repairs was the fatal shape, but not the only one — a skill authored in a Doc is prose on the way in and markdown on the way out, and the exporter marks up characters the author never typed.

Four shapes, all invisible in the editor, found by running real export shapes through the extractor and the skill parser:

What the author sees in the Doc What the export stores Before this PR
name: "Brake Shoes" (typed, so Docs curls it) name: “Brake Shoes” parses, and the skill is named “Brake Shoes” — quotes included
description: "...part_number..." ...part\_number... \_ is not a valid JSON escape, so json.loads throws, declared_skill swallows the ValueError, and the document silently isn't a skill
name: styled bold **name:** partitions to a key nothing is looking for
an empty first paragraph above the block a leading blank line fatal twice: the block is no longer at the top, and list_source_skills filters on content LIKE '---%' before anything parses

Repaired at extraction, beside the rule fix and for the same reason given there: the markup belongs to the exporter, not the author or the format. Escaped rules are still repaired anywhere in the document; the rest only between the delimiters, because frontmatter is the one region with no markup in it — a \_ or a ** below the block is the author writing markdown and stays. Only the wrapping quote pair is straightened, so a curly apostrophe mid-sentence is left as their text.

Context: this is the failure mode Heavi flagged in Slack ("weird Google Docs formatting stuff that messed up frontmatter"), and the curly-quote row is the likeliest one to bite them — the example we handed them is quoted, and Docs curls a typed quote by default.

Verification

  • 6 new tests in test_drive_extraction_pipeline.py, including one asserting the body below the block is not rewritten (that text is what the agent reads).
  • pytest backend/tests/test_drive_extraction_pipeline.py — 31 passed. With test_source_backed_skills, test_skill_metadata, test_every_skill_creation_path, test_folder_skills, test_skill_agent_read, test_github_skill_import: 86 passed, 0 failed.
  • ruff check backend/ cli/ and ruff format --check backend/ cli/ clean.

Not covered

  • No live Drive round-trip. Every shape here is constructed, not observed coming back from the API — unlike the \--- fix, which was verified end to end against a real Doc. Worth one live check before this ships.
  • A UTF-8 BOM is not handled. If Drive's export ever emits one it fails the same LIKE '---%' prefilter as the blank-line case. Left alone rather than guessed at.

🤖 Generated with Claude Code


Note

Medium Risk
A one-time migration rewrites user document content in drive_documents with strict guards, but incorrect filtering could corrupt stored text; extraction changes what gets stored for all Google Doc exports going forward.

Overview
Extends Google Doc → markdown extraction so skills authored in Docs can declare valid frontmatter after export. unescape_exported_rules is replaced by repair_exported_frontmatter, which still fixes escaped --- rules but also normalizes delimiter blocks (BOM/leading blanks, --- normalization), strips exporter emphasis from keys, unescapes CommonMark punctuation in values, and re-encodes curly-wrapped quotes as JSON strings. Repairs apply only inside a detected frontmatter block and the result is kept only when declared_skill succeeds, so ordinary docs that open with horizontal rules stay unchanged; markdown below the closing delimiter is not rewritten.

Migration 0186 backfills existing drive_documents.content with a frozen copy of that logic, updating rows only when the repair newly yields a valid skill or fixes curly-quoted skill metadata, and marks affected rows embed_stale. Integration tests cover export shapes; an isolated DB test asserts the backfill repairs damaged exports without touching prose or clean hand-authored SKILL.md files.

Reviewed by Cursor Bugbot for commit 3886660. Bugbot is set up for automated code reviews on this repo. Configure here.

The escaped `---` was the fatal one, but not the only one. A skill authored in
a Doc is prose on the way in and markdown on the way out, and the exporter
marks up characters the author never typed. Four shapes, all invisible in the
editor:

  - Docs curls a typed quote, so `name: "Brake Shoes"` — the example we handed
    Heavi in Slack — comes back as `name: “Brake Shoes”` and names the skill
    with the quotes still in it.
  - An underscore exports as `\_`, which is not a valid JSON escape, so a
    description mentioning a part_number cost the document its whole skill
    status. Silently: it just stopped appearing.
  - A key the author styled exports as `**name:**`, which partitions to a key
    nothing is looking for.
  - An empty first paragraph exports as a leading blank line, which is fatal
    twice — the block is no longer at the top, and the listing query filters on
    `content LIKE '---%'` before anything gets to parse it.

Repaired at extraction, beside the rule fix and for the same reason: the markup
is the exporter's, not the author's. Escaped rules are still repaired anywhere
in the document, but the rest only between the delimiters — frontmatter is the
one region with no markup in it, so a `\_` or a `**` below the block is the
author writing markdown and stays. Only the wrapping quote pair is straightened;
a curly apostrophe mid-sentence is their text.

Verified by parsing each shape through the real extractor and skill parser:
all eight now declare, with the name and description the author typed.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
@assert-app

assert-app Bot commented Aug 12, 2026

Copy link
Copy Markdown

Review on Assert →

3 clusters identified

@vercel

vercel Bot commented Aug 12, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
stash-web Ready Ready Preview Aug 18, 2026 8:57am

Request Review

@cursor

cursor Bot commented Aug 18, 2026

Copy link
Copy Markdown

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_bef3e14b-8dd6-4a0f-96b9-f52b69a2e06f)

The frontmatter repair rewrote anything between two dash-lines, gated only
on "the first line is a rule" — which mangled ordinary prose in any Doc
that happens to open with a divider, in every connected source. The rewrite
is now kept only when the result declares a valid skill; anything else is
returned byte-identical.

Also closed while in there:
- json.dumps for quoted values (an interior quote or backslash used to kill
  the skill or corrupt the value), and single curly quotes straighten too.
- The full CommonMark escapable-punctuation set is unescaped, not a
  hand-picked subset.
- Italic strips like bold; an empty first paragraph with hard-break spaces
  and a leading BOM no longer hide the block.
- Delimiters normalize to exactly `---`, so the repaired span and the span
  parse_frontmatter reads are the same span.
- The scan is bounded to FRONTMATTER_SCAN_BYTES instead of splitting a 4 MB
  document into a line list.
- Migration 0186 backfills already-extracted rows (re-extraction is keyed
  on Drive's modifiedTime, so damaged rows never heal on their own). It
  rewrites a row only when repair turns it into a skill it wasn't, or when
  its declared values are wrapped in Docs' curly quotes; a hand-authored
  SKILL.md stays exactly as written. Repaired rows are marked embed_stale.

Verified: 39 tests in test_drive_extraction_pipeline.py (8 new), a
data-level migration test on an isolated database, and the skill/migration
suites (source_backed_skills, skill_metadata, every_skill_creation_path,
folder_skills, skill_agent_read, github_skill_import, migrations,
migration_chain, migration_placeholders) all passing locally; ruff clean.
Still not covered: a live Drive round-trip, unchanged from the PR note.

Co-Authored-By: Claude Fable 5 <[email protected]>
@cursor

cursor Bot commented Aug 18, 2026

Copy link
Copy Markdown

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_bb07c314-589b-4066-a264-1fc29a7fa9f1)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant