fix(seo): bot snapshots had no h1, no internal links, leaked canonicals - #29
Merged
Merged
Conversation
…ed canonicals
Googlebot, Bingbot, GPTBot, ClaudeBot and PerplexityBot do not read the SPA.
cloudflare-worker.js routes any bot UA sending `Accept: text/html` to the
seo-snapshot edge function, so its output IS the indexed page.
Two generators were building that output independently:
- regenerate-snapshot (publish path) -> header, h1, author meta, footer
- seo-snapshot's inline self-heal -> <body><article>{content}</article>
The self-heal version had no <h1>, no internal links and only a minimal
Article schema. Because it UPLOADS what it generates into the seo-snapshots
bucket, and the bucket is served first from then on, every self-heal
permanently replaced a good snapshot with a broken one.
Measured on the live site before this fix (111 sitemap URLs, parsed with a
real browser engine, JS disabled):
- 77 of 111 URLs served zero <h1>
- median internal links per article: 0
- GSC: rich_results null site-wide; defi-marketing-team-metrics regressed
from indexed to "Crawled - currently not indexed"
Separately, several articles are authored in the CMS as complete standalone
HTML documents. Embedding that inside <article> leaked a second <title> and a
second <link rel="canonical">. On the BR and ES copies of
web2-vs-web3-marketing the leaked canonical pointed at
/publications/web2-vs-web3-marketing-br, which GSC classifies as a Soft 404:
the article was telling Google its authoritative version was a dead URL.
Changes:
- _shared/snapshot-html.ts is now the only builder of snapshot HTML, used by
both functions, so the two paths cannot drift again.
- sanitizePublicationHtml strips nested DOCTYPE/html/head/body plus any
stray title/link/meta, so full-document content can never leak head tags.
- Snapshots now emit an h1 (when the content lacks one), a nav with crawl
paths to the hub, about, tools and the audit page, hreflang siblings, a
footer link, and BreadcrumbList alongside Article schema.
- validateSnapshot() gates every upload in both functions: exactly one
title, exactly one canonical matching the expected URL, an h1, at least
three internal links, no nested document scaffolding, JSON-LD present,
150+ words. A failing snapshot is refused, not stored.
- seo-snapshot also validates on storage READ, so objects poisoned by the
old code are detected and rebuilt in place instead of being served
forever.
Verified: deno check passes on all three files; the builder was run against
the real CMS content for all three locales of web2-vs-web3-marketing
(1 title, 1 correct canonical, h1 present, 4 internal links, validation PASS)
and the old lean template is correctly rejected by the guard.
Co-Authored-By: Claude Fable 5 <[email protected]>
✅ Deploy Preview for mangabeira ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: Repository: gogrowth-co/coderabbit/.coderabbit.yaml Review profile: ASSERTIVE Plan: Pro Plus Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
gogrowth-co
added a commit
that referenced
this pull request
Aug 23, 2026
fix(worker): repair bot-facing snapshot HTML at the edge (temporary, until #29 deploys)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What Googlebot was actually reading
cloudflare-worker.jsroutes any bot UA sendingAccept: text/htmlto theseo-snapshotedge function. Its output is the indexed page. Humans never see it, so this was invisible in the browser.Two generators built that output independently, and they drifted:
regenerate-snapshot(on publish)seo-snapshotself-heal fallback<body><article>{content}</article></body>The fallback had no
<h1>, no internal links, and only a minimal Article schema. It uploads what it generates into theseo-snapshotsbucket, and the bucket is served first from then on — so every self-heal permanently replaced a good snapshot with a broken one.Measured on the live site before this fix
111 sitemap URLs, parsed with a real browser engine, JS disabled:
<h1>rich_resultsnull site-wide;defi-marketing-team-metricsregressed from indexed to Crawled – currently not indexedThe canonical leak
Several articles are authored in the CMS as complete standalone HTML documents. Embedded inside
<article>, that leaked a second<title>and a second<link rel="canonical">. On the BR and ES copies ofweb2-vs-web3-marketingthe leaked canonical pointed at/publications/web2-vs-web3-marketing-br— which GSC classifies as a Soft 404. The article was telling Google its authoritative version was a dead URL.Changes
_shared/snapshot-html.tsis now the only builder of snapshot HTML, used by both functions, so the paths cannot drift again.sanitizePublicationHtmlstrips nestedDOCTYPE/html/head/bodyand any straytitle/link/meta.h1(when content lacks one), a nav with crawl paths, hreflang siblings, a footer link, andBreadcrumbListalongsideArticle.validateSnapshot()gates every upload: exactly one title, one canonical matching the expected URL, an h1, ≥3 internal links, no nested scaffolding, JSON-LD present, 150+ words. Failures are refused, not stored.seo-snapshotvalidates on storage read too, so objects poisoned by the old code are rebuilt in place rather than served forever.Verification
deno checkpasses on all three files.web2-vs-web3-marketing: 1 title, 1 correct canonical, h1 present, 4 internal links, validation PASS.Deploy
Edge functions here deploy only via Lovable Cloud (see
DEPLOY.md). After merge, trigger the deploy, then regenerate snapshots with{all:true}.🤖 Generated with Claude Code