docusaurus-plugin-smart-404

Docs sites get hammered with AI-hallucinated and near-miss URLs — /docs/instalation, /docs/api/getting-started.md, links an LLM confidently made up. This package turns those dead ends into useful answers:

1. Build time — a Docusaurus plugin emits route-manifest.json (every final route, with titles) into your build output.
2. Edge time — a runtime-agnostic handler fuzzy-matches the missed path against the manifest and answers with a confidence-gated redirect, a suggestion page, or a plain 404 — with content negotiation so AI agents get markdown and humans get HTML.

Zero runtime dependencies. Thin adapters for Vercel middleware, Cloudflare Workers, and Netlify Edge Functions.

Install

npm install docusaurus-plugin-smart-404
# or
bun add docusaurus-plugin-smart-404

Quickstart

1. Emit the manifest (Docusaurus)

// docusaurus.config.js
export default {
  plugins: [
    [
      'docusaurus-plugin-smart-404',
      {
        // all options optional
        filename: 'route-manifest.json',
        exclude: ['/search', '/tags/**'],
        emitRedirectsFile: true,
        aliases: {
          '/old-docs/intro': '/docs/intro',
        },
      },
    ],
  ],
};

After docusaurus build, your build/ directory contains route-manifest.json:

{
  "version": 1,
  "baseUrl": "/",
  "routes": [
    { "path": "/docs/installation", "title": "Installation" },
    { "path": "/docs/configuration", "title": "Configuration" }
  ]
}

With emitRedirectsFile: true, a Netlify/Cloudflare-Pages _redirects file is also written from your curated aliases map (exact matches, from to 301 format). If your site already ships a _redirects (e.g. from static/_redirects), the alias rules are appended to it — your existing rules are never overwritten. Aliases are for redirects you know; the fuzzy matcher handles the ones you can't predict.

2. Handle misses at the edge

Load the manifest (import it, inline it, or fetch it once at cold start) and wire up the adapter for your host.

Vercel (middleware)

// middleware.ts
import { createSmart404Middleware } from 'docusaurus-plugin-smart-404/edge/vercel';
import manifest from './build/route-manifest.json'; // exists after `docusaurus build`

const smart404 = createSmart404Middleware(manifest);

export default async function middleware(request: Request) {
  return smart404(request); // Response to short-circuit, undefined to fall through
}

// scope it so the matcher only sees docs paths, not assets or APIs
export const config = { matcher: ['/docs/:path*', '/blog/:path*'] };

The middleware falls through (undefined) for non-GET/HEAD requests, for paths that exist in the manifest, and for anything the matcher isn't confident about.

Warning: unlike the Cloudflare/Netlify adapters, Vercel middleware runs before the origin — it can't see whether a page really 404s. Any real URL inside the middleware matcher scope that is absent from the manifest (routes removed with the plugin's exclude option, static files under the docs path) will be hijacked into a redirect or suggestion page. Keep exclude patterns and static assets disjoint from the middleware matcher scope.

Cloudflare Workers

import { withSmart404 } from 'docusaurus-plugin-smart-404/edge/cloudflare';
import manifest from './route-manifest.json';

export default {
  async fetch(request, env) {
    const handler = withSmart404((r) => env.ASSETS.fetch(r), manifest);
    return handler(request);
  },
};

The wrapper fetches the origin first and only runs the matcher when the origin answered 404 — real pages are never touched.

Netlify Edge Functions

// netlify/edge-functions/smart-404.ts
import { createSmart404EdgeFunction } from 'docusaurus-plugin-smart-404/edge/netlify';
import manifest from '../../build/route-manifest.json' with { type: 'json' };

export default createSmart404EdgeFunction(manifest);

# netlify.toml
[[edge_functions]]
path = "/*"
function = "smart-404"

Uses the context.next() pattern: the site handles the request first; the matcher only sees genuine 404s.

Any other runtime

The core is a plain function over WHATWG Request/Response:

import { handleMiss } from 'docusaurus-plugin-smart-404/edge';

const response = await handleMiss(request, manifest, { redirectThreshold: 0.8 });
// Response => serve it; null => not a miss we handle, serve your normal 404

What the visitor gets

Match confidence	Answer
Exact match after normalization (case, trailing slash, .html suffix, //, /index, percent-encoding)	301 permanent redirect — the page exists, the URL was just written differently
Exact match after stripping a .md/.mdx suffix	302 temporary redirect — the page exists, but the caller asked for a markdown representation you might serve for real one day (see below)
Fuzzy match ≥ redirectThreshold with a clear lead over the runner-up	302 temporary redirect
Best candidate ≥ suggestThreshold (or an ambiguous tie at the top)	404 with a "did you mean" page listing the closest routes
Nothing close	null — your plain 404 flows through

Content negotiation

Suggestion pages are negotiated per caller:

• Accept: text/markdown, or a user-agent matching /claude|gpt|openai|perplexity|bot/i → a markdown body: a note that the path does not exist plus a link list of the closest real routes. Agents read it and self-correct in one round trip.
• Everyone else → a minimal semantic HTML page with the same links.

Both are served with status 404 and Vary: Accept, User-Agent, so nothing wrong ever gets indexed and shared caches never serve markdown to a browser (or HTML to an agent). Redirects preserve the request's query string.

Options

Plugin options

Option	Type	Default	Description
filename	string	'route-manifest.json'	Manifest filename written into outDir. Bare filename, no path separators.
exclude	string[]	[]	Glob patterns of routes to omit from the manifest (* = one segment, ** = any depth). The 404 route (* and the /404.html page) is always excluded. Excluded routes are invisible to the matcher — with the Vercel adapter, keep them out of the middleware matcher scope (see warning above).
emitRedirectsFile	boolean	false	Also write a _redirects file (Netlify/Cloudflare Pages format) from aliases. Requires a non-empty aliases.
aliases	Record<string, string>	—	Curated exact from → to redirects for the _redirects file.

Options are validated loudly — unknown keys and wrong types throw at config time, not silently at build time.

Matcher options

Accepted by handleMiss and every adapter:

Option	Type	Default	Description
redirectThreshold	number	0.75	Minimum fuzzy score to redirect instead of suggest.
suggestThreshold	number	0.4	Minimum score for a route to appear as a suggestion.
minGap	number	0.1	Minimum lead the best candidate needs over the runner-up to redirect. Ties suggest instead — never guess between two plausible pages.
maxSuggestions	number	5	Maximum suggestions listed.

How scoring works

Paths are normalized (lowercase, URL-decoded, trailing slash / .md / .mdx / .html / /index stripped, // collapsed), then tokenized on /, -, _, .:

score = 0.6 × jaccard(token sets) + 0.4 × (1 − normalizedLevenshtein(last segments))

Whole-path token overlap carries most of the weight; the final slug's edit distance settles typos. Plain DP Levenshtein, no dependencies.

The soft-404 SEO guardrail

301 is reserved for the alias class — cases where normalization proves the page exists and only the spelling of the URL differed. A 301 tells crawlers "this URL is that page, permanently," and consolidates link equity.

Fuzzy matches are never 301. A fuzzy match is a probabilistic guess; if it's wrong, a 301 permanently welds a hallucinated URL to an unrelated page in search indexes and caches, and downstream tools stop ever re-checking it. Fuzzy redirects are always 302, and anything ambiguous is a 404 suggestion page — honest to crawlers, still helpful to the visitor.

.md/.mdx misses are also never 301. Stripping a markdown extension is a representation guess, not pure spelling normalization: if you later serve genuine markdown at .md URLs (llms.txt-style), agents and caches holding a permanent redirect would skip your origin forever. Those get 302 even though the underlying page provably exists.

Preventing hallucinated links at the source

This package treats the symptom. To reduce hallucinated URLs in the first place, publish an llms.txt at your site root — a curated markdown index of your real docs URLs. Agents that fetch it link to pages that exist; agents that don't will still land on this package's markdown suggestion page and self-correct. The route-manifest.json this plugin emits is a convenient source of truth for generating one.

Package exports

Entry	Contents
docusaurus-plugin-smart-404	The Docusaurus plugin (default export)
docusaurus-plugin-smart-404/matcher	normalize, score, decide + shared types
docusaurus-plugin-smart-404/edge	handleMiss core handler
docusaurus-plugin-smart-404/edge/vercel	createSmart404Middleware
docusaurus-plugin-smart-404/edge/cloudflare	withSmart404
docusaurus-plugin-smart-404/edge/netlify	createSmart404EdgeFunction

Dual CJS + ESM builds with type declarations.

License

MIT