feat(crawler): P3b — remove Playwright entirely, Firecrawl everywhere#9
Merged
Conversation
- screenshot.ts: Firecrawl full-page screenshot + fetchScreenshotBuffer, returns sniffed contentType (png/jpeg) for Gemini vision - techStackDetector: Firecrawl-rendered HTML for script/html/meta signatures + plain fetch for response headers (cf-ray/x-vercel/x-nf survive); window-globals signatures dropped (browserless tradeoff, per blueprint) - performanceInsights.ts deleted — zero callers; deepAudit already uses Google PageSpeed Insights - analyzeCompetitor + dailyAnalysis: inline chromium blocks -> captureScreenshot() - pricing_snapshots.source now 'firecrawl' on write (union keeps 'playwright' for old rows) - Removed playwright + @trigger.dev/build deps and the Trigger playwright build extension; tests rewritten against Firecrawl mocks Verified: npm run verify green (typecheck, 482 tests, build). Co-Authored-By: Claude Fable 5 <[email protected]>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Kilo Code Review could not run — your account is out of credits. Add credits or switch to a free model to enable reviews on this change. |
The changes job read outputs from the wrong step (steps.filter has no backend/frontend/any outputs, and 'files' is an input not an output), so 'any' was always empty and Lint/TypeScript/Tests/Build always skipped. Co-Authored-By: Claude Fable 5 <[email protected]>
…on passes /api/webhook constructs Webhooks() at module scope; standardwebhooks throws 'Secret can't be empty' during build page-data collection without it. Pre-existing failure surfaced by the changes-filter fix (Build never ran before). Co-Authored-By: Claude Fable 5 <[email protected]>
This was referenced Jul 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Completes the Firecrawl consolidation (P3b of
docs/FIRECRAWL_CONSOLIDATION.md). Theplaywrightdependency and the Trigger.dev playwright build extension are gone — container builds get lighter and there's no Chromium to install anywhere.What moved
screenshot.ts— rewritten on Firecrawl (full-page screenshot format → buffer). Returns a sniffedcontentType(png/jpeg) so Gemini vision gets the real mime type instead of assumed jpeg.techStackDetector.ts— Firecrawl-rendered HTML (script srcs parsed from markup) + a plainfetchfor response headers. Header signatures (cf-ray, x-vercel, x-nf, x-amz-cf) survive; onlywindowglobals signatures were dropped — the accepted browserless tradeoff from the blueprint.performanceInsights.ts— deleted outright: it had zero callers;deepAuditalready uses Google PageSpeed Insights.analyzeCompetitor/dailyAnalysis— inline chromium screenshot blocks replaced withcaptureScreenshot(); screenshot failure now skips the competitor instead of crashing the run.pricing_snapshots.sourcewritten asfirecrawl(type union keepsplaywrightfor reading old rows).Verification
npm run verifygreen: typecheck clean, 482 tests pass, production build succeeds.FIRECRAWL_API_KEYset in Trigger.dev env vars — the vision/daily-analysis jobs now call Firecrawl too.🤖 Generated with Claude Code