feat(crawler): Firecrawl-v2 consolidation — P1 extractor, P2 geo/screenshots, P3a cascade deletion#6
Merged
Conversation
Replace 3-tier scraper cascade + fake geo emulation + Gemini-vision pricing with a single Firecrawl v2 scrape (real location proxies, json/changeTracking/screenshot). Keeps the deterministic diff engine and AEO 5-model engine untouched. Net ~-1,400 LOC, drops Playwright. Co-Authored-By: Claude Opus 4.8 <[email protected]>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Kilo Code Review could not run — your account is out of credits. Add credits or switch to a free model to enable reviews on this change. |
…shadow parity Add scrapePage.ts: single Firecrawl `scrape` with `json` structured extraction mapped to the canonical PricingSchema, so the deterministic diff engine consumes it unchanged. Real geo via location.country, changeTracking(json) as an unchanged-page pre-filter, gated behind FIRECRAWL_EXTRACTOR. Additive only — no cutover (that's P3). Fix: firecrawl-js@4 silently drops Zod v4 schemas (detects Zod via v3 internals), so `json` came back empty. Send z.toJSONSchema() instead. Verified live against vercel.com/pricing and linear.app/pricing. Add scripts/shadow-parity.test.ts + `npm run shadow`: runs the Firecrawl and Playwright extractors on the same live URLs and reports schema/diff parity, using diffPricing itself as the oracle. Gated off the normal suite unless SHADOW_URLS. Tests: 7 new (mapper flows through real diffPricing); 589 + shadow skipped green; typecheck:ci clean. Co-Authored-By: Claude Opus 4.8 <[email protected]>
scrapePage.ts gains opt-in captureScreenshot → Firecrawl screenshot format, returns screenshotUrl; fetchScreenshotBuffer() bridges Firecrawl's hosted URL to R2's Buffer API. Adds tests asserting geo (location.country + languages) actually reaches Firecrawl. Still flag-gated; live pipeline cutover is P3. Co-Authored-By: Claude Opus 4.8 <[email protected]>
Add docs/COMPETITIVE_PARITY_GUIDE.md (top-10 feature ranking + PAR-1..8 parity roadmap vs Competitors.app/Visualping/Peec/Klue). Ignore .agents/, .claude/, skills-lock.json. Co-Authored-By: Claude Fable 5 <[email protected]>
…ractor (flagged) checkPricingContext now dual-paths behind FIRECRAWL_EXTRACTOR: flag on → scrapePricing() (structured json + real geo proxy + hosted screenshot via fetchScreenshotBuffer→R2); flag off → unchanged Playwright geo path. Default stays Playwright, so this is reversible and ships without a cutover. Also strips the dead Firecrawl-upgrade branch (all three arms called scrapeWithGeoContext anyway) and its getCurrencySymbols/shouldUpgradeToPlaywright/ fetchPageWithFallback uses. domHash derived from Firecrawl markdown; currencyDetected from the extractor's currency; screenshot upload guarded for the nullable hosted case. Parity: Firecrawl extraction verified correct on 8 live SaaS pricing pages (Vercel/Linear/Slack/Notion/Zapier/Figma/Buffer/Dropbox) via ground-truth check. No price_raw normalization needed — the diff engine already numeric-normalizes via extractNumericPrice, so format drift cannot trigger false alerts. Deletions (Playwright/Cheerio/vision/geo + deps + trigger extension) deferred until the automated shadow-parity control can run in a browser-equipped env. Test: added a case asserting the flag routes to Firecrawl and skips Playwright. Co-Authored-By: Claude Opus 4.8 <[email protected]>
Adds a `shadow-parity` job that runs scripts/shadow-parity.test.ts — Firecrawl scrapePricing() vs the Playwright control on live pricing pages, with diffPricing() as the oracle. This is the gate the P3 deletions wait on: prove the extractor matches before deleting the Playwright/vision cascade. Opt-in only (real Firecrawl credits + live network + a browser, flaky by nature): runs on workflow_dispatch or a PR labeled "ci:shadow-parity", never on normal pushes/PRs. Not a required check. Dispatch inputs flow through env, never inlined into run:. Requires repo secret FIRECRAWL_API_KEY. Co-Authored-By: Claude Opus 4.8 <[email protected]>
Records what the code review turned up: the shadow-parity gate imports geoPlaywright (so deletion must follow a green gate, not precede it); the gate needs the FIRECRAWL_API_KEY secret; the flag defaults off so geoPlaywright is still the live path; and `playwright` is load-bearing for five modules, so the dep only leaves once all are migrated. Splits P3 into P3a (post-gate pricing cascade deletion) and P3b (separate five-module Playwright migration). Co-Authored-By: Claude Opus 4.8 <[email protected]>
…rawl-only - checkPricingContext: single Firecrawl path; FIRECRAWL_EXTRACTOR flag and bestScraper/decideScraper plumbing removed (no fallback left to gate) - delete geoPlaywright, geoContext, decideScraper, visionPricing, cheerio.ts, playwright.ts, crawler barrel (zero external consumers) + their tests - drop cheerio dep; KEEP playwright dep + screenshot.ts + Trigger extension for the P3b survivors (techStack/performance/screenshots) - shadow-parity script + CI job repurposed as opt-in live Firecrawl smoke - uploadScreenshot sniffs PNG/JPEG magic bytes (was mislabeling image/webp) - cutover authorized without automated Playwright control: 8/8 ground-truth + 5/5 live Firecrawl extraction on CI default URLs (2026-07-10) Co-Authored-By: Claude Fable 5 <[email protected]>
Bharath-code
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Consolidates the pricing scrape engine onto Firecrawl v2 per
docs/FIRECRAWL_CONSOLIDATION.md. Net −2,478 lines vs main.scrapePage.ts: single Firecrawlscrapewith structuredjsonextraction into the canonicalPricingSchema; the deterministic diff engine (the moat) consumes it unchanged. Includes the Zod-v4 → JSON-Schema workaround (firecrawl-js silently drops Zod v4 schemas).fetchScreenshotBuffer), real geo vialocation.country.checkPricingContextis Firecrawl-only. DeletedgeoPlaywright,geoContext,decideScraper,visionPricing,cheerio.ts,playwright.ts, the crawler barrel, their tests, thecheeriodep, theFIRECRAWL_EXTRACTORflag, andbestScraperplumbing.playwrightdep,screenshot.ts, and the Trigger build extension — five surviving consumers (techStack, performance, screenshots,analyzeCompetitor,dailyAnalysis) migrate in P3b.FIRECRAWL_API_KEY— not yet added).uploadScreenshotsniffs PNG/JPEG magic bytes; it previously labeled every uploadimage/webp(all callers were actually sending PNG).Quality upgrade, not just deletion
Real regional proxy exits replace Playwright locale spoofing, and structured extraction replaces Gemini-vision parsing (fewer hallucinations, no vision-token spend).
Verification
npm run verifygreen (typecheck + full vitest suite + production build)🤖 Generated with Claude Code