Lead-gen: first-party capture + masked CRM delivery + daily sync - #192
Draft
JakeSCahill wants to merge 13 commits into
Draft
Lead-gen: first-party capture + masked CRM delivery + daily sync#192JakeSCahill wants to merge 13 commits into
JakeSCahill wants to merge 13 commits into
Conversation
First-party, identity-linked capture for sales lead-gen (capture only — no external delivery yet, pending final legal confirm). Keyed to Auth0 sub. - Names: extract given_name/family_name from the Auth0 profile claim (scope already requested) in upstream.mjs; thread through the callback into the docs session and the user store (null-preserving). - MCP usage (self-captured, no AgentCat/MCPcat vendor): recordUser now stores the MCP client (parsed from User-Agent); a registerTool() wrapper records per-user tool-call count + tool names via recordToolUsage(). - Agent intent: the wrapper injects an optional `context` field on every tool's input schema (no-secrets prompt) and captures it per call. It is stripped before the real handler runs so it never leaks into downstream (bump/kapa) tool forwarding. Kept capped in our store only (hygiene). Group 1 (email/company/sign-up/last-sign-in) was already captured by recordUser — verified, no change. Kapa engagement pull is a follow-up. Co-Authored-By: Claude Opus 4.8 <[email protected]>
Scheduled server-side pull of each signed-in user's Kapa chatbot engagement (conversation count + last interaction) into our user store. Capture only — no external delivery yet (held pending final legal confirm). - lib/kapa-owner.mjs: extract external_owner_id = sha256(salt + sub) into a shared module (single source of truth); kapa-session.mjs now imports it so the owner id stays byte-identical (changing it orphans history). - lib/kapa-engagement.mjs: fetchEngagement(sub) mints an owner-scoped agent session and pages GET /agent/v1/agent/threads/ (verified endpoints — the query API exposes neither external_owner_id nor an owner filter). - store.mjs: recordKapaEngagement() + forEachUser() over the user store. - kapa-engagement-sync.mjs: @daily scheduled function that syncs all sub-keyed users. Cost note in-file for scaling later. Co-Authored-By: Claude Opus 4.8 <[email protected]>
Extends the Kapa engagement pull to also collect, per signed-in user: - competitor mentions — deterministic keyword/rules over the (Kapa-masked) question text (whole-word, case-insensitive; deliberately excludes "Kafka" since Redpanda is Kafka-compatible and it would be pure noise). - a capped sample of masked questions — Kapa masks secrets/PII before storage, so these are already safe; kept for the later topic/intent classification step. Stored per-user (kapaCompetitorMentions, kapaMaskedQuestions). Capture only — no external delivery. Topic/intent/use-case classification (a dedicated LLM step, not the Kapa agent) is deferred pending a model choice. Co-Authored-By: Claude Opus 4.8 <[email protected]>
First-party, identity-linked docs pageview capture (replaces anonymous Plausible/Heap/Algolia for lead purposes). Signed-in users only; capture only, no external delivery. - docs-activity.mjs: POST /docs-activity validates the docs session and records a pageview against the user's sub. Same-origin (cookie rides along), always 204 (leaks nothing about auth state), anonymous ignored. - store.recordDocsActivity: recent pages (cap 10), page counts (capped 100), product areas (by Antora component), pricing/get-started visit count, last-active, and capped search terms. capCounts() bounds growth. Docs search terms are our own capture (not Kapa-masked) → treated as content, kept in-store, scrub/derive before any delivery. The docs-ui beacon that feeds this is a separate commit. Co-Authored-By: Claude Opus 4.8 <[email protected]>
The CRM send was a fire-and-forget fetch() in the request path. On Lambda, the runtime can freeze the moment it responds, dropping in-flight async work, and a bare send has no retry -- so an unknown fraction of leads were silently never delivered. Move delivery to crm-deliver-background.mjs (a Netlify Background Function): post-response execution up to 15 min, with Netlify's built-in retries on failure. store.mjs now enqueues via a fast local invoke (202) instead of POSTing the CRM directly. Optional CRM_INVOKE_SECRET guards the endpoint so it can't be used to inject fake leads. Tests: tests/crm-delivery.test.ts (10) cover the enqueue path and the background handler (delivery, retry-on-5xx, secret guard, malformed body, capture-only when unset). Full suite: 74 passed, 3 skipped. Co-Authored-By: Claude Opus 4.8 <[email protected]>
…-22) Legal signed off on delivering the full lead profile to Salesforce, including conversation-derived content, provided that content is masked first. - lib/redact.mjs: first-party, dependency-free regex redaction. Masks emails, IPs, phone numbers, Luhn-valid cards, SSNs, tokens/keys, bearer tokens, credentialed URLs, and key=value secrets. Regex only, no NER, so names/orgs are not stripped (the basis legal approved); swap in Presidio later if needed. - store.mjs: buildLeadPayload(record) maps a stored record to the full CRM payload (identity + tool usage + Kapa counts + docs activity as-is; the three free-text fields redacted). First-sight delivery now sends this. Adds deliverLeadUpdate(sub) for enriched upserts. - kapa-engagement-sync: upserts each user's enriched masked profile daily, since first-sight delivery only carries identity. No-op until CRM_WEBHOOK_URL is set. - crm-deliver-background: HMAC-signs the body (X-Redpanda-Signature) when CRM_SIGNING_SECRET is set, so Salesforce can verify origin without IP allowlisting. Delivery stays gated on CRM_WEBHOOK_URL. Tests: 11 new (redaction, payload mapping, signing); full suite 85 passed, 3 skipped. Co-Authored-By: Claude Opus 4.8 <[email protected]>
The no-decision fields from the sales spreadsheet that need no new classifier: - company: capture Auth0 org_name (recordUser + docs-login callback) and forward it; fall back to a domain-derived name (acme.com -> "Acme") when the login has no org. - toolsInvoked: list of MCP tool names (alongside the existing per-tool counts). - docsTopPages: top 5 pages by view count. - primaryProductArea: highest-count docs component, normalized to the sales enum via an editable PRODUCT_AREA_MAP (unknown keys pass through; the Kafka-API bucket and real component values still need confirming with docs/product). Not touched (pending decisions): source value, pricing-counter rename, and the derived-intent fields (topic, question intent, product-interest, use-case) that need the sales enums + a classifier approach. Tests: +2 (company via org_name and domain fallback, area normalization, top-N). Full suite 87 passed, 3 skipped. Co-Authored-By: Claude Opus 4.8 <[email protected]>
… visits docs.redpanda.com has no pricing/sign-up/trial pages (those live on the marketing site), so the old commercial-intent matcher only ever fired on get-started paths and docsPricingVisits promised a signal we can't see here. - docs-activity: replace the COMMERCIAL patterns with a single get-started segment matcher (landing + sub-pages, any product; not mid-word). - store: rename docsPricingVisits -> docsGettingStartedVisits in both the stored record and buildLeadPayload; rename the recordDocsActivity flag commercial -> gettingStarted. Getting-started activity is a real onboarding-intent signal for a docs audience. True pricing-visit data isn't available from the signed-in docs feed. Full suite: 87 passed, 3 skipped. Co-Authored-By: Claude Opus 4.8 <[email protected]>
- Remove primaryProductArea (speculative enum mapping, low value) and its PRODUCT_AREA_MAP/normalizeProductArea helpers. docsProductAreas (raw component -> count) still carries the product signal. - Omit kapaCompetitorMentions from the delivered payload for now to keep the feed clean. The in-store capture (recordKapaEngagement) is untouched, so re-enabling delivery later is a one-line change. Full suite: 87 passed, 3 skipped. Co-Authored-By: Claude Opus 4.8 <[email protected]>
Make provenance obvious in the delivered contract. buildLeadPayload is now grouped by source (identity / mcp* / kapa* / docs*), and the MCP-sourced fields carry an mcp prefix: client -> mcpClient, tools -> mcpTools, toolsInvoked -> mcpToolsInvoked, toolCalls -> mcpToolCalls, lastToolAt -> mcpLastToolAt, agentIntents -> mcpAgentIntents The masked free-text field in each group now sits with its source (mcpAgentIntents, kapaQuestions, docsSearchQueries). Payload keys only; stored record field names are unchanged. Full suite: 87 passed, 3 skipped. Co-Authored-By: Claude Opus 4.8 <[email protected]>
Scheduled functions cap at 30s, so the old all-users-in-one-loop sync couldn't scale. Rework into Netlify's scheduled-trigger -> background-worker pattern, and process only active users instead of the whole (growing) user base. - lib/pending.mjs: pending-delivery index. Capture paths (recordUser, recordToolUsage, recordDocsActivity, kapa-session) mark a user pending on activity; the sync processes only pending users, so cost stays ~O(daily-active), not O(total signups). - lead-sync.mjs: @daily scheduled trigger (30s). Does no work itself — fires the two background workers and returns. - crm-sync-background.mjs: worker (15-min limit). Lists pending users, pulls Kapa engagement + enqueues one masked-profile delivery each with bounded concurrency (8) and a time budget; clears markers as it goes. Resumable if ever behind. - store-cleanup-background.mjs: worker. Deletes records with no genuine activity in the retention window (LEAD_RETENTION_DAYS, default 30) so the store stays bounded. Retention basis = lastActivityAt (real activity, not sync bookkeeping). - store.mjs: markPending on capture; add deleteUser + lastActivityAt helpers. - Both workers guarded by CRM_INVOKE_SECRET; delivery still gated on CRM_WEBHOOK_URL. - Removed kapa-engagement-sync.mjs (superseded). At 500 active/day: ~45s of worker time vs a 15-min budget; unaffected by total user growth. Tests: +10 (pending index, both workers, trigger, lastActivityAt); full suite 97 passed, 3 skipped. Co-Authored-By: Claude Opus 4.8 <[email protected]>
JakeSCahill
changed the title
…ist field
fetchEngagement read thread.question_answers[].question off the threads LIST,
but the live API never returns that field there — the list only carries
{id, last_activity_at, message_count, title}. So kapaQuestions was ALWAYS empty
(and the competitor scan, reading the same field, never fired). Verified against
the live API 2026-07-24: question text lives only in the per-thread detail
(GET threads/{id}/ -> messages[] where role==='user', text in .content).
- Rewrite fetchEngagement: page the list for counts/recency + ids, then fetch
question text from the most-recent threads' detail (bounded N+1: MAX_THREAD_DETAILS
20, MAX_QUESTIONS 25) so a chatty user can't blow the sync's time budget.
- Kapa masks question text at source (project PII config), so kapaQuestions is
delivered verbatim — no local re-masking. docsSearchQueries and
mcpAgentIntents.context are our own first-party capture and STILL redact.
Verified live: fetchEngagement now returns 6 questions for a real user (was 0).
Tests: 98 passed, 3 skipped.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
`source` was a hardcoded "mcp" literal — inaccurate once capture also fired from
the docs browser login, and uninformative either way. Replace it with
lastChannel: the channel of the user's MOST RECENT sign-in ('mcp' via an AI
client, or 'docs-login' via the docs browser).
- recordUser takes a `channel` arg; stored as lastChannel (this call's channel
wins, else keep prior). Tagged at both callers: mcp.mjs ('mcp'),
mcp-oauth.mjs ('docs-login').
- buildLeadPayload delivers lastChannel instead of source.
Chosen over a cumulative channels set (goes stale once both are seen) and a
most-used channel (frozen by early heavy use). Relative channel usage is already
readable from the mcp*/docs* magnitudes; lastChannel adds the freshest single
signal without staleness.
Full suite: 98 passed, 3 skipped.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
End-to-end first-party lead-gen pipeline for the docs MCP server and docs site: capture signed-in user activity, then once a day deliver a masked, identity-linked profile to the CRM / Salesforce. Legal cleared delivery (including masked conversation content) on 2026-07-22.
Pipeline
@daily): a 30s scheduled trigger fires two background workers — one refreshes pending users' Kapa engagement and delivers one coalesced masked-profile upsert each; the other runs retention cleanup.CRM_WEBHOOK_URL(no external delivery until configured).Production architecture (designed for ~500 logged-in users/day)
LEAD_RETENTION_DAYS, default 30).Data delivered (per user, upsert by
sub)Grouped by source so provenance is explicit —
mcp*/kapa*/docs*plus identity. Free-text conversation fields (mcpAgentIntents.context,kapaQuestions,docsSearchQueries) are redacted first-party before delivery (regex: email, IP, phone, card, SSN, tokens, credentialed URLs, key=value secrets). Regex only, no NER — names/orgs are not stripped; legal signed off on this basis.Env vars (all optional; delivery off until the first is set)
CRM_WEBHOOK_URL— the on/off switch for deliveryCRM_SIGNING_SECRET— enablesX-Redpanda-Signature(HMAC-SHA256) so Salesforce can verify origin (Netlify has no static egress IPs)CRM_INVOKE_SECRET— guards the internal background endpointsLEAD_RETENTION_DAYS— default 30Deliberately not included / follow-ups
sourceis hardcoded"mcp"; may split MCP vs docs-login sign-in later.docsGettingStartedVisitsis the onboarding-intent signal instead.Tests
vitest: 97 passed, 3 skipped. Covers oauth/session/mcp plus redaction, payload mapping, HMAC signing, the pending index, the sync + cleanup workers, and retention.🤖 Generated with Claude Code