[Epic] MCP App Usage Telemetry: Account level view

[davethegut]

[Epic] MCP App Usage Telemetry: Account level view

Status: In Progress
Author: David Elgut (PM, Security AI)
Date: 2026-05-29
Epic Link: (to be filled when published as a GitHub issue)
Target Release: Rolling — instrumentation shipped; analytics pipeline next

---

Background

The Elastic Security MCP App puts Security workflows — alert triage, attack discovery, case management, detection rules, threat hunting, and sample-data generation — directly inside an MCP host such as Claude Desktop. Until recently we had no visibility into which of these surfaces customers actually use once the app is installed in the wild. We could not answer basic adoption questions: which views get opened, which tools get invoked, whether a feature is dead weight or load-bearing.

PR #29 (merged May 26, 2026) closed that gap. It wires the MCP App into Elastic's V3 analytics shipper (@elastic/ebt) and emits two closed-schema, anonymised events — mcp_tool_called and view_rendered — that respect the user's existing Kibana telemetry opt-in. The implementation is deliberately privacy-conservative: no PII, no free-form text, no cluster_name, and a fail-closed opt-in model.

Data is already flowing. Events are visible today in the stack-telemetry securitysolution view. This epic captures that completed instrumentation work as a record, and scopes the next phase: turning the raw event stream into PM-grade insight by correlating cluster identity to customers, sourcing the data from the forthcoming golden table, and surfacing it in a dedicated tab of the Security AI telemetry dashboard.

---

User Story / Problem Statement

As a Security AI Product Manager,
I want to see anonymised MCP App usage telemetry — which views are rendered and which tools are called, by how many clusters — surfaced in the PM telemetry dashboard,
so that I can measure adoption, prioritize investment toward the surfaces customers actually use, and tell a data-backed story about MCP App traction.

Problem Details

• No adoption signal before PR feat(analytics): add anonymised usage telemetry via @elastic/ebt #29. We shipped six MCP views and ~45 tracked tools with zero feedback on which were used. Roadmap calls were made on intuition, not behavior.
• Raw events are not yet PM-consumable. The data now lands in stack-telemetry, but it is anonymised at the cluster_uuid level with no link to a named customer or user, and it is not yet surfaced anywhere a PM looks day-to-day.
• No customer correlation. A cluster_uuid alone cannot answer "is this a strategic account?" or "how many distinct customers are exercising attack discovery?" — the questions that actually drive prioritization.

---

The Solution

Two phases. Phase 1 (shipped — Explore) instruments the MCP App and ships anonymised usage events under the user's Kibana opt-in. Phase 2 (next — Expand) builds the analytics path: render an "MCP App" tab in the sec-pm-telemetry-dashboard and correlate cluster identity to customers, sourced from the golden table when it lands.

To avoid dangling multi-phase work, Phase 2 is sequenced so value lands early: a v0 tab on anonymised, cluster-level metrics (view renders, tool-call volume and success rate, active clusters) can ship from currently queryable data without waiting on the golden table or identity enrichment. Customer-level breakdowns layer in once WS2 (identity) and WS3 (golden table) complete.

UX: Before

A PM asking "how is the MCP App being used?" had nowhere to look. No dashboard, no query, no event stream. The answer was anecdote.

UX: After

• Usage events ship automatically from every opted-in install (shipped).
• A PM opens the Security AI telemetry dashboard, selects the MCP App tab, and sees view-render counts, tool-call volume and success rates, active clusters, and — once identity enrichment lands — active customers and account-level breakdowns over a selectable date range.

---

Architecture

graph TD
    subgraph app [MCP App - shipped in PR 29]
        Views["React views (view_rendered)"]
        Tools["Tracked MCP tools (mcp_tool_called)"]
        OptIn["Kibana opt-in mirror (GET /api/telemetry/v2/config)"]
        Ctx["Anonymised context: cluster_uuid, version, license, app_version"]
    end
    Views --> Shipper["@elastic/ebt V3 shipper"]
    Tools --> Shipper
    OptIn -->|"gates shipping (fail-closed)"| Shipper
    Ctx --> Shipper
    Shipper -->|"opt-in only"| Telemetry["telemetry.elastic.co"]
    Telemetry --> Golden["Golden table (forthcoming)"]
    Golden --> BQ["BigQuery refresher (sec-pm-telemetry-dashboard)"]
    Identity["Identity mapping: cluster_uuid -> account/user"] -.->|"enrichment (open design)"| BQ
    BQ --> Tab["MCP App dashboard tab"]

Loading

Notes:

• The V3 shipper requires cluster_uuid to ship at all; events without cluster context are dropped (fail-closed).
• Identity enrichment (dotted edge) is an open design question — the join source is not yet decided (see Open Questions).
• The dashboard consumes data through its existing pattern: a BigQuery refresher registered in registry.ts, cached behind /api/metrics, rendered by a tab component wired into App.tsx.

---

Goals & Non-Goals

Goals

• Visibility into which MCP views are rendered and which tools are called, by volume and success rate (shipped).
• Respect the user's Kibana telemetry opt-in, fail-closed (shipped).
• Correlate cluster_uuid to the associated customer/account so PMs can reason about adoption per customer.
• Source the data from the golden table once available, and surface it in a dedicated dashboard tab.

Non-Goals

• No PII or free-form capture. Schemas are closed (Zod + EBT); no user identifiers, no alert/case/rule bodies, no ES|QL query text.
• No per-keystroke event volume. High-frequency emitters (e.g. live-validation) must not flood the stream (see execute-esql decision below).
• No cluster_name collection. It is user-controlled and frequently leaks company/environment identifiers, undermining the anonymised framing.
• No per-install segmentation in v1. Two installs on the same cluster share cluster_uuid and are indistinguishable; acceptable for now.

---

Work Streams

Work Stream 1: Telemetry instrumentation (DONE — shipped in PR #29)

• Wire the MCP App into Elastic's V3 analytics shipper via @elastic/ebt
• Emit mcp_tool_called (server-side: tool_id, duration_ms, success) for every tracked tool handler
• Emit view_rendered (client-side: view_id enum of 6 views) once per top-level view mount
• Add the registerTrackedAppTool wrapper so new tools get instrumented automatically
• Mirror the Kibana telemetry opt-in (GET /api/telemetry/v2/config) once at startup, fail-closed when optIn is false/null or the fetch errors
• Attach anonymised context to every event: cluster_uuid, cluster_version, license_id/status/type, mcp_app_version (deliberately omit cluster_name)
• Enforce closed schemas end-to-end (Zod on the client→server tool boundary, EBT schema server-side)
• Document the event catalog, opt-out story, and codebase map in docs/telemetry.md
• Add unit tests + typecheck coverage for the analytics client, context loader, and telemetry service
• Add MCP_APP_TELEMETRY_ENV=staging developer override to route events to telemetry-staging.elastic.co
• Fix the execute-esql over-emission (~7k events/session from Monaco live-validation) and the related threat-hunt UI bug
• Confirm events land in the stack-telemetry securitysolution view

Work Stream 2: Identity enrichment — cluster → customer/user (NEXT)

• Decide the join source for mapping cluster_uuid (and/or license_id) to an account/org and its associated users — open design question (golden table vs. existing BigQuery identity tables)
• Secure product/privacy sign-off for using license_id as a customer-linkable key (open thread from PR feat(analytics): add anonymised usage telemetry via @elastic/ebt #29 review)
• Define the enrichment contract: what dimensions PMs need (account name, ARR tier, deployment type) and where they come from
• Validate that enrichment respects the anonymised framing and privacy constraints

Work Stream 3: Golden table access (NEXT)

• Obtain access to the golden table once it lands
• Confirm its schema covers both MCP event types and the full context block
• Validate freshness/latency is adequate for dashboard refresh cadence (~6h)

Work Stream 4: Dashboard tab in sec-pm-telemetry-dashboard (NEXT)

• Ship v0 on currently-available anonymised data (cluster-level metrics), then layer in customer dimensions after WS2/WS3 land
• Create a BigQuery query module (e.g. src/server/queries/mcp-app-usage.ts) returning a CategoryMetrics payload
• Register the refresher in src/server/queries/registry.ts
• Sync src/client/hooks/expected-categories.ts (and its snapshot test) with the new category
• Build the dashboard component (e.g. McpAppUsageDashboard.tsx) reusing MetricCard, TrendChart, MonthlyBarChart
• Wire the new tab into the NAV array and render switch in src/client/App.tsx
• npm run typecheck && npm run build && npm test

---

Child Issues

Issue	Description	Deliverable
ISSUE-001	Decide and document the cluster→customer identity join source (golden table vs. BigQuery identity tables)	Design note + chosen join path
ISSUE-002	Secure product/privacy sign-off for license_id as a customer-linkable key	Documented approval (or alternative key)
ISSUE-003	Obtain golden table access and validate schema coverage for MCP events + context	Verified access + schema mapping
ISSUE-004	Add the "MCP App" tab to sec-pm-telemetry-dashboard (query module → registry → tab UI)	Live tab showing view/tool usage metrics

---

Key Design Decisions

Decision	Rationale	Tradeoffs
Anonymise the feed: no cluster_name, no install_id in v1	Keeps the stream privacy-safe and avoids leaking user-controlled identifiers	Two installs on one cluster are indistinguishable; no per-install segmentation
Fail-closed opt-in mirrored from Kibana, read once at startup	Single, user-controlled privacy knob; never ship when telemetry is off	Requires MCP host restart after flipping the Kibana setting; no live polling
Closed Zod + EBT schemas	A buggy or malicious view cannot smuggle free-form text/PII into the pipeline	Adding a field is a deliberate, multi-file, schema-impacting change
Include license_id in context	Enables (pending sign-off) correlation to a customer for Cloud deployments	Org-identifying; needs explicit product/privacy sign-off before relying on it
Exclude/throttle high-frequency emitters (e.g. execute-esql live validation)	A single session emitted ~7k events from per-keystroke validation, drowning dashboards	Lose keystroke-level granularity (not needed for adoption metrics)
Ship a v0 dashboard tab on anonymised cluster-level data before identity enrichment	Decouples user-facing value from the golden-table ETA and privacy sign-off, avoiding dangling multi-phase work	v0 cannot show customer names/ARR until WS2/WS3 land

---

Success Criteria

Metrics

Metric	Current	Target	Timeline
MCP usage events landing in telemetry	Live in stack-telemetry	Sustained, opt-in-respecting flow	Met (PR #29)
Active clusters emitting MCP events (baseline)	TBD — capture from stack-telemetry	Establish baseline, then track week-over-week growth	WS4 v0
Cluster→customer correlation available	None	Active customers derivable from events	After WS2 + WS3
MCP App tab live in PM dashboard	None	v0 (anonymised) tab live; customer breakdowns after WS2/WS3	After WS4
Opt-in fidelity	Fail-closed by design	0 events shipped when Kibana opt-in is off	Met (PR #29)

Acceptance Checklist

• Anonymised mcp_tool_called and view_rendered events ship under the Kibana opt-in
• Events visible in the stack-telemetry securitysolution view
• cluster_uuid reliably maps to a customer/account via an approved join source
• Golden table access secured and schema validated
• "MCP App" tab live in sec-pm-telemetry-dashboard with passing typecheck/build/tests

---

Technical Requirements

• Performance: Telemetry must not block tool handlers; events queue in-memory and ship asynchronously. Dashboard queries follow the repo's cost-capped BigQuery governance.
• Security & Privacy: Closed schemas only; no PII; license_id usage gated on sign-off; honor the Kibana opt-in fail-closed.
• Compatibility: @elastic/ebt@^1.4.1 (adds ~+560 KB raw / +113 KB gzipped to the bundle — one-time .mcpb download cost).
• Observability: Opt-in resolution, batching, and shipping decisions are logged to stderr under the telemetry scope.
• Testing: Unit + typecheck on the analytics path (shipped); typecheck/build/test gate on the dashboard tab (WS4).

---

Related Epics

• MCP App Security — Vision (work_docs/prd/mcp-app-security-vision.md) — parent vision for the Security MCP App; this epic delivers its usage-measurement pillar.
• PR #29 — the merged implementation this epic records and extends.

---

Stakeholders

Role	Name	Responsibility
Product Manager	David Elgut	Requirements, prioritization, acceptance
Engineering (MCP App)	Kenneth Kreindler (KDKHD)	Telemetry instrumentation (shipped)
Reviewer / Collaborator	davethegut	Review, staging validation, privacy sign-off
Data / Analytics	TBD	Golden table access, identity join
Dashboard	TBD	MCP App tab implementation