Skip to content

[Feature] Profile Environment — map stack data for context-aware agent workflows #44

Description

@seth-goodwin

Problem

Agent skills in this repo (alert triage, threat hunt, query generation, correlation, etc.) often assume a well-instrumented Elastic Security "Showroom" deployment — endpoint telemetry, common ECS fields, entity analytics, cloud logs, network data, and so on.

Real customer environments vary widely. Some have Endpoint only, others are cloud-heavy with sparse host telemetry, others use custom index names or missing fields. MSSP analysts may work across many workspaces with different profiles. Without knowing what's actually in the stack, agents generate queries and recommendations that look correct in a demo but fail or return empty results in production.

Analysts shouldn't have to manually explain their environment every session.

Goal

Provide a Profile Environment workflow that inspects the connected Elastic/Kibana stack, summarizes what data exists and how usable it is, and saves that profile to the workspace so other skills can tailor output to the customer's actual environment.

The exact inspection methods, storage format, and aggregation mechanics are up to the implementer.

Context

  • Low-level building blocks already exist in this repo (list-indices, get-mapping, execute-esql, check-existing-sample-data, threat hunt workbench).
  • Agent skills live under skills/ and would benefit from shared environment context (field availability, log sources, coverage gaps, canonical pivot fields, etc.).
  • Multi-cluster support is an open enhancement (#22); profiles should be designed with cluster identity in mind.
  • MSSP workflows may eventually need a portfolio profile aggregated across customer workspaces for cross-customer hunting. Local profiles should use a structured format that could support this later, but portfolio sync is not required for v1.

Scope

Local profile (v1)

One profile per workspace, saved at a consistent location (e.g. environment-profile.md at workspace root). Should capture, as applicable:

  • Which data streams / indices exist (security alerts, endpoint, network, cloud, identity, etc.)
  • Rough volume and recency
  • Key ECS / security fields present or populated (not just mapped)
  • Integrations or sources inferable from index naming or metadata
  • Notable gaps ("no network telemetry", "no cloud audit logs", "alerts only, no raw events")
  • Advanced capabilities where detectable (Entity Analytics, Attack Discovery, correlation corpus, Cases, etc.)
  • Cluster binding (which cluster the profile was built against)
  • Timestamp and any sections the agent could not inspect (permissions, timeouts)
    Analysts should be able to review, edit, and confirm the profile.

Portfolio profile (MSSP — future)

A derived summary across customer workspaces for cross-customer hunt planning and coverage visibility. Built from local profiles, not a full re-probe of every cluster. Out of scope for v1 unless trivial to scaffold.

Acceptance criteria

  • Agent can profile the connected environment and produce a human-readable summary for analyst review.
  • Profile is saved to the workspace at a documented, consistent path.
  • Profile records which cluster it was built against.
  • Output includes enough detail for other skills to adapt queries and recommendations (indices, fields, gaps, capabilities).
  • Profile notes what could not be inspected and why, where applicable.
  • Agent skill added under skills/profile-environment/, consistent with existing skills in this repo.
  • Documented in repo (what is inspected, how to refresh, known limitations, sensitive data guidance).

Out of scope (for now)

  • Continuous monitoring or alerting on data quality degradation.
  • Fixing ingestion gaps (profile should report them, not remediate).
  • Full Fleet / Integration Manager UI parity.
  • Automated portfolio sync across workspaces.
  • Automatic write-back to Elasticsearch or Kibana saved objects.
  • Replacing Elastic's native observability or stack monitoring products.

Open questions

  • Markdown vs YAML vs JSON for the profile format?
  • What APIs and queries give the best signal vs cost (ES|QL, _cat, Fleet API, Kibana features API)?
  • How should other skills consume the profile — reference file, dedicated MCP tool, injected skill context?
  • How should skills behave when no profile exists, or when profile cluster ≠ active MCP cluster?
  • How should demo/sample data clusters be flagged (check-existing-sample-data)?
  • Default gitignore for profiles containing customer topology?
  • Refresh policy — on demand only, stale after N days, on cluster switch?

Related

  • Parent initiative (optional link): security-team#17827
  • Multi-cluster: #22
  • Existing primitives: list-indices, get-mapping, threat-hunt, check-existing-sample-data

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions