Skip to content

Add ambient LLM incident watcher with AI agents#321

Merged
clcollins merged 31 commits into
mainfrom
srepd/ambient-watcher
Jun 10, 2026
Merged

Add ambient LLM incident watcher with AI agents#321
clcollins merged 31 commits into
mainfrom
srepd/ambient-watcher

Conversation

@clcollins

Copy link
Copy Markdown
Owner

Summary

  • Add collapsible watcher pane below incident table with dynamic 2/3-1/3 height split, mouse scrolling, and typewriter response display
  • Add :agent command (CLI subprocess) and :watcher command (LLM API) with shared rich incident context (alerts, OCM cluster info, service logs, limited support, notes)
  • Add ambient heuristic pattern detectors: service storm, cluster storm, urgency shift — with LLM synthesis when provider is healthy
  • Add configurable system prompts (agent_system_prompt, watcher_system_prompt), emoji toggle, and agent_cli_command
  • Switch command prefix from / to : (vim-style), add / key for future search, remove f and i key bindings
  • Fix flag markers not appearing until manual refresh — all flag operations now trigger immediate table rebuild
  • Add elapsed countdown timer and provider health status to footer
  • Suppress noisy mouse/OCM log messages from journal
  • Full documentation: docs/ai-agents.md, docs/configuration.md, updated flag-conditions.md, llm-providers.md, README

Closes #305
Closes #312
Closes #315

Test plan

  • make test-all passes (fmt-check, vet, lint, test, test-race)
  • golangci-lint clean
  • Watcher pane toggles with w, resizes dynamically, scrolls with mouse wheel
  • :agent queries dispatch to CLI agent with full incident context, responses typewrite into watcher pane
  • :watcher queries dispatch to LLM provider with full context, responses typewrite into watcher pane
  • Ambient detectors fire on incident list updates, synthesize via LLM when healthy
  • Health check status and countdown timer visible in footer
  • :flag/:flags/:unflag commands work with new : prefix
  • Flag markers appear immediately on add/remove/load
  • emoji: false switches all markers to text fallbacks
  • Dev mode (srepd --dev) works with watcher features

🤖 Generated with Claude Code

clcollins and others added 28 commits June 9, 2026 10:55
Phase 1 of #305: introduces the watcher pane layout with a bordered
viewport below the table, toggle via 'w' key, dynamic table height
adjustment, resize handling, and status header showing provider name,
health, and activity state.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Phase 2 of #305: redirect /agent CLI responses from the incident
viewer to the watcher pane. Add ring buffer for observation history,
configurable emoji/text markers (🤖/📡 vs ☻/☺) via emoji config bool,
per-line marker prefixing, word wrap at viewport width, mouse wheel
scrolling, and persistent input field during agent queries.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Mirrors vim behavior: pressing / opens command input with the slash
already typed, ready for /agent or /flag commands. The i and : keys
continue to open input without pre-fill.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Use vim-style : prefix for commands (:agent, :flag, :flags, :unflag).
Reserve / for future search. Remove f and i key bindings — use : or /
to enter input mode. Both : and / pre-fill their character in the
input field.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Phase 4 of #305: pure Go detectors that analyze the incident list for
cross-incident patterns — service storms (3+ on same service), cluster
storms (2+ on same cluster), and high urgency clustering. Detectors
run on incident list updates and flag changes. Observations are
deduplicated with a 5-minute cooldown.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Set viper.SetDefault("emoji", true) so emoji markers are used when
the config key is absent. Auto-expand the watcher pane when detectors
produce new observations.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Phase 5 of #305: when a pattern detector fires and the LLM provider
is healthy, dispatch a Query to synthesize a richer natural-language
observation. Falls back to raw heuristic text when the provider is
unavailable or offline. Watcher header shows analyzing state.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Flag changes (add, remove, clear, load) rebuilt the match cache but
never triggered a table row rebuild — markers only appeared on the
next 15-second poll. Now all flag operations dispatch an immediate
table rebuild. Flags load also kicks off OCM enrichment for any
unenriched incidents so cluster-based flags match as soon as
enrichment completes. Also rebuild flag cache when cluster enrichment
arrives so new cluster data is matched against existing flags.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Type :watcher <question> to query the configured LLM provider with
rich incident context: selected incident details, alerts, cluster
info, service logs, limited support history, notes, and the full
incident queue summary. Responses appear in the watcher pane with
the watcher marker prefix. Input stays focused for follow-up queries.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
buildWatcherContext now reads alerts and notes from incidentCache
(populated by OCM enrichment) instead of selectedIncidentAlerts/Notes
(only populated on manual view). Also includes SOP links, cloud
provider, and extracts cluster context into a reusable helper.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Table always gets at least 10 rows. When watcher is expanded: if
enough room, split 2/3 table and 1/3 watcher (default 10 rows each
minimum). If tight, table gets its minimum and watcher gets the rest
(min 5 rows). All layout constants are named, no magic numbers.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Move the [AI Watcher] status from its own header line into the
existing footer as a right-justified column alongside "Watching for
updates...". Only shown when the watcher pane is expanded. Saves one
line of vertical space for the table and watcher content.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Convert all error paths in agent CLI, watcher query, and provider
status checks from permanent setStatus to flashNotification so errors
appear temporarily in the status bar and auto-clear.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Pad idle state to match the width of spinner + analyzing label so the
footer line doesn't wrap when transitioning between states. Bold the
analyzing text when active. Use lipgloss.Width for spinner measurement
instead of a magic number.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Revert persistent input focus — all commands (:agent, :watcher, :flag)
now blur the input and return focus to the table immediately after
dispatch, matching vim's :w behavior.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Use theme highlight color (white) instead of bold for the analyzing
label. Add MaxWidth to both footer columns to prevent line wrapping.
Remove idle padding — MaxWidth handles overflow by truncation.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Render the watcher status first, measure its actual width including
ANSI codes via lipgloss.Width, then give the left column the
remaining space. Eliminates wrapping regardless of spinner or
styled text content.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Responses from :agent, :watcher, and LLM synthesis now display
word-by-word in the watcher pane at 3 words per 30ms tick, giving
a fast typewriter effect. Uses tea.Tick and a SetLast buffer method
to update the last entry in-place as words are revealed.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Show a counting-up timer next to "analyzing..." in the footer status
when an agent or watcher query is in flight. The spinner already
triggers re-renders so the timer updates live. Resets when a new
query starts.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Count down from the query timeout (60s for agent/watcher, 30s for
synthesis) instead of counting up. Extract timeout constants from
hardcoded values into named constants.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Add tea.MouseMsg to the high-frequency message filter to prevent
mouse motion events from flooding the journal.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Add ocmServiceLogsMsg and limitedSupportMsg to the high-frequency
message filter to prevent large OCM responses from flooding the
journal.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
The agent CLI now receives the same rich context as the watcher:
selected incident details, alerts, cluster info, service logs,
limited support, notes, and full queue summary — all piped via
stdin alongside the system prompt and user query.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Add Padding(0, 1) to the watcher container style to match the table
cell padding, aligning text indentation between the two panes.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Move agent and watcher system prompts from hardcoded constants to
configurable agent_system_prompt and watcher_system_prompt config
keys with sensible defaults. Both :agent and :watcher now use the
same buildWatcherContext for incident data. Extract cell padding
into named constants for consistency between table and watcher pane.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Replace strings.Fields (which collapses all whitespace) with a
custom splitter that preserves newlines as separate tokens. The
typewriter now reconstructs multi-line responses correctly.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
…cher

Phase 6 of #305:
- New docs/ai-agents.md: quickstart, architecture, commands, context,
  prompts, markers, health checks, detectors, timeouts, privacy
- New docs/configuration.md: CLI args section + full config file
  reference with every key, type, default, and description
- Update docs/flag-conditions.md: /flag -> :flag prefix, remove f key
- Update docs/llm-providers.md: add watcher integration section
- Update README.md: add AI agents and config doc features, update key
  bindings (remove f/i, add w), update command prefixes, link to new
  docs, add emoji/prompt config keys to optional table
- New docs/plans/062-ambient-watcher.md: plan doc for the branch

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Fix gofmt alignment in keymap.go, model.go, watcher.go. Add tests
for SetLast, buildIncidentSummary, isWatcherCommand, parseWatcherQuery,
splitKeepingNewlines, and buildWatcherContext. Escape pipe character
in configuration.md markdown tables.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
@codecov-commenter

codecov-commenter commented Jun 9, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 76.79671% with 113 lines in your changes missing coverage. Please review.
✅ Project coverage is 68.30%. Comparing base (3844410) to head (6ad87dd).

Files with missing lines Patch % Lines
pkg/tui/watcher.go 82.53% 25 Missing and 15 partials ⚠️
pkg/tui/commands.go 3.70% 25 Missing and 1 partial ⚠️
pkg/tui/msgHandlers.go 63.33% 8 Missing and 3 partials ⚠️
pkg/tui/tui.go 84.50% 8 Missing and 3 partials ⚠️
pkg/tui/model.go 57.14% 9 Missing ⚠️
pkg/tui/layout.go 76.92% 5 Missing and 1 partial ⚠️
pkg/tui/views.go 82.85% 5 Missing and 1 partial ⚠️
cmd/root.go 60.00% 2 Missing ⚠️
pkg/tui/claude.go 90.90% 1 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #321      +/-   ##
==========================================
+ Coverage   67.75%   68.30%   +0.55%     
==========================================
  Files          45       46       +1     
  Lines        6906     7323     +417     
==========================================
+ Hits         4679     5002     +323     
- Misses       1915     1988      +73     
- Partials      312      333      +21     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

…arker

The flag_marker config key is effectively dead — markers are controlled
by the emoji toggle. Update documentation to point users to emoji: false
instead of flag_marker config.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
clcollins and others added 2 commits June 9, 2026 14:05
30+ tests covering: watcher toggle, prompt/response/synthesis message
handlers, typewriter ticks, mouse events, input mode :watcher dispatch,
renderWatcherPane, renderWatcherStatus, renderFooter watcher branch,
buildClusterContext, advanceTypewriter, and buildWatcherContext with
cached alerts and notes.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Use tea.MouseMsg{Action: MouseActionPress, Button: MouseButtonWheelDown}
instead of the deprecated tea.MouseMsg{Type: MouseWheelDown}.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
@clcollins clcollins merged commit 9d5aace into main Jun 10, 2026
10 checks passed
@clcollins clcollins deleted the srepd/ambient-watcher branch June 10, 2026 00:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants