feat(observability): metrics, alarms, error tracking, cache verification (fixes #17)#27
Merged
Merged
Conversation
…ion (fixes #17) - shared/metrics.py: CloudWatch custom metrics (subscription verifications, query-limit hits, NB API errors, token-refresh failures, agent latency) - shared/observability.py: Sentry error tracking wired into all Lambda handlers; no-ops when SENTRY_DSN is unset (sentry-sdk optional dep) - CloudWatch alarms in template.yaml (error rate, webhook failures, token-refresh failures, nation-not-found spikes) - Cache hit/miss logged from Anthropic response usage to verify prompt caching is actually active - Tests for metrics + observability (+14) Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
17 tasks
…vability # Conflicts: # src/lambdas/nat_agent_streaming/handler.py
Independent review of #27 noted the ResultMessage instrumentation (cache usage + agent latency emit) shipped untested — _FakeResultMessage lacked usage/duration_ms and no test injected a ResultMessage into the stream. Added the fields to the fake (mirroring the real SDK) and a test asserting record_cache_usage receives the usage dict and a latency metric is emitted. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #17.
What
shared/metrics.py): subscription verifications, query-limit hits, NB API errors, token-refresh failures, agent latency.template.yaml): elevated error rate, Stripe webhook failures, token-refresh failures, "nation not found" spikes.shared/observability.py): Sentry wired into all Lambda handlers; no-ops whenSENTRY_DSNis unset (so it's safe to deploy before provisioning Sentry — DSN goes in under Consolidate to one domain, replace placeholder URLs, load real secrets #14 secrets).Verified
pytest -q→ 556 passed (= 542 on clean main + 14 new; no existing tests lost — confirmed by per-file collection diff)mypy src/→ clean (24 files)sentry-sdk>=2.0.0to requirements.txt (optional at runtime)Provenance
Authored by a pr-driver Opus drive that hit the 80-turn ceiling before committing; working tree preserved, independently re-verified (tests + mypy + collection-count audit) from the main session, then committed and opened here. No adversarial self-review ran — recommend a careful
/shipreview of the metric/alarm wiring and the Sentry init path before merge.