PLT-2119: Replace per-tag PSLs with single tag-agnostic PrometheusServiceLevel

[eddiebarth1]

Closes PLT-2119

Problem

Per-deploy PrometheusServiceLevel objects ({app}-{target}-{tag}-servicelevels) caused Sloth to regenerate burn-rate recording rules on every deploy, resetting 1h/6h windows and producing duplicate alert series. The result: chronic SLO violations fired N times (once per revision) and burn-rate trends were untrustworthy.

What changed

• One shared PSL per app/target ({app}-{target}-servicelevels) — stable across deploys.
• Tag-preserving SLI queries: SLI events use sum by (tag) (rate(...)) instead of sum(rate(...)). Sloth wraps these without additional aggregation and its burn-rate alert template (max(...) without (sloth_window)) preserves the tag label through to the alert series. Per-revision rollback granularity is preserved at the Prometheus query layer rather than via per-deploy Kubernetes objects.
• Single writer: ResourceSyncer.syncServiceLevels upserts the PSL once per reconcile from the leading releasable revision (mirrors syncSLORules pattern). No per-incarnation churn, no multi-writer oscillation when SLO config differs between concurrent revisions.
• Rolling migration: cleanupLegacyServiceLevels runs on each Releasing/Canarying reconcile, scrubbing any legacy per-tag PSL for the current revision. Self-limiting (no-op once cleaned); handles infrequently-deployed services without coordinated rollout.
• Retirement cleanup: DeleteServiceLevels wired into Deleting/Failing to remove the shared PSL when an app/target is fully retired.

Canary/rollback behavior (unchanged from main)

IsRevisionTriggered matches alert samples by sample.Metric["tag"]. Because the new SLI queries preserve tag through Sloth's recording rules and burn-rate alerts, per-revision SLO-driven rollback continues to work exactly as it did before this change. No changes to prometheus/api.go.

Validation

• tag is the universal convention in mono SLI queries — confirmed across rito, lite, medium2, tutu, echo, bevy
• Sloth wraps SLI as (error)/(total) with no additional aggregation (slok/sloth/internal/plugin/slo/core/sli_rules_v1/plugin.go)
• Sloth burn-rate alert template max(...) without (sloth_window) preserves all labels except window (slok/sloth/internal/plugin/slo/core/alert_rules_v1/plugin.go)

Test plan

• go test ./...
• go vet ./..., make lint
• Staging: confirm one PSL per app/target, recording rules include sum by (tag), alerts carry tag=<revision>
• Verify burn-rate windows survive a deploy without resetting
• Force a canary violation, confirm rollback fires for the offending tag

🤖 Generated with Claude Code