Skip to content

Add environment coherence guard: verify PROMETHEUS_URL and OPENCOST_URL observe the same cluster #95

Description

@e-300

Problem:
pkg/prometheus/client.go hardcodes the demo Prometheus as a silent fallback when PROMETHEUS_URL is unset. A misconfigured run compares the local OpenCost API against demo metrics, producing false failures, or worse, false passes against the wrong cluster.
This caused false failures the team hit while running the suite locally.
Nothing in the current tickets covers it: #74's smoke tests verify each endpoint is reachable, but not that the two endpoints observe the same cluster.

Proposed test:

  1. Fail fast if either env var is unset (refuse the silent fallback).
  2. Query both endpoints for the same fixed, completed historical window — kube_pod_container_status_running from Prometheus, /allocation aggregated by pod — using a shared end timestamp.
  3. Filter synthetic entries (idle, *-unmounted-pvcs), compare pod-name sets, require overlap above a threshold (~50%; same-cluster is ~90%+, different-cluster is ~0%, so churn can't cross the boundary).
  4. t.Fatal on mismatch, printing both URLs and sample pods from each side — if environments are incoherent, all downstream ground-truth results are meaningless.
    Fresh clusters with no scrape history fall back to a short recent window. Shares window-alignment machinery with Stabilize pod-count test: shared timestamp, churn tolerance, retries #94. Runs before the rest of the suite as a precondition guard.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions