Add local k3d dev stack for integration tests

.gitignore

@@ -0,0 +1 @@
+dev/logs/

README.md

@@ -14,6 +14,15 @@ git submodule update --init --recursive
 go mod download
 ```
 
+## Local Dev Stack
+
+For a reproducible k3d-based environment (Prometheus + OpenCost + seed workloads), see [dev/README.md](dev/README.md).
+
+```sh
+./dev/up.sh
+./dev/scripts/port-forward.sh   # separate terminal
+```
+
 ## Running Tests Locally
 
 First, you'll need to configure access to a running instance of OpenCost.

dev/README.md

@@ -0,0 +1,118 @@
+# Local OpenCost Dev Stack
+
+Scripted local environment for running OpenCost integration tests against a k3d cluster with Prometheus, OpenCost, and seed workloads.
+
+## Prerequisites
+
+- [Docker](https://docs.docker.com/get-docker/)
+- [k3d](https://k3d.io/) v5+
+- [kubectl](https://kubernetes.io/docs/tasks/tools/)
+- [helm](https://helm.sh/docs/intro/install/) v3+
+- Go (see root `go.mod`)
+
+Clone submodules and download dependencies from the repo root:
+
+```sh
+git submodule update --init --recursive
+go mod download
+```
+
+## Quick start
+
+```sh
+# Start the stack
+./dev/up.sh
+
+# In another terminal: port-forward services
+./dev/scripts/port-forward.sh
+
+# Export env vars for tests
+export OPENCOST_URL='http://localhost:9003'
+export PROMETHEUS_URL='http://localhost:9090'
+export OPENCOST_MCP_URL='http://localhost:8081'
+
+# Run a representative test suite
+./test/bats/bin/bats ./test/integration/query/count/test.bats
+```
+
+Tear down:
+
+```sh
+./dev/down.sh
+```
+
+## What gets deployed
+
+| Component | Namespace | Notes |
+|-----------|-----------|-------|
+| k3d cluster `opencost-dev` | — | 1 server + 2 agents |
+| Prometheus | `prometheus-system` | Includes kube-state-metrics with annotation allowlist |
+| OpenCost | `opencost` | UI, MCP, custom pricing enabled |
+| Seed workloads | `workload-test` | nginx Deployment + web StatefulSet with test annotations |
+
+Helm values live in `dev/helm/`. Workload manifests live in `dev/manifests/workloads/`.
+
+## Helper scripts
+
+| Script | Purpose |
+|--------|---------|
+| `dev/scripts/port-forward.sh` | Forward Prometheus (9090) and OpenCost (9003, 8081) |
+| `dev/scripts/deploy-workloads.sh` | Apply seed workloads |
+| `dev/scripts/toggle-prometheus.sh on\|off` | Scale Prometheus server for outage/resilience testing |
+| `dev/scripts/collect-logs.sh [dir]` | Dump pod status and logs for debugging |
+
+## Running integration tests
+
+After port-forwarding and exporting env vars:
+
+```sh
+# Recommended smoke tests
+./test/bats/bin/bats ./test/integration/query/count/test.bats
+
+# Broader allocation API tests
+./test/bats/bin/bats ./test/integration/api/allocation/test.bats
+```
+
+### Local test matrix (k3d)
+
+| Suite | Expected on local stack |
+|-------|-------------------------|
+| `query/count` | Pass |
+| `api/allocation` — Pod/Namespace labels | Pass |
+| `api/allocation` — Pod/Namespace annotations | Pass after kube-state-metrics allowlist is applied and metrics have scraped (~5–10 min) |
+| `api/allocation` — controller kind consistency | May fail (OpenCost 500 on `controllerKind` filter on small clusters) |
+| `api/allocation` — negative idle costs | May fail (`__idle__` missing on small/fresh clusters) |
+| Demo-only tests (GPU, cloud LB, etc.) | Skip or fail — use `demo.infra.opencost.io` for those |
+
+Verify annotation metrics are flowing:
+
+```sh
+curl -s 'http://localhost:9090/api/v1/query?query=kube_pod_annotations' | head -c 500
+kubectl logs -n prometheus-system -l app.kubernetes.io/name=kube-state-metrics | grep -i allowlist
+```
+
+## Troubleshooting
+
+**Prometheus annotation metrics empty**
+
+Re-apply Helm values and wait for kube-state-metrics to restart:
+
+```sh
+helm upgrade --install prometheus prometheus-community/prometheus \
+  -n prometheus-system -f dev/helm/prometheus-values.yaml
+```
+
+**Port 9090 already in use**
+
+Stop the conflicting process or change the local port in `port-forward.sh`.
+
+**OpenCost pods not ready**
+
+```sh
+kubectl get pods -A
+./dev/scripts/collect-logs.sh
+```
+
+**MCP health check**
+
+`GET /healthz` on port 8081 may require an MCP session header — that is expected. Tests use the allocation API on port 9003.

dev/down.sh

@@ -0,0 +1,3 @@
+#!/usr/bin/env bash
+set -euo pipefail
+k3d cluster delete opencost-dev

dev/helm/opencost-values.yaml

@@ -0,0 +1,17 @@
+opencost:
+  prometheus:
+    internal: #where the Prometheus server is deployed
+      enabled: true
+      namespaceName: prometheus-system
+      serviceName: prometheus-server
+  prometheusDataSource:
+    queryResolutionSeconds: 60 #how often to query the Prometheus server
+  exporter:
+    defaultClusterId: default-cluster #the default cluster ID
+  mcp:
+    enabled: true #enables the MCP server
+  ui:
+    enabled: true #enables the UI
+  customPricing:
+    enabled: true #enables the custom pricing
+    createConfigmap: true #creates a configmap for the custom pricing

dev/helm/prometheus-values.yaml

@@ -0,0 +1,10 @@
+server:
+  fullnameOverride: prometheus-server #overrides the default fullname of the Prometheus server
+#kube-state-metrics is a Prometheus exporter for Kubernetes state metrics
+kube-state-metrics:
+  enabled: true #enables the kube-state-metrics
+  metricAnnotationsAllowList: #allows all annotations to be collected
+    - "pods=[*]" #allows all pods to be collected
+    - "namespaces=[*]" #allows all namespaces to be collected
+  extraArgs:
+    - --metric-annotations-allowlist=pods=[*],namespaces=[*] #allows all annotations to be collected

dev/manifests/workloads/seed.yaml

@@ -0,0 +1,57 @@
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: workload-test
+  annotations:
+    opencost.io/test: "true"
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: nginx
+  namespace: workload-test
+spec:
+  replicas: 2
+  selector:
+    matchLabels:
+      app: nginx
+  template:
+    metadata:
+      labels:
+        app: nginx
+      annotations:
+        opencost.io/pod: "seed"
+    spec:
+      containers:
+      - name: nginx
+        image: nginx:stable
+        resources:
+          requests:
+            cpu: 100m
+            memory: 128Mi
+---
+apiVersion: apps/v1
+kind: StatefulSet
+metadata:
+  name: web
+  namespace: workload-test
+spec:
+  serviceName: web
+  replicas: 1
+  selector:
+    matchLabels:
+      app: web
+  template:
+    metadata:
+      labels:
+        app: web
+      annotations:
+        opencost.io/pod: "stateful"
+    spec:
+      containers:
+      - name: nginx
+        image: nginx:stable
+        resources:
+          requests:
+            cpu: 50m
+            memory: 64Mi

dev/scripts/collect-logs.sh

@@ -0,0 +1,26 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
+OUT_DIR="${1:-$REPO_ROOT/dev/logs/$(date +%Y%m%d-%H%M%S)}"
+
+mkdir -p "$OUT_DIR"
+
+echo "Collecting logs to $OUT_DIR"
+
+kubectl get pods -A -o wide >"$OUT_DIR/pods.txt" 2>&1 || true
+
+for ns in opencost prometheus-system workload-test kube-system; do
+  kubectl get all -n "$ns" >"$OUT_DIR/${ns}-resources.txt" 2>&1 || true
+done
+
+kubectl logs -n opencost -l app.kubernetes.io/instance=opencost --all-containers --tail=-1 \
+  >"$OUT_DIR/opencost.log" 2>&1 || true
+
+kubectl logs -n prometheus-system -l app.kubernetes.io/name=prometheus --all-containers --tail=-1 \
+  >"$OUT_DIR/prometheus-server.log" 2>&1 || true
+
+kubectl logs -n prometheus-system -l app.kubernetes.io/name=kube-state-metrics --all-containers --tail=-1 \
+  >"$OUT_DIR/kube-state-metrics.log" 2>&1 || true
+
+echo "Done. Logs written to $OUT_DIR"

dev/scripts/deploy-workloads.sh

@@ -0,0 +1,7 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
+
+kubectl apply -f "$REPO_ROOT/dev/manifests/workloads/"
+echo "Seed workloads applied from dev/manifests/workloads/"

dev/scripts/port-forward.sh

@@ -0,0 +1,20 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+echo "Forwarding Prometheus  -> localhost:9090"
+echo "Forwarding OpenCost    -> localhost:9003 (API), localhost:8081 (MCP)"
+echo "Press Ctrl+C to stop."
+echo ""
+
+kubectl port-forward -n prometheus-system svc/prometheus-server 9090:80 &
+PROM_PID=$!
+
+kubectl port-forward -n opencost svc/opencost 9003:9003 8081:8081 &
+OC_PID=$!
+
+cleanup() {
+  kill "$PROM_PID" "$OC_PID" 2>/dev/null || true
+}
+trap cleanup EXIT INT TERM
+
+wait

dev/scripts/toggle-prometheus.sh

@@ -0,0 +1,28 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+NAMESPACE=prometheus-system
+DEPLOYMENT=prometheus-server
+
+usage() {
+  echo "Usage: $0 [on|off]"
+  echo "  on  - scale Prometheus server to 1 replica"
+  echo "  off - scale Prometheus server to 0 replicas"
+  exit 1
+}
+
+STATE="${1:-}"
+case "$STATE" in
+  on)
+    REPLICAS=1
+    ;;
+  off)
+    REPLICAS=0
+    ;;
+  *)
+    usage
+    ;;
+esac
+
+kubectl scale deployment "$DEPLOYMENT" -n "$NAMESPACE" --replicas="$REPLICAS"
+echo "Prometheus $DEPLOYMENT scaled to $REPLICAS replica(s) in $NAMESPACE"

dev/up.sh

@@ -0,0 +1,46 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+CLUSTER_NAME=opencost-dev
+REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+
+helm repo add prometheus-community https://prometheus-community.github.io/helm-charts 2>/dev/null || true
+helm repo add opencost https://opencost.github.io/opencost-helm-chart 2>/dev/null || true
+helm repo update
+
+if ! k3d cluster list | grep -q "$CLUSTER_NAME"; then
+  k3d cluster create "$CLUSTER_NAME" --agents 2
+fi
+
+kubectl create namespace prometheus-system --dry-run=client -o yaml | kubectl apply -f -
+kubectl create namespace opencost --dry-run=client -o yaml | kubectl apply -f -
+
+helm upgrade --install prometheus prometheus-community/prometheus \
+  -n prometheus-system \
+  -f "$REPO_ROOT/dev/helm/prometheus-values.yaml"
+
+helm upgrade --install opencost opencost/opencost \
+  -n opencost \
+  -f "$REPO_ROOT/dev/helm/opencost-values.yaml"
+
+echo "Waiting for core deployments..."
+kubectl wait --for=condition=available deployment/prometheus-server \
+  -n prometheus-system --timeout=300s
+kubectl wait --for=condition=available deployment/opencost \
+  -n opencost --timeout=300s
+
+"$REPO_ROOT/dev/scripts/deploy-workloads.sh"
+
+echo ""
+echo "Stack is up. Check pods: kubectl get pods -A"
+echo ""
+echo "Port-forward (in a separate terminal):"
+echo "  $REPO_ROOT/dev/scripts/port-forward.sh"
+echo ""
+echo "Export test env vars:"
+echo "  export OPENCOST_URL='http://localhost:9003'"
+echo "  export PROMETHEUS_URL='http://localhost:9090'"
+echo "  export OPENCOST_MCP_URL='http://localhost:8081'"
+echo ""
+echo "Run tests:"
+echo "  ./test/bats/bin/bats ./test/integration/query/count/test.bats"