StatusPods checks all pods cluster-wide, causing false failures from unrelated test debris

[floatingman]

Problem

The StatusPods function in extensions/workloads/pods/pod_status.go lists all pods across all namespaces on a cluster and treats any pod in Failed phase with a non-nil Terminated status as an error. This causes false test failures when:

1. Helm operation jobs (kubectl-* pods) complete and enter Error state — Rancher deploys multiple helm-operation jobs for the same task; if one succeeds, the others end in a terminated/error state
2. Previous test suites leave behind namespaces with failed pods (e.g., longhorn-system with CrashLoopBackOff pods)
3. Multiple test suites run concurrently or sequentially against the same cluster

Impact

In rancher/tests, running pit.daily tagged tests sequentially causes fleet tests to fail with pods are not healthy in <cluster> because chart tests (monitoring, longhorn, istio) leave Error-state pods on the downstream cluster. The fleet VerifyGitRepo function calls StatusPods as a final validation step, and it finds these orphaned pods.

When the downstream cluster is cleaned up (deleting Failed pods and stale namespaces), the same fleet tests pass in ~150s that previously timed out after 900+ seconds.

Reproduction

1. Run chart tests (longhorn, monitoring, istio) against a downstream cluster
2. Without cleanup, run fleet tests against the same cluster
3. Fleet tests fail with pods are not healthy in <cluster>
4. Clean up: kubectl delete pods --field-selector=status.phase=Failed -A
5. Re-run fleet tests — they pass

Root Cause

StatusPods calls steveClient.List(nil) with no namespace filter, then IsPodReady returns errors for any PodFailed pod with a non-nil Terminated status — including completed Helm job pods and debris from other test suites.

// pod_status.go:34-36
pods, err := steveClient.List(nil)  // lists ALL pods across ALL namespaces
// ...
for _, pod := range pods.Data {
    isReady, err := IsPodReady(&pod)  // treats Failed pods as errors

Proposed Solution

Add a StatusPodsInNamespace variant (or a namespaceFilter parameter to StatusPods) that only checks pods in namespaces relevant to the test. For example:

// Option A: Namespace-scoped variant
func StatusPodsInNamespace(client *rancher.Client, clusterID string, namespace string) []error

// Option B: Options pattern with label selectors
type StatusPodsOptions struct {
    Namespace     string
    LabelSelector string
}
func StatusPodsWithOptions(client *rancher.Client, clusterID string, opts *StatusPodsOptions) []error

Alternative/Additional Fix

In IsPodReady, the logic for PodFailed pods could skip pods from well-known system namespaces (e.g., kube-system) or pods with specific labels (e.g., helm-operation jobs), since these are expected transient failures from Rancher's Helm controller.

Environment

• shepherd version: v0.0.0-20260430211500-1f50d155268e
• Discovered in: rancher/tests pit.daily test suite