`runtime error: invalid memory address or nil pointer dereference` in v25.1.2

[chrisseto]

Reported in the community slack

Redpanda CR

apiVersion: cluster.redpanda.com/v1alpha2
kind: Redpanda
metadata:
  name: redpanda-example
  namespace: redpanda
spec:
  chartRef: {}
  clusterSpec:
    nameOverride: redpanda-example
    fullnameOverride: redpanda-example
    commonLabels:
      env: staging
    statefulset:
      replicas: 3
      podAntiAffinity:
        type: soft
        topologyKey: kubernetes.io/hostname
        weight: 100
    external:
      enabled: true
      service:
        enabled: true
      type: LoadBalancer
      domain: redpanda.staging.example.com
      externalDns:
        enabled: true
      annotations:
        service.beta.kubernetes.io/alibaba-cloud-loadbalancer-address-type: intranet
        service.beta.kubernetes.io/alibaba-cloud-loadbalancer-vswitch-id: redacted
    auth:
      sasl:
        enabled: true
        mechanism: SCRAM-SHA-512
        secretRef: "redpanda-superuser-credentials"
    storage:
      persistentVolume:
        enabled: true
        storageClass: alicloud-disk-essd
        size: 20Gi
    tls:
      enabled: false
    listeners:
      admin:
        tls:
          enabled: true
        external:
          default:
            enabled: false
      schemaRegistry:
        tls:
          enabled: true
        external:
          default:
            enabled: false
      kafka:
        tls:
          enabled: false
        external:
          default:
            enabled: true
            authenticationMethod: sasl
      rpc:
        tls:
          enabled: true
      http:
        enabled: false
        external:
          default:
            enabled: false
    logging:
      logLevel: debug
      usageStats:
        enabled: false
    resources:
      requests:
        cpu: 100m
        memory: 2Gi
      limits:
        memory: 2Gi
    monitoring:
      enabled: true
      scrapeInterval: 30s

    post_install_job:
      enabled: false
    post_upgrade_job:
      enabled: false

    # RedPanda Console
    console:
      enabled: true
      annotations:
        argocd.argoproj.io/sync-wave: "5"
      replicaCount: 1
      resources:
        requests:
          cpu: 10m
          memory:    ###16Mi

Stack Trace

{
    "level": "error",
    "ts": "2025-08-07T07:46:46.978Z",
    "msg": "error reconciling cluster config",
    "controller": "redpanda",
    "controllerGroup": "cluster.redpanda.com",
    "controllerKind": "Redpanda",
    "Redpanda": {
        "name": "redpanda-sinbad",
        "namespace": "redpanda"
    },
    "namespace": "redpanda",
    "name": "redpanda-sinbad",
    "reconcileID": "49e23fd2-1c2c-4cbe-9ce7-b2c8237d5119",
    "error": "recovered panic: runtime error: invalid memory address or nil pointer dereference\n(1) runtime error: invalid memory address or nil pointer dereference\nError types: (1) runtime.errorString",
    "errorVerbose": "recovered panic: runtime error: invalid memory address or nil pointer dereference\n(1) attached stack trace\n  -- stack trace:\n  | github.com/redpanda-data/redpanda-operator/operator/internal/controller/redpanda.(*RedpandaReconciler).reconcileClusterConfig\n  | \t/work/operator/internal/controller/redpanda/redpanda_controller.go:596\n  | [...repeated from below...]\nWraps: (2) attached stack trace\n  -- stack trace:\n  | github.com/redpanda-data/redpanda-operator/operator/internal/controller/redpanda.(*RedpandaReconciler).clusterConfigFor.func1\n  | \t/work/operator/internal/controller/redpanda/redpanda_controller.go:648\n  | runtime.gopanic\n  | \t/nix/store/5xvi25nqmbrg58aixp4zgczilfnp7pwg-go-1.24.3/share/go/src/runtime/panic.go:792\n  | runtime.panicmem\n  | \t/nix/store/5xvi25nqmbrg58aixp4zgczilfnp7pwg-go-1.24.3/share/go/src/runtime/panic.go:262\n  | runtime.sigpanic\n  | \t/nix/store/5xvi25nqmbrg58aixp4zgczilfnp7pwg-go-1.24.3/share/go/src/runtime/signal_unix.go:925\n  | github.com/redpanda-data/redpanda-operator/operator/internal/controller/redpanda.(*RedpandaReconciler).clusterConfigFor\n  | \t/work/operator/internal/controller/redpanda/redpanda_controller.go:670\n  | github.com/redpanda-data/redpanda-operator/operator/internal/controller/redpanda.(*RedpandaReconciler).reconcileClusterConfig\n  | \t/work/operator/internal/controller/redpanda/redpanda_controller.go:594\n  | github.com/redpanda-data/redpanda-operator/operator/internal/controller/redpanda.(*RedpandaReconciler).Reconcile\n  | \t/work/operator/internal/controller/redpanda/redpanda_controller.go:286\n  | sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n  | \t/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:119\n  | sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n  | \t/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:334\n  | sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n  | \t/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:294\n  | sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n  | \t/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:255\n  | runtime.goexit\n  | \t/nix/store/5xvi25nqmbrg58aixp4zgczilfnp7pwg-go-1.24.3/share/go/src/runtime/asm_amd64.s:1700\nWraps: (3) secondary error attachment\n  | runtime error: invalid memory address or nil pointer dereference\n  | (1) runtime error: invalid memory address or nil pointer dereference\n  | Error types: (1) runtime.errorString\nWraps: (4) recovered panic: runtime error: invalid memory address or nil pointer dereference\n  | (1) runtime error: invalid memory address or nil pointer dereference\n  | Error types: (1) runtime.errorString\nError types: (1) *withstack.withStack (2) *withstack.withStack (3) *secondary.withSecondaryError (4) *errutil.leafError",
    "stacktrace": "github.com/go-logr/logr.Logger.Error\n\t/root/go/pkg/mod/github.com/go-logr/logr@v1.4.2/logr.go:301\ngithub.com/redpanda-data/redpanda-operator/operator/internal/controller/redpanda.(*RedpandaReconciler).Reconcile\n\t/work/operator/internal/controller/redpanda/redpanda_controller.go:290\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:334\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:294\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:255"
}

Additional Details:

• I have deployed v25.1.1 of the RedPanda Operator Helm chart using ArgoCD
• I am running the Operator with image tag v25.1.2: v25.1.2@sha256:a905ec74dfa71a6a0967dae7d1bbbeeaff96139c8306ad28b90eefe09969b4f4"
• I've pulled the CRDs from here and installed them outside of the Helm chart (with ArgoCD using Kustomize)

[chrisseto]

My current suspicion is that the following block is what's causing this issue.

   post_install_job:
      enabled: false
    post_upgrade_job:
      enabled: false

The cluster configuration syncer is currently using the post_install_job to power cluster config reconcilation. Disabling it is likely what's causing the nil dereference.

EDIT: This was indeed the issue! The work around is to remove any post_install_job/post_upgrade_job references.