From 027d19630e3d7633d0f349ae696c6ce80fb41d12 Mon Sep 17 00:00:00 2001 From: JakeSCahill Date: Mon, 27 Jul 2026 10:46:24 +0100 Subject: [PATCH 1/2] docs: regenerate CRD reference from operator main with the Pipeline CRD Regenerated with the same GA ignore config as before. Operator main now includes the merged Pipeline CRD (redpanda-operator#1677), which ships as a beta feature of the 26.2 operator, so the reference gains the Pipeline types alongside the existing Stretch Cluster and ShadowLink types. --- modules/reference/pages/k-crd.adoc | 500 +++++++++++++++++++++++++---- 1 file changed, 445 insertions(+), 55 deletions(-) diff --git a/modules/reference/pages/k-crd.adoc b/modules/reference/pages/k-crd.adoc index f102b181c2..cbea5e6444 100644 --- a/modules/reference/pages/k-crd.adoc +++ b/modules/reference/pages/k-crd.adoc @@ -9,6 +9,7 @@ .Resource Types - xref:{anchor_prefix}-github-com-redpanda-data-redpanda-operator-operator-api-redpanda-v1alpha2-console[$$Console$$] - xref:{anchor_prefix}-github-com-redpanda-data-redpanda-operator-operator-api-redpanda-v1alpha2-group[$$Group$$] +- xref:{anchor_prefix}-github-com-redpanda-data-redpanda-operator-operator-api-redpanda-v1alpha2-pipeline[$$Pipeline$$] - xref:{anchor_prefix}-github-com-redpanda-data-redpanda-operator-operator-api-redpanda-v1alpha2-redpanda[$$Redpanda$$] - xref:{anchor_prefix}-github-com-redpanda-data-redpanda-operator-operator-api-redpanda-v1alpha2-redpandabrokerpool[$$RedpandaBrokerPool$$] - xref:{anchor_prefix}-github-com-redpanda-data-redpanda-operator-operator-api-redpanda-v1alpha2-redpandarole[$$RedpandaRole$$] @@ -276,7 +277,7 @@ Redpanda will use the `internal_topic_replication_factor` cluster config value. [id="{anchor_prefix}-github-com-redpanda-data-redpanda-operator-operator-api-redpanda-v1alpha2-auth"] == Auth -Auth configures authentication in the Helm values. See xref:manage:kubernetes/security/authentication/sasl-kubernetes.adoc[] +Auth configures authentication in the Helm values. See xref:manage:kubernetes/security/authentication/sasl-kubernetes.adoc[]. @@ -360,9 +361,29 @@ BootstrapUser configures the user used to bootstrap Redpanda when SASL is enable | Field | Description | *`name`* __string__ | Name specifies the name of the bootstrap user created for the cluster, if unspecified + defaults to "kubernetes-controller". + +NOTE: for the StretchCluster kind this field is currently ignored; the bootstrap + +username is fixed to "kubernetes-controller". Only secretKeyRef is honored there. + | *`secretKeyRef`* __link:https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.28/#secretkeyselector-v1-core[$$SecretKeySelector$$]__ | Specifies the location where the generated password will be written or a pre-existing + -password will be read from. + +password will be read from. For the StretchCluster kind the referenced key is required + +(there is no default key at the CRD level). + + + +For the StretchCluster kind the referenced Secret is read and replicated by the + +operator to every member cluster in the same namespace, so whoever can set this field + +is trusted to name a Secret whose contents may be copied across all member clusters of + +the mesh. The operator never overwrites an existing Secret of that name on a member + +cluster. + + + +IMPORTANT: for the StretchCluster kind the bootstrap password is consumed only when the + +cluster is first bootstrapped and cannot be rotated afterwards by editing this Secret, + +changing secretKeyRef to a different location, or removing secretKeyRef. Redpanda keeps + +the SCRAM credential it was bootstrapped with, so changing the source after bootstrap + +makes the operator and brokers authenticate with a credential the cluster never learned. + +Rotate the bootstrap user out of band (via the admin API) instead. + | *`mechanism`* __string__ | Specifies the authentication mechanism to use for the bootstrap user. Options are `SCRAM-SHA-256` and `SCRAM-SHA-512`. + +NOTE: for the StretchCluster kind this field is currently ignored; the mechanism is + +taken from sasl.mechanism. + |=== @@ -386,7 +407,7 @@ Budget configures the management of disruptions affecting the Pods in the Statef [id="{anchor_prefix}-github-com-redpanda-data-redpanda-operator-operator-api-redpanda-v1alpha2-cpu"] == CPU -CPU configures CPU resources for containers. See xref:manage:kubernetes/manage-resources.adoc[] +CPU configures CPU resources for containers. See xref:manage:kubernetes/manage-resources.adoc[]. @@ -397,7 +418,7 @@ CPU configures CPU resources for containers. See xref:manage:kubernetes/manage-r [cols="25a,75a", options="header"] |=== | Field | Description -| *`cores`* __link:https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.28/#quantity-resource-api[$$Quantity$$]__ | Specifies the number of CPU cores available to the application. Redpanda makes use of a thread per core model. For details, see xref:get-started:architecture.adoc[] For this reason, Redpanda should only be given full cores. Note: You can increase cores, but decreasing cores is not currently supported. See the GitHub issue:https://github.com/redpanda-data/redpanda/issues/350. This setting is equivalent to `--smp`, `resources.requests.cpu`, and `resources.limits.cpu`. For production, use `4` or greater. + +| *`cores`* __link:https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.28/#quantity-resource-api[$$Quantity$$]__ | Specifies the number of CPU cores available to the application. Redpanda makes use of a thread per core model. For details, see xref:get-started:architecture.adoc[]. For this reason, Redpanda should only be given full cores. Note: You can increase cores, but decreasing cores is not currently supported. See the GitHub issue:https://github.com/redpanda-data/redpanda/issues/350. This setting is equivalent to `--smp`, `resources.requests.cpu`, and `resources.limits.cpu`. For production, use `4` or greater. + | *`overprovisioned`* __boolean__ | Specifies whether Redpanda assumes it has all of the provisioned CPU. This should be `true` unless the container has CPU affinity. Equivalent to: `--idle-poll-time-us 0`, `--thread-affinity 0`, and `--poll-aio 0`. If the value of full cores in `resources.cpu.cores` is less than `1`, this setting is set to `true`. + |=== @@ -520,6 +541,7 @@ ClusterSource defines how to connect to a particular Redpanda cluster. .Appears in: - xref:{anchor_prefix}-github-com-redpanda-data-redpanda-operator-operator-api-redpanda-v1alpha2-consolespec[$$ConsoleSpec$$] - xref:{anchor_prefix}-github-com-redpanda-data-redpanda-operator-operator-api-redpanda-v1alpha2-groupspec[$$GroupSpec$$] +- xref:{anchor_prefix}-github-com-redpanda-data-redpanda-operator-operator-api-redpanda-v1alpha2-pipelinespec[$$PipelineSpec$$] - xref:{anchor_prefix}-github-com-redpanda-data-redpanda-operator-operator-api-redpanda-v1alpha2-rolespec[$$RoleSpec$$] - xref:{anchor_prefix}-github-com-redpanda-data-redpanda-operator-operator-api-redpanda-v1alpha2-schemaspec[$$SchemaSpec$$] - xref:{anchor_prefix}-github-com-redpanda-data-redpanda-operator-operator-api-redpanda-v1alpha2-shadowlinkspec[$$ShadowLinkSpec$$] @@ -595,15 +617,15 @@ Config configures Redpanda config properties supported by Redpanda that may not [cols="25a,75a", options="header"] |=== | Field | Description -| *`rpk`* __link:https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.28/#rawextension-runtime-pkg[$$RawExtension$$]__ | Specifies cluster configuration properties. See xref:reference:cluster-properties.adoc[] + -| *`cluster`* __link:https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.28/#rawextension-runtime-pkg[$$RawExtension$$]__ | Specifies cluster configuration properties. See xref:reference:cluster-properties.adoc[] + +| *`rpk`* __link:https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.28/#rawextension-runtime-pkg[$$RawExtension$$]__ | Specifies cluster configuration properties. See xref:reference:cluster-properties.adoc[]. + +| *`cluster`* __link:https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.28/#rawextension-runtime-pkg[$$RawExtension$$]__ | Specifies cluster configuration properties. See xref:reference:cluster-properties.adoc[]. + | *`extraClusterConfiguration`* __xref:{anchor_prefix}-github-com-redpanda-data-redpanda-operator-operator-api-redpanda-v1alpha2-clusterconfiguration[$$ClusterConfiguration$$]__ | Holds values (or references to values) that should be used to configure the cluster; these + are resolved late in order to avoid embedding secrets directly into bootstrap configurations + exposed as Kubernetes configmaps. + -| *`node`* __link:https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.28/#rawextension-runtime-pkg[$$RawExtension$$]__ | Specifies broker configuration properties. See xref:reference:node-properties.adoc[] + -| *`tunable`* __link:https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.28/#rawextension-runtime-pkg[$$RawExtension$$]__ | Specifies tunable configuration properties. See xref:reference:tunable-properties.adoc[] + -| *`schema_registry_client`* __link:https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.28/#rawextension-runtime-pkg[$$RawExtension$$]__ | Specifies tunable configuration properties. See xref:reference:tunable-properties.adoc[] + -| *`pandaproxy_client`* __link:https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.28/#rawextension-runtime-pkg[$$RawExtension$$]__ | Specifies tunable configuration properties. See xref:reference:tunable-properties.adoc[] + +| *`node`* __link:https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.28/#rawextension-runtime-pkg[$$RawExtension$$]__ | Specifies broker configuration properties. See xref:reference:node-properties.adoc[]. + +| *`tunable`* __link:https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.28/#rawextension-runtime-pkg[$$RawExtension$$]__ | Specifies tunable configuration properties. See xref:reference:tunable-properties.adoc[]. + +| *`schema_registry_client`* __link:https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.28/#rawextension-runtime-pkg[$$RawExtension$$]__ | Specifies tunable configuration properties. See xref:reference:tunable-properties.adoc[]. + +| *`pandaproxy_client`* __link:https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.28/#rawextension-runtime-pkg[$$RawExtension$$]__ | Specifies tunable configuration properties. See xref:reference:tunable-properties.adoc[]. + |=== @@ -711,7 +733,7 @@ the dynamic configuration, while its "synonym" will be the default. + [id="{anchor_prefix}-github-com-redpanda-data-redpanda-operator-operator-api-redpanda-v1alpha2-connectormonitoring"] == ConnectorMonitoring -ConnectorMonitoring configures monitoring resources for Connectors. See xref:manage:kubernetes/monitoring/monitor-redpanda.adoc[] +ConnectorMonitoring configures monitoring resources for Connectors. See xref:manage:kubernetes/monitoring/monitor-redpanda.adoc[]. @@ -1017,7 +1039,7 @@ on remote Kubernetes clusters. [id="{anchor_prefix}-github-com-redpanda-data-redpanda-operator-operator-api-redpanda-v1alpha2-enterprise"] == Enterprise -Enterprise configures an Enterprise license key to enable Redpanda Enterprise features. Requires the post-install job to be enabled (default). See xref:get-started:licenses.adoc[] +Enterprise configures an Enterprise license key to enable Redpanda Enterprise features. Requires the post-install job to be enabled (default). See xref:get-started:licenses.adoc[]. @@ -1799,7 +1821,7 @@ specified, this field takes precedence over [Certificate.CAEnabled]. + [id="{anchor_prefix}-github-com-redpanda-data-redpanda-operator-operator-api-redpanda-v1alpha2-listeners"] == Listeners -Listeners configures settings for listeners, including HTTP Proxy, Schema Registry, the Admin API and the Kafka API. See xref:manage:kubernetes/networking/configure-listeners.adoc[] +Listeners configures settings for listeners, including HTTP Proxy, Schema Registry, the Admin API and the Kafka API. See xref:manage:kubernetes/networking/configure-listeners.adoc[]. @@ -1842,7 +1864,7 @@ LivenessProbe configures liveness probes to monitor the health of the Pods and r [id="{anchor_prefix}-github-com-redpanda-data-redpanda-operator-operator-api-redpanda-v1alpha2-logging"] == Logging -Logging configures logging settings in the Helm values. See xref:manage:kubernetes/troubleshooting/troubleshoot.adoc[] +Logging configures logging settings in the Helm values. See xref:manage:kubernetes/troubleshooting/troubleshoot.adoc[]. @@ -1898,7 +1920,7 @@ MetadataTemplate defines additional metadata to associate with a resource. [id="{anchor_prefix}-github-com-redpanda-data-redpanda-operator-operator-api-redpanda-v1alpha2-monitoring"] == Monitoring -Monitoring configures monitoring resources for Redpanda. See xref:manage:kubernetes/monitoring/monitor-redpanda.adoc[] +Monitoring configures monitoring resources for Redpanda. See xref:manage:kubernetes/monitoring/monitor-redpanda.adoc[]. @@ -1965,6 +1987,28 @@ and `patternType` must be `literal` + |=== +[id="{anchor_prefix}-github-com-redpanda-data-redpanda-operator-operator-api-redpanda-v1alpha2-namedvaluesource"] +== NamedValueSource + +NamedValueSource binds a name to a value provider so the pipeline YAML +can reference it via ${NAME} interpolation. + + + +.Appears in: +- xref:{anchor_prefix}-github-com-redpanda-data-redpanda-operator-operator-api-redpanda-v1alpha2-pipelinespec[$$PipelineSpec$$] + +[cols="25a,75a", options="header"] +|=== +| Field | Description +| *`name`* __string__ | Name is the environment-variable name the pipeline YAML references. + +Must match standard env-var characters: [A-Z_][A-Z0-9_]*. + +| *`source`* __xref:{anchor_prefix}-github-com-redpanda-data-redpanda-operator-operator-api-redpanda-v1alpha2-valuesource[$$ValueSource$$]__ | Source is the value provider. Exactly one of inline / configMapKeyRef + +/ secretKeyRef / externalSecretRef must be set; the ValueSource + +XValidation rules enforce this. + +|=== + + [id="{anchor_prefix}-github-com-redpanda-data-redpanda-operator-operator-api-redpanda-v1alpha2-networking"] == Networking @@ -2107,6 +2151,351 @@ PersistentVolume configures configurations for a PersistentVolumeClaim to use to |=== +[id="{anchor_prefix}-github-com-redpanda-data-redpanda-operator-operator-api-redpanda-v1alpha2-pipeline"] +== Pipeline + +Pipeline defines a Redpanda Connect pipeline managed by the operator. + + + + + +[cols="25a,75a", options="header"] +|=== +| Field | Description +| *`apiVersion`* __string__ | `cluster.redpanda.com/v1alpha2` +| *`kind`* __string__ | `Pipeline` +| *`kind`* __string__ | Kind is a string value representing the REST resource this object represents. + +Servers may infer this from the endpoint the client submits requests to. + +Cannot be updated. + +In CamelCase. + +More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds + +| *`apiVersion`* __string__ | APIVersion defines the versioned schema of this representation of an object. + +Servers should convert recognized schemas to the latest internal value, and + +may reject unrecognized values. + +More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources + +| *`metadata`* __link:https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.28/#objectmeta-v1-meta[$$ObjectMeta$$]__ | Refer to the Kubernetes API documentation for fields of `metadata`. + +| *`spec`* __xref:{anchor_prefix}-github-com-redpanda-data-redpanda-operator-operator-api-redpanda-v1alpha2-pipelinespec[$$PipelineSpec$$]__ | Spec defines the desired state of the Pipeline. + +| *`status`* __xref:{anchor_prefix}-github-com-redpanda-data-redpanda-operator-operator-api-redpanda-v1alpha2-pipelinestatus[$$PipelineStatus$$]__ | Status represents the current observed state of the Pipeline. + +|=== + + +[id="{anchor_prefix}-github-com-redpanda-data-redpanda-operator-operator-api-redpanda-v1alpha2-pipelinebudget"] +== PipelineBudget + +PipelineBudget configures a PodDisruptionBudget for the pipeline. + + + +.Appears in: +- xref:{anchor_prefix}-github-com-redpanda-data-redpanda-operator-operator-api-redpanda-v1alpha2-pipelinespec[$$PipelineSpec$$] + +[cols="25a,75a", options="header"] +|=== +| Field | Description +| *`maxUnavailable`* __integer__ | MaxUnavailable defines the maximum number of pipeline pods that can be + +unavailable during a voluntary disruption. Defaults to 1 if not set. + +|=== + + +[id="{anchor_prefix}-github-com-redpanda-data-redpanda-operator-operator-api-redpanda-v1alpha2-pipelinephase"] +== PipelinePhase (string) + +PipelinePhase describes the lifecycle phase of a Pipeline. + + + +.Appears in: +- xref:{anchor_prefix}-github-com-redpanda-data-redpanda-operator-operator-api-redpanda-v1alpha2-pipelinestatus[$$PipelineStatus$$] + + + +[id="{anchor_prefix}-github-com-redpanda-data-redpanda-operator-operator-api-redpanda-v1alpha2-pipelinespec"] +== PipelineSpec + +PipelineSpec defines the desired state of a Redpanda Connect pipeline. + + + +.Appears in: +- xref:{anchor_prefix}-github-com-redpanda-data-redpanda-operator-operator-api-redpanda-v1alpha2-pipeline[$$Pipeline$$] + +[cols="25a,75a", options="header"] +|=== +| Field | Description +| *`configYaml`* __string__ | ConfigYAML is the user-supplied Redpanda Connect pipeline YAML. + +Reference cluster-bound or sensitive values from .valueSources via + +${NAME} interpolation; the operator resolves them at render time. + + + +When .cluster is set, the operator inline-merges connection fields + +(seed_brokers, tls, sasl) into any `input.redpanda` and + +`output.redpanda` blocks in this YAML, derived from the resolved + +cluster connection and .userRef. Users only need to write the + +per-plugin fields (topic, key, consumer_group, etc.); brokers, TLS, + +and SASL are filled in by the operator. + + + +User-side keys win on conflict — set a key explicitly (for example, + +seed_brokers pointing at a different cluster) and the operator's + +generated value is skipped for that key. + + + +The merge targets the `redpanda` input/output plugins specifically. + +Any `redpanda_common` blocks the user authors are passed through + +unchanged — the operator does not inject connection fields into + +them. + +| *`displayName`* __string__ | DisplayName is a human-readable name for the pipeline. + +Maps to the pipeline display name when migrating to Redpanda Cloud. + +| *`description`* __string__ | Description is an optional description of what this pipeline does. + +Maps to the pipeline description when migrating to Redpanda Cloud. + +| *`tags`* __object (keys:string, values:string)__ | Tags are key-value pairs for organizing and filtering pipelines. + +Maps to pipeline tags when migrating to Redpanda Cloud. + +| *`configFiles`* __object (keys:string, values:string)__ | ConfigFiles defines additional configuration files to mount alongside + +the main pipeline configuration. Each entry maps a filename to its content. + +Files are mounted in the /config directory alongside connect.yaml. + +The key "connect.yaml" is reserved and cannot be used. + +Maps to pipeline config files when migrating to Redpanda Cloud. + +| *`replicas`* __integer__ | Replicas is the number of pipeline replicas to run. + + + +This field backs the Pipeline's scale subresource, so `kubectl scale + +pipeline/`, HorizontalPodAutoscaler, and KEDA ScaledObjects all + +read and write it through `/scale`. To autoscale a pipeline, point the + +autoscaler at the Pipeline itself — NOT at its Deployment, whose + +replica count the operator continuously resets to this field: + + + +scaleTargetRef: + +apiVersion: cluster.redpanda.com/v1alpha2 + +kind: Pipeline + +name: + + + +CPU and memory HPAs work out of the box: pipeline pods always carry + +resource requests (operator defaults apply when .resources is unset). + +To scale on the metrics Redpanda Connect itself emits (input_received, + +output_sent, processor_latency_ns, ...) feed them to the autoscaler — + +e.g. scrape the pods' `http` port at /metrics (the operator's + +monitoring PodMonitor does this) into Prometheus, then use + +prometheus-adapter for HPA custom metrics or a KEDA prometheus + +trigger. + + + +When an autoscaler manages this field, omit it from applied manifests + +(or have your GitOps tool ignore it) so config syncs don't undo the + +autoscaler's writes. + +| *`image`* __string__ | Image is the container image for the Redpanda Connect deployment. + +| *`serviceAccountName`* __string__ | ServiceAccountName is the ServiceAccount to bind to the pipeline pod. + +When unset, the namespace's default ServiceAccount is used. + + + +Setting this is the recommended way to scope per-pipeline cloud-IAM + +trust (e.g. IRSA on EKS, Workload Identity on GKE, Pod Identity on + +AKS). Annotating the namespace's default SA works but grants every + +pipeline in the namespace the same role — naming a Pipeline-specific + +SA here keeps the trust boundary per-pipeline. + + + +The operator does NOT create the ServiceAccount; provision it + +(along with the appropriate cloud-IAM annotations) out-of-band. + +| *`paused`* __boolean__ | Paused stops the pipeline by scaling replicas to zero when set to true. + + + +Paused wins over .replicas, including values an autoscaler writes + +through the scale subresource: while paused the Deployment stays at + +zero (the pipeline reports Stopped) even if HPA/KEDA keep updating + +.replicas, and unpausing resumes at the current .replicas count. + +| *`resources`* __link:https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.28/#resourcerequirements-v1-core[$$ResourceRequirements$$]__ | Resources defines the compute resource requirements for the pipeline pods. + +| *`extraInitContainers`* __link:https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.28/#container-v1-core[$$Container$$] array__ | ExtraInitContainers are additional init containers run to completion, + +in order, before the pipeline's lint and connect containers start. Use + +them for setup steps the pipeline depends on — fetching certificate + +material, warming a cache, or waiting on an external dependency. They run + +ahead of the operator's built-in `lint` init container, so anything they + +write into a volume from .extraVolumes is visible to lint and to the + +connect runtime (mount it into the pipeline via .extraVolumeMounts). + + + +This is a raw container passthrough with no operator-applied policy; the + +pod's service account and security posture apply. It is distinct from a + +long-lived plugin sidecar (which would be an init container with + +restartPolicy: Always and is not expressed here). + + + +Example (with the backing volume and the mount that exposes the staged + +file to the pipeline containers): + +spec: + +extraVolumes: + +- name: shared + +emptyDir: {} + +extraVolumeMounts: + +- name: shared + +mountPath: /shared + +readOnly: true + +extraInitContainers: + +- name: fetch-certs + +image: curlimages/curl:8.11.0 + +command: ["sh", "-c", "curl -fsSL $CERT_URL -o /shared/ca.pem"] + +volumeMounts: + +- name: shared + +mountPath: /shared + +| *`extraVolumes`* __link:https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.28/#volume-v1-core[$$Volume$$] array__ | ExtraVolumes are additional volumes added to the pipeline pod, typically + +backing an .extraInitContainers staging step or an .extraVolumeMounts + +entry. Raw passthrough: any Kubernetes volume source is accepted. The + +names "config", "cluster-tls-ca", and "cluster-tls-client" are reserved + +by the operator. + +| *`extraVolumeMounts`* __link:https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.28/#volumemount-v1-core[$$VolumeMount$$] array__ | ExtraVolumeMounts are additional volume mounts applied to the built-in + +lint init container and the connect container (both, so the linted view + +of the filesystem matches the runtime's). Mounts for + +.extraInitContainers are declared on those containers directly. + +| *`valueSources`* __xref:{anchor_prefix}-github-com-redpanda-data-redpanda-operator-operator-api-redpanda-v1alpha2-namedvaluesource[$$NamedValueSource$$] array__ | ValueSources is a list of named values the pipeline YAML can reference + +via ${NAME} interpolation. Each value is fetched at render time from + +inline / ConfigMap / Secret / ExternalSecret and projected into the + +pipeline pod as an environment variable. One named pull per entry — + +avoids the bag-of-Secrets env-splat pattern. + + + +Example: + +spec: + +valueSources: + +- name: S3_SECRET_KEY + +source: + +secretKeyRef: + +name: s3-creds + +key: secret_access_key + +configYaml: \| + +output: + +aws_s3: + +bucket: my-bucket + +credentials: + +secret: ${S3_SECRET_KEY} + + + +See: xref:redpanda-connect:configuration/secrets.adoc[] + +| *`annotations`* __object (keys:string, values:string)__ | Annotations specifies additional annotations to apply to the pipeline pod + +template. These are merged with any operator-level commonAnnotations, with + +per-pipeline annotations taking precedence. Useful for integrations like + +Datadog autodiscovery that rely on pod annotations. + +| *`tolerations`* __link:https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.28/#toleration-v1-core[$$Toleration$$] array__ | Tolerations for the pipeline pods, allowing them to be scheduled on tainted nodes. + +| *`nodeSelector`* __object (keys:string, values:string)__ | NodeSelector constrains pipeline pods to nodes with matching labels — the + +simplest way to pin a pipeline to a specific Kubernetes node pool (match + +the node pool's label, e.g. eks.amazonaws.com/nodegroup or a custom + +label). Combine with Tolerations if the node pool is tainted. + +| *`affinity`* __link:https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.28/#affinity-v1-core[$$Affinity$$]__ | Affinity sets pod affinity/anti-affinity for the pipeline pods. Use it for + +node-pool scheduling that NodeSelector can't express — required-OR-of-pools + +(multiple acceptable node pools), preferred (soft) node-pool placement, or + +pod anti-affinity to spread a pipeline across nodes. It is merged with any + +auto-generated zone affinity from Zones: the zone requirement is AND-ed + +into the node affinity so both constraints apply. + +| *`topologySpreadConstraints`* __link:https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.28/#topologyspreadconstraint-v1-core[$$TopologySpreadConstraint$$] array__ | TopologySpreadConstraints controls how pipeline pods are spread across + +topology domains such as availability zones. When Zones is specified, + +a default topology spread constraint is generated automatically. + +Any constraints specified here are used in addition to (or instead of) + +the auto-generated zone constraint. + +| *`zones`* __string array__ | Zones specifies the availability zones across which pipeline pods should + +be spread. When set, the controller configures: + +- A node affinity to schedule pods only on nodes in these zones + +- A topology spread constraint to distribute pods evenly across zones + +The zone label used is "topology.kubernetes.io/zone". + +| *`budget`* __xref:{anchor_prefix}-github-com-redpanda-data-redpanda-operator-operator-api-redpanda-v1alpha2-pipelinebudget[$$PipelineBudget$$]__ | Budget configures a PodDisruptionBudget for the pipeline Deployment, + +protecting pipeline pods from voluntary disruptions such as node drains + +and cluster autoscaler evictions. When not set, no PDB is created. + +| *`cluster`* __xref:{anchor_prefix}-github-com-redpanda-data-redpanda-operator-operator-api-redpanda-v1alpha2-clustersource[$$ClusterSource$$]__ | ClusterSource declaratively binds the pipeline's redpanda input/output + +to a Redpanda cluster. Mirrors the ClusterSource pattern used by the + +User/Topic CRDs: + + + +- clusterRef: point at an existing Redpanda CR by name, in the same + +namespace as the Pipeline (cross-namespace references are rejected — + +the resolved TLS/SASL Secrets must be mountable by the pipeline + +pod). The operator resolves the internal broker addresses + TLS + +material automatically; the SASL identity is taken from .userRef. + +SECURITY: when .userRef is NOT set and the referenced cluster has + +SASL enabled, the pipeline authenticates as the cluster's bootstrap + +superuser — its credentials are projected into the pipeline pod's + +environment. Set .userRef so each pipeline runs with an ACL-scoped + +identity instead. + +- staticConfiguration: hard-code brokers, TLS, and SASL. The password + +is a ValueSource so it can come from inline / Secret / ConfigMap / + +ExternalSecret. + + + +When unset, the pipeline runs against whatever brokers the user wires + +inline in configYaml (e.g. an external Kafka, Confluent Cloud, etc.). + +| *`userRef`* __xref:{anchor_prefix}-github-com-redpanda-data-redpanda-operator-operator-api-redpanda-v1alpha2-pipelineuserref[$$PipelineUserRef$$]__ | UserRef binds the pipeline to a User CR. When set alongside + +.cluster.clusterRef, the operator reads the referenced User's + +password Secret + SASL mechanism and uses the User's metadata.name + +as the SASL username, emitting REDPANDA_SASL_USERNAME / _PASSWORD / + +_MECHANISM env vars in the pipeline pod and a `sasl:` block in the + +auto-generated `redpanda` config. + + + +Set this when the cluster the pipeline talks to has SASL enabled. + +On unauthenticated clusters (and in clusterRef-only modes that + +only need broker discovery), leave it empty. + + + +CEL restrictions: + +- userRef must NOT be set alongside .cluster.staticConfiguration — + +the static path carries its own inline SASL config. + +- userRef must NOT be set without .cluster.clusterRef — there's no + +cluster context to authenticate against otherwise. + + + +The referenced User CR is expected to live in the same namespace as + +the Pipeline and to declare ACLs scoped to the topics, schema + +subjects, and consumer groups this pipeline reads/writes. The + +operator does NOT auto-create or modify the User CR — ACL scoping + +stays an explicit, auditable user-controlled action. + +|=== + + +[id="{anchor_prefix}-github-com-redpanda-data-redpanda-operator-operator-api-redpanda-v1alpha2-pipelinestatus"] +== PipelineStatus + +PipelineStatus defines the observed state of a Pipeline. + + + +.Appears in: +- xref:{anchor_prefix}-github-com-redpanda-data-redpanda-operator-operator-api-redpanda-v1alpha2-pipeline[$$Pipeline$$] + +[cols="25a,75a", options="header"] +|=== +| Field | Description +| *`observedGeneration`* __integer__ | ObservedGeneration is the last observed generation of the Pipeline. + +| *`conditions`* __link:https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.28/#condition-v1-meta[$$Condition$$] array__ | Conditions holds the conditions for the Pipeline. + +| *`phase`* __xref:{anchor_prefix}-github-com-redpanda-data-redpanda-operator-operator-api-redpanda-v1alpha2-pipelinephase[$$PipelinePhase$$]__ | Phase describes the current phase of the pipeline lifecycle. + +| *`replicas`* __integer__ | Replicas is the number of pipeline pods observed on the underlying + +Deployment. The scale subresource reports it as status.replicas, which + +is how HPA and KEDA observe the pipeline's current scale. + +| *`readyReplicas`* __integer__ | ReadyReplicas is the number of ready pipeline pods. + +| *`selector`* __string__ | Selector is the label selector for this pipeline's pods, in string + +form. The scale subresource reports it as status.selector, which is + +how HPA and KEDA discover the pods backing this Pipeline when + +computing per-pod (cpu/memory/custom) metrics. + +|=== + + +[id="{anchor_prefix}-github-com-redpanda-data-redpanda-operator-operator-api-redpanda-v1alpha2-pipelineuserref"] +== PipelineUserRef + +PipelineUserRef points at a User CR whose password Secret + SCRAM +mechanism the pipeline will use to authenticate to Redpanda. + + + +.Appears in: +- xref:{anchor_prefix}-github-com-redpanda-data-redpanda-operator-operator-api-redpanda-v1alpha2-pipelinespec[$$PipelineSpec$$] + +[cols="25a,75a", options="header"] +|=== +| Field | Description +| *`name`* __string__ | Name of the User CR (in the same namespace as the Pipeline). + +|=== + + [id="{anchor_prefix}-github-com-redpanda-data-redpanda-operator-operator-api-redpanda-v1alpha2-podantiaffinity"] == PodAntiAffinity @@ -2357,7 +2746,7 @@ DEPRECATED: Use sideCars.securityContext + [id="{anchor_prefix}-github-com-redpanda-data-redpanda-operator-operator-api-redpanda-v1alpha2-rackawareness"] == RackAwareness -RackAwareness configures rack awareness in the Helm values. See xref:manage:kubernetes/kubernetes-rack-awareness.adoc[] +RackAwareness configures rack awareness in the Helm values. See xref:manage:kubernetes/kubernetes-rack-awareness.adoc[]. @@ -2476,7 +2865,7 @@ More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api- [id="{anchor_prefix}-github-com-redpanda-data-redpanda-operator-operator-api-redpanda-v1alpha2-redpandaclusterspec"] == RedpandaClusterSpec -RedpandaClusterSpec defines the desired state of a Redpanda cluster. These settings are the same as those defined in the Redpanda Helm chart. The values in these settings are passed to the Redpanda Helm chart through Flux. For all default values and links to more documentation, see xref:reference:redpanda-helm-spec.adoc[] +RedpandaClusterSpec defines the desired state of a Redpanda cluster. These settings are the same as those defined in the Redpanda Helm chart. The values in these settings are passed to the Redpanda Helm chart through Flux. For all default values and links to more documentation, see xref:reference:redpanda-helm-spec.adoc[]. For descriptions and default values, see xref:k-redpanda-helm-spec.adoc[]. @@ -2533,7 +2922,7 @@ documentation](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-po [id="{anchor_prefix}-github-com-redpanda-data-redpanda-operator-operator-api-redpanda-v1alpha2-redpandaconnectors"] == RedpandaConnectors -RedpandaConnectors configures Redpanda Connectors. Redpanda Connectors is a package that includes Kafka Connect and built-in connectors, sometimes known as plugins. See xref:deploy:deployment-option/self-hosted/kubernetes/k-deploy-connectors.adoc[] +RedpandaConnectors configures Redpanda Connectors. Redpanda Connectors is a package that includes Kafka Connect and built-in connectors, sometimes known as plugins. See xref:deploy:deployment-option/self-hosted/kubernetes/k-deploy-connectors.adoc[]. @@ -4113,7 +4502,7 @@ API of a Redpanda cluster where the object should be created. + [id="{anchor_prefix}-github-com-redpanda-data-redpanda-operator-operator-api-redpanda-v1alpha2-storage"] == Storage -Storage configures storage-related settings in the Helm values. See xref:manage:kubernetes/storage.adoc[] +Storage configures storage-related settings in the Helm values. See xref:manage:kubernetes/storage.adoc[]. @@ -4615,7 +5004,7 @@ Forked from UsageStats: removed deprecated Organization field and unused Cluster [id="{anchor_prefix}-github-com-redpanda-data-redpanda-operator-operator-api-redpanda-v1alpha2-tls"] == TLS -TLS configures TLS in the Helm values. See xref:manage:kubernetes/security/tls.adoc[] +TLS configures TLS in the Helm values. See xref:manage:kubernetes/security/tls.adoc[]. @@ -4646,7 +5035,7 @@ Task states [id="{anchor_prefix}-github-com-redpanda-data-redpanda-operator-operator-api-redpanda-v1alpha2-tiered"] == Tiered -Tiered configures storage for the Tiered Storage cache. See xref:manage:kubernetes/tiered-storage-kubernetes.adoc[] +Tiered configures storage for the Tiered Storage cache. See xref:manage:kubernetes/tiered-storage-kubernetes.adoc[]. @@ -4683,44 +5072,44 @@ TieredConfig configures Tiered Storage, which requires an Enterprise license con [cols="25a,75a", options="header"] |=== | Field | Description -| *`cloud_storage_enabled`* __xref:{anchor_prefix}-github-com-redpanda-data-redpanda-operator-operator-api-apiutil-jsonboolean[$$JSONBoolean$$]__ | Enables Tiered Storage, if a license key is provided. See xref:reference:cluster-properties.adoc[] + -| *`cloud_storage_api_endpoint`* __string__ | See xref:reference:cluster-properties.adoc[] + -| *`cloud_storage_api_endpoint_port`* __integer__ | See xref:reference:cluster-properties.adoc[] + -| *`cloud_storage_bucket`* __string__ | See xref:reference:cluster-properties.adoc[] + -| *`cloud_storage_azure_container`* __string__ | See xref:reference:cluster-properties.adoc[] + -| *`cloud_storage_azure_managed_identity_id`* __string__ | See xref:reference:cluster-properties.adoc[] + -| *`cloud_storage_azure_storage_account`* __string__ | See xref:reference:cluster-properties.adoc[] + -| *`cloud_storage_azure_shared_key`* __string__ | See xref:reference:cluster-properties.adoc[] + -| *`cloud_storage_azure_adls_endpoint`* __string__ | See xref:reference:cluster-properties.adoc[] + -| *`cloud_storage_azure_adls_port`* __integer__ | See xref:reference:cluster-properties.adoc[] + -| *`cloud_storage_cache_check_interval`* __integer__ | See xref:reference:tunable-properties.adoc[] + -| *`cloud_storage_cache_directory`* __string__ | See xref:reference:node-properties.adoc[] + -| *`cloud_storage_cache_size`* __string__ | See xref:reference:cluster-properties.adoc[] + -| *`cloud_storage_credentials_source`* __string__ | See xref:reference:cluster-properties.adoc[] + -| *`cloud_storage_disable_tls`* __boolean__ | See xref:reference:cluster-properties.adoc[] + -| *`cloud_storage_enable_remote_read`* __boolean__ | See xref:reference:tunable-properties.adoc[] + -| *`cloud_storage_enable_remote_write`* __boolean__ | See xref:reference:tunable-properties.adoc[] + -| *`cloud_storage_initial_backoff_ms`* __integer__ | See xref:reference:tunable-properties.adoc[] + -| *`cloud_storage_manifest_upload_timeout_ms`* __integer__ | See xref:reference:tunable-properties.adoc[] + -| *`cloud_storage_max_connection_idle_time_ms`* __integer__ | See xref:reference:tunable-properties.adoc[] + -| *`cloud_storage_max_connections`* __integer__ | See xref:reference:cluster-properties.adoc[] + -| *`cloud_storage_reconciliation_interval_ms`* __integer__ | Deprecated: See xref:reference:tunable-properties.adoc[] + -| *`cloud_storage_region`* __string__ | See xref:reference:cluster-properties.adoc[] + -| *`cloud_storage_segment_max_upload_interval_sec`* __integer__ | See xref:reference:tunable-properties.adoc[] + -| *`cloud_storage_segment_upload_timeout_ms`* __integer__ | See xref:reference:tunable-properties.adoc[] + -| *`cloud_storage_trust_file`* __string__ | See xref:reference:cluster-properties.adoc[] + -| *`cloud_storage_upload_ctrl_d_coeff`* __integer__ | See xref:reference:tunable-properties.adoc[] + -| *`cloud_storage_upload_ctrl_max_shares`* __integer__ | See xref:reference:tunable-properties.adoc[] + -| *`cloud_storage_upload_ctrl_min_shares`* __integer__ | See xref:reference:tunable-properties.adoc[] + -| *`cloud_storage_upload_ctrl_p_coeff`* __integer__ | See xref:reference:tunable-properties.adoc[] + -| *`cloud_storage_upload_ctrl_update_interval_ms`* __integer__ | See xref:reference:tunable-properties.adoc[] + +| *`cloud_storage_enabled`* __xref:{anchor_prefix}-github-com-redpanda-data-redpanda-operator-operator-api-apiutil-jsonboolean[$$JSONBoolean$$]__ | Enables Tiered Storage, if a license key is provided. See xref:reference:cluster-properties.adoc[]. + +| *`cloud_storage_api_endpoint`* __string__ | See xref:reference:cluster-properties.adoc[]. + +| *`cloud_storage_api_endpoint_port`* __integer__ | See xref:reference:cluster-properties.adoc[]. + +| *`cloud_storage_bucket`* __string__ | See xref:reference:cluster-properties.adoc[]. + +| *`cloud_storage_azure_container`* __string__ | See xref:reference:cluster-properties.adoc[]. + +| *`cloud_storage_azure_managed_identity_id`* __string__ | See xref:reference:cluster-properties.adoc[]. + +| *`cloud_storage_azure_storage_account`* __string__ | See xref:reference:cluster-properties.adoc[]. + +| *`cloud_storage_azure_shared_key`* __string__ | See xref:reference:cluster-properties.adoc[]. + +| *`cloud_storage_azure_adls_endpoint`* __string__ | See xref:reference:cluster-properties.adoc[]. + +| *`cloud_storage_azure_adls_port`* __integer__ | See xref:reference:cluster-properties.adoc[]. + +| *`cloud_storage_cache_check_interval`* __integer__ | See xref:reference:tunable-properties.adoc[]. + +| *`cloud_storage_cache_directory`* __string__ | See xref:reference:node-properties.adoc[]. + +| *`cloud_storage_cache_size`* __string__ | See xref:reference:cluster-properties.adoc[]. + +| *`cloud_storage_credentials_source`* __string__ | See xref:reference:cluster-properties.adoc[]. + +| *`cloud_storage_disable_tls`* __boolean__ | See xref:reference:cluster-properties.adoc[]. + +| *`cloud_storage_enable_remote_read`* __boolean__ | See xref:reference:tunable-properties.adoc[]. + +| *`cloud_storage_enable_remote_write`* __boolean__ | See xref:reference:tunable-properties.adoc[]. + +| *`cloud_storage_initial_backoff_ms`* __integer__ | See xref:reference:tunable-properties.adoc[]. + +| *`cloud_storage_manifest_upload_timeout_ms`* __integer__ | See xref:reference:tunable-properties.adoc[]. + +| *`cloud_storage_max_connection_idle_time_ms`* __integer__ | See xref:reference:tunable-properties.adoc[]. + +| *`cloud_storage_max_connections`* __integer__ | See xref:reference:cluster-properties.adoc[]. + +| *`cloud_storage_reconciliation_interval_ms`* __integer__ | Deprecated: See xref:reference:tunable-properties.adoc[]. + +| *`cloud_storage_region`* __string__ | See xref:reference:cluster-properties.adoc[]. + +| *`cloud_storage_segment_max_upload_interval_sec`* __integer__ | See xref:reference:tunable-properties.adoc[]. + +| *`cloud_storage_segment_upload_timeout_ms`* __integer__ | See xref:reference:tunable-properties.adoc[]. + +| *`cloud_storage_trust_file`* __string__ | See xref:reference:cluster-properties.adoc[]. + +| *`cloud_storage_upload_ctrl_d_coeff`* __integer__ | See xref:reference:tunable-properties.adoc[]. + +| *`cloud_storage_upload_ctrl_max_shares`* __integer__ | See xref:reference:tunable-properties.adoc[]. + +| *`cloud_storage_upload_ctrl_min_shares`* __integer__ | See xref:reference:tunable-properties.adoc[]. + +| *`cloud_storage_upload_ctrl_p_coeff`* __integer__ | See xref:reference:tunable-properties.adoc[]. + +| *`cloud_storage_upload_ctrl_update_interval_ms`* __integer__ | See xref:reference:tunable-properties.adoc[]. + |=== [id="{anchor_prefix}-github-com-redpanda-data-redpanda-operator-operator-api-redpanda-v1alpha2-topic"] == Topic -Topic defines the CRD for Topic resources. See xref:manage:kubernetes/manage-topics.adoc[] +Topic defines the CRD for Topic resources. See xref:manage:kubernetes/manage-topics.adoc[]. @@ -4762,7 +5151,7 @@ More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api- [id="{anchor_prefix}-github-com-redpanda-data-redpanda-operator-operator-api-redpanda-v1alpha2-topicspec"] == TopicSpec -TopicSpec defines the desired state of the topic. See xref:manage:kubernetes/manage-topics.adoc[] +TopicSpec defines the desired state of the topic. See xref:manage:kubernetes/manage-topics.adoc[]. @@ -4782,7 +5171,7 @@ See xref:reference:cluster-properties.adoc[] and + xref:get-started:architecture.adoc[] + | *`replicationFactor`* __integer__ | Specifies the number of replicas the topic should have. Must be odd value. + When absent this will default to the Redpanda cluster configuration `default_topic_replications`. + -See xref:reference:cluster-properties.adoc[] + +See xref:reference:cluster-properties.adoc[]. + | *`overwriteTopicName`* __string__ | Changes the topic name from the value of `metadata.name`. + | *`additionalConfig`* __object (keys:string, values:string)__ | Adds extra topic configurations. This is a free-form map of any configuration options that topics can have. + Examples: + @@ -5146,6 +5535,7 @@ ValueSource represents where a value can be pulled from - xref:{anchor_prefix}-github-com-redpanda-data-redpanda-operator-operator-api-redpanda-v1alpha2-kafkasaslawsmskiam[$$KafkaSASLAWSMskIam$$] - xref:{anchor_prefix}-github-com-redpanda-data-redpanda-operator-operator-api-redpanda-v1alpha2-kafkasaslgssapi[$$KafkaSASLGSSAPI$$] - xref:{anchor_prefix}-github-com-redpanda-data-redpanda-operator-operator-api-redpanda-v1alpha2-kafkasasloauthbearer[$$KafkaSASLOAuthBearer$$] +- xref:{anchor_prefix}-github-com-redpanda-data-redpanda-operator-operator-api-redpanda-v1alpha2-namedvaluesource[$$NamedValueSource$$] - xref:{anchor_prefix}-github-com-redpanda-data-redpanda-operator-operator-api-redpanda-v1alpha2-schemaregistrysasl[$$SchemaRegistrySASL$$] - xref:{anchor_prefix}-github-com-redpanda-data-redpanda-operator-operator-api-redpanda-v1alpha2-shadowlinkschemaregistrybasicauthentication[$$ShadowLinkSchemaRegistryBasicAuthentication$$] From 5bc9019c32ba5d30417f6bde714fe35bfc5a92bf Mon Sep 17 00:00:00 2001 From: JakeSCahill Date: Tue, 28 Jul 2026 09:49:21 +0100 Subject: [PATCH 2/2] fix: correct cross-component xref to Redpanda Connect secrets page urlToXref converted the godoc URL https://docs.redpanda.com/redpanda-connect/configuration/secrets/ into xref:redpanda-connect:configuration/secrets.adoc, treating the site URL slug as a module in the streaming component. The Connect component is named 'connect', so the correct resource ID is xref:connect:configuration:secrets.adoc. --- modules/reference/pages/k-crd.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/reference/pages/k-crd.adoc b/modules/reference/pages/k-crd.adoc index cbea5e6444..6a4849a8ce 100644 --- a/modules/reference/pages/k-crd.adoc +++ b/modules/reference/pages/k-crd.adoc @@ -2372,7 +2372,7 @@ credentials: + secret: ${S3_SECRET_KEY} + -See: xref:redpanda-connect:configuration/secrets.adoc[] + +See: xref:connect:configuration:secrets.adoc[] + | *`annotations`* __object (keys:string, values:string)__ | Annotations specifies additional annotations to apply to the pipeline pod + template. These are merged with any operator-level commonAnnotations, with + per-pipeline annotations taking precedence. Useful for integrations like +