From ffd23639455205444dcbac910fb883345828f970 Mon Sep 17 00:00:00 2001 From: JakeSCahill Date: Fri, 17 Jul 2026 11:22:12 +0100 Subject: [PATCH 01/14] docs: surface Redpanda Connect Pipeline resource for operator users Three discovery surfaces for the 26.2 Pipeline CRD (DOC-2275): - new Manage > Kubernetes page routing to the full guide in the Connect docs, positioned next to Manage Kafka Connect - Pipeline row in the CRD reference index - TIP on Manage Kafka Connect pointing at the first-class alternative --- modules/ROOT/nav.adoc | 1 + .../pages/kubernetes/k-connect-pipelines.adoc | 30 +++++++++++++++++++ .../pages/kubernetes/k-manage-connectors.adoc | 2 ++ modules/reference/pages/k-crd-index.adoc | 4 +++ 4 files changed, 37 insertions(+) create mode 100644 modules/manage/pages/kubernetes/k-connect-pipelines.adoc diff --git a/modules/ROOT/nav.adoc b/modules/ROOT/nav.adoc index 97e56aa5fd..30dc0e4f05 100644 --- a/modules/ROOT/nav.adoc +++ b/modules/ROOT/nav.adoc @@ -110,6 +110,7 @@ *** xref:manage:kubernetes/k-cluster-property-configuration.adoc[Cluster Properties] *** xref:manage:kubernetes/k-manage-topics.adoc[Manage Topics] *** xref:manage:kubernetes/k-manage-connectors.adoc[Manage Kafka Connect] +*** xref:manage:kubernetes/k-connect-pipelines.adoc[Redpanda Connect Pipelines] *** xref:manage:kubernetes/storage/index.adoc[Storage] **** xref:manage:kubernetes/storage/k-volume-types.adoc[Volume Types] **** xref:manage:kubernetes/storage/k-configure-storage.adoc[Configure Storage] diff --git a/modules/manage/pages/kubernetes/k-connect-pipelines.adoc b/modules/manage/pages/kubernetes/k-connect-pipelines.adoc new file mode 100644 index 0000000000..68f6d2758d --- /dev/null +++ b/modules/manage/pages/kubernetes/k-connect-pipelines.adoc @@ -0,0 +1,30 @@ += Redpanda Connect Pipelines in Kubernetes +:description: Run Redpanda Connect pipelines declaratively with the Redpanda Operator using the Pipeline resource. +:env-kubernetes: true +:page-categories: Management, Integration +:page-topic-type: concept + +// DRAFT (DOC-2275): Hold until the Pipeline resource ships in a released operator +// and rp-connect-docs#457 is published. See that PR for the full guide this page routes to. + +Run xref:connect:home:index.adoc[Redpanda Connect] pipelines as Kubernetes resources managed by the Redpanda Operator. You define each pipeline in a `Pipeline` custom resource, and the operator deploys it, validates its configuration before startup, rolls it on every change, and reports its health in the resource status, using the same declarative workflow as your Topic and User resources. + +This feature requires an xref:get-started:licensing/overview.adoc[Enterprise Edition license] and Redpanda Operator or later. +// TODO(DOC-2275): Set the real minimum operator version when the feature ships. + +== When to use the Pipeline resource + +Use the Pipeline resource when you want: + +* *Declarative, GitOps-friendly management*: Pipelines are plain Kubernetes manifests that you can version, review, and deploy with tools such as Argo CD or Flux. +* *Validated rollouts*: The operator lints every configuration before the pipeline starts, and rolls the workload automatically when the configuration changes. +* *Cluster integration*: A pipeline can reference a Redpanda resource in the same namespace, and the operator injects the connection details, TLS material, and credentials for you. +* *Built-in connectors*: Redpanda Connect ships hundreds of connectors and processors, including AI integrations, without the JVM or plugin management that Kafka Connect requires. + +For Kafka Connect, which is community-supported with Redpanda, see xref:manage:kubernetes/k-manage-connectors.adoc[]. To run Redpanda Connect on Kubernetes without the operator, see xref:connect:install:helm-chart.adoc[]. + +== Get started + +For installation steps, pipeline examples, secrets handling, and lifecycle management, see xref:connect:install:kubernetes-operator.adoc[Run Pipelines with the Redpanda Operator] in the Redpanda Connect documentation. + +For the Pipeline CRD source file, see the xref:reference:k-crd-index.adoc[CRD reference]. diff --git a/modules/manage/pages/kubernetes/k-manage-connectors.adoc b/modules/manage/pages/kubernetes/k-manage-connectors.adoc index 7937fe1d48..dc7fc59bf0 100644 --- a/modules/manage/pages/kubernetes/k-manage-connectors.adoc +++ b/modules/manage/pages/kubernetes/k-manage-connectors.adoc @@ -8,6 +8,8 @@ When you have Kafka Connect deployed, you can create and manage connectors using include::shared:partial$community-supported-connectors.adoc[] +TIP: To build streaming data pipelines with enterprise support and no Kafka Connect infrastructure, run Redpanda Connect pipelines declaratively with the Redpanda Operator. See xref:manage:kubernetes/k-connect-pipelines.adoc[]. + == Prerequisites - xref:deploy:redpanda/kubernetes/k-production-deployment.adoc[Deploy a Redpanda cluster with Redpanda Console]. diff --git a/modules/reference/pages/k-crd-index.adoc b/modules/reference/pages/k-crd-index.adoc index cc3f6044cc..9cf9373571 100644 --- a/modules/reference/pages/k-crd-index.adoc +++ b/modules/reference/pages/k-crd-index.adoc @@ -27,6 +27,10 @@ The CRD source definitions are maintained in the https://github.com/redpanda-dat | https://github.com/redpanda-data/redpanda-operator/blob/main/operator/config/crd/bases/cluster.redpanda.com_consoles.yaml[`cluster.redpanda.com_consoles.yaml`^] | Defines Redpanda Console resources. +| Pipeline +| https://github.com/redpanda-data/redpanda-operator/blob/main/operator/config/crd/bases/cluster.redpanda.com_pipelines.yaml[`cluster.redpanda.com_pipelines.yaml`^] +| Defines Redpanda Connect pipeline resources. See xref:connect:install:kubernetes-operator.adoc[Run Pipelines with the Redpanda Operator]. + | Schema | https://github.com/redpanda-data/redpanda-operator/blob/main/operator/config/crd/bases/cluster.redpanda.com_schemas.yaml[`cluster.redpanda.com_schemas.yaml`^] | Defines Schema Registry resources. From 897b6ca64f876d09dc81d110779db170db49bd96 Mon Sep 17 00:00:00 2001 From: JakeSCahill Date: Fri, 17 Jul 2026 11:37:06 +0100 Subject: [PATCH 02/14] docs: fix stacked admonitions and concept intro pattern per style review --- modules/manage/pages/kubernetes/k-connect-pipelines.adoc | 2 +- modules/manage/pages/kubernetes/k-manage-connectors.adoc | 4 +--- 2 files changed, 2 insertions(+), 4 deletions(-) diff --git a/modules/manage/pages/kubernetes/k-connect-pipelines.adoc b/modules/manage/pages/kubernetes/k-connect-pipelines.adoc index 68f6d2758d..94d5acd2d1 100644 --- a/modules/manage/pages/kubernetes/k-connect-pipelines.adoc +++ b/modules/manage/pages/kubernetes/k-connect-pipelines.adoc @@ -7,7 +7,7 @@ // DRAFT (DOC-2275): Hold until the Pipeline resource ships in a released operator // and rp-connect-docs#457 is published. See that PR for the full guide this page routes to. -Run xref:connect:home:index.adoc[Redpanda Connect] pipelines as Kubernetes resources managed by the Redpanda Operator. You define each pipeline in a `Pipeline` custom resource, and the operator deploys it, validates its configuration before startup, rolls it on every change, and reports its health in the resource status, using the same declarative workflow as your Topic and User resources. +Run xref:connect:home:index.adoc[Redpanda Connect] pipelines as Kubernetes resources managed by the Redpanda Operator. You define each pipeline in a `Pipeline` custom resource, and the operator deploys it, validates its configuration before startup, rolls it on every change, and reports its health in the resource status. Pipelines follow the same declarative workflow as your Topic and User resources, so you can build and operate streaming data pipelines with the same GitOps tooling you already use for cluster resources. This page explains when to choose the Pipeline resource over other options and where to find the deployment guide. This feature requires an xref:get-started:licensing/overview.adoc[Enterprise Edition license] and Redpanda Operator or later. // TODO(DOC-2275): Set the real minimum operator version when the feature ships. diff --git a/modules/manage/pages/kubernetes/k-manage-connectors.adoc b/modules/manage/pages/kubernetes/k-manage-connectors.adoc index dc7fc59bf0..2424cd24fc 100644 --- a/modules/manage/pages/kubernetes/k-manage-connectors.adoc +++ b/modules/manage/pages/kubernetes/k-manage-connectors.adoc @@ -4,12 +4,10 @@ :page-categories: Management, Integration :env-kubernetes: true -When you have Kafka Connect deployed, you can create and manage connectors using Redpanda Console or the Kafka Connect REST API. +When you have Kafka Connect deployed, you can create and manage connectors using Redpanda Console or the Kafka Connect REST API. To build streaming data pipelines with enterprise support and without Kafka Connect infrastructure, consider running xref:manage:kubernetes/k-connect-pipelines.adoc[Redpanda Connect pipelines] with the Redpanda Operator instead. include::shared:partial$community-supported-connectors.adoc[] -TIP: To build streaming data pipelines with enterprise support and no Kafka Connect infrastructure, run Redpanda Connect pipelines declaratively with the Redpanda Operator. See xref:manage:kubernetes/k-connect-pipelines.adoc[]. - == Prerequisites - xref:deploy:redpanda/kubernetes/k-production-deployment.adoc[Deploy a Redpanda cluster with Redpanda Console]. From a2b9e52f88fc7827815615cc700543f1928a8a9a Mon Sep 17 00:00:00 2001 From: David Yu Date: Fri, 24 Jul 2026 11:00:30 -0700 Subject: [PATCH 03/14] docs: expand Connect Pipeline docs with worked flows, autoscaling, licensing, and what's new (DOC-2275) Turn the discovery page into a full operator-side guide covering every flow from redpanda-operator#1677: enable the Connect controller, first pipeline, clusterRef+userRef cluster binding (with the rendered-config injection contract), staticConfiguration for external clusters, per-pipeline cloud IAM via serviceAccountName (IRSA worked example), and valueSources with all four source types. Adds status/condition reference and troubleshooting. Add a dedicated autoscaling page with step-by-step HPA (cpu/memory) and KEDA (consumer-group lag, Connect metrics via Prometheus, scale-to-zero) examples, all grounded in manifests validated end-to-end on EKS. Document the no-license/expired-license behavior for pipelines in the licensing overview, and add the operator what's-new entry (version section marked TODO until the ship vehicle is confirmed). Route the CRD index row to the local guide instead of the unpublished connect-docs page, which removes the two expected broken-xref warnings. Co-Authored-By: Claude Fable 5 --- modules/ROOT/nav.adoc | 1 + .../get-started/pages/licensing/overview.adoc | 4 + .../pages/release-notes/operator.adoc | 18 + .../k-autoscale-connect-pipelines.adoc | 314 +++++++++ .../pages/kubernetes/k-connect-pipelines.adoc | 634 +++++++++++++++++- modules/reference/pages/k-crd-index.adoc | 2 +- 6 files changed, 964 insertions(+), 9 deletions(-) create mode 100644 modules/manage/pages/kubernetes/k-autoscale-connect-pipelines.adoc diff --git a/modules/ROOT/nav.adoc b/modules/ROOT/nav.adoc index 30dc0e4f05..864b10e471 100644 --- a/modules/ROOT/nav.adoc +++ b/modules/ROOT/nav.adoc @@ -111,6 +111,7 @@ *** xref:manage:kubernetes/k-manage-topics.adoc[Manage Topics] *** xref:manage:kubernetes/k-manage-connectors.adoc[Manage Kafka Connect] *** xref:manage:kubernetes/k-connect-pipelines.adoc[Redpanda Connect Pipelines] +**** xref:manage:kubernetes/k-autoscale-connect-pipelines.adoc[Autoscaling] *** xref:manage:kubernetes/storage/index.adoc[Storage] **** xref:manage:kubernetes/storage/k-volume-types.adoc[Volume Types] **** xref:manage:kubernetes/storage/k-configure-storage.adoc[Configure Storage] diff --git a/modules/get-started/pages/licensing/overview.adoc b/modules/get-started/pages/licensing/overview.adoc index 652500993f..b5c82af8b9 100644 --- a/modules/get-started/pages/licensing/overview.adoc +++ b/modules/get-started/pages/licensing/overview.adoc @@ -243,6 +243,10 @@ The following enterprise features are available with a valid Enterprise Edition | Run Redpanda Connect using a FIPS-compliant version of `rpk`, the Redpanda command-line interface (CLI). | No change. +| xref:manage:kubernetes/k-connect-pipelines.adoc[Pipelines in Kubernetes] (Pipeline resource) +| Run Redpanda Connect pipelines declaratively with the Redpanda Operator. The operator's Connect controller requires a valid license, configured through the operator's `enterprise.licenseSecretRef` Helm chart values. +| When the license is missing, expired, or invalid, the operator stops creating and updating pipeline workloads: new Pipeline resources report `License: False` in their status and are never deployed, and existing pipelines keep running with their last-applied configuration but no longer receive updates. Applying a valid license resumes reconciliation automatically, without recreating any Pipeline resources. + | xref:connect:components:redpanda/about.adoc[Redpanda Connect configuration service] | A configuration block that you can use to send logs and status events to a topic on a Redpanda cluster. | No change. diff --git a/modules/get-started/pages/release-notes/operator.adoc b/modules/get-started/pages/release-notes/operator.adoc index 03e72a7cf7..b326189a38 100644 --- a/modules/get-started/pages/release-notes/operator.adoc +++ b/modules/get-started/pages/release-notes/operator.adoc @@ -10,6 +10,24 @@ See also: * xref:upgrade:k-rolling-upgrade.adoc[] +// TODO(DOC-2275): Confirm the operator version that ships the Pipeline resource +// before un-drafting. The feature is currently expected in v26.3.x; if it lands +// in a v26.2.x release instead, fold this section into v26.2.x below. +== Redpanda Operator v26.3.x + +link:https://github.com/redpanda-data/redpanda-operator/blob/release/v26.3.x/operator/CHANGELOG.md[Changelog^] + +=== Redpanda Connect pipelines (beta) + +The Redpanda Operator introduces the Pipeline custom resource, which runs xref:connect:home:index.adoc[Redpanda Connect] pipelines as declaratively managed Kubernetes workloads. You define the pipeline configuration in the resource, and the operator deploys it, lints it before startup, rolls it automatically on configuration and credential changes, and reports health through status conditions. A pipeline can reference a Redpanda resource and a User resource, and the operator injects broker addresses, TLS material, and SASL credentials into the rendered configuration for you. + +Pipelines also implement the Kubernetes scale subresource, so you can scale them with `kubectl scale`, a HorizontalPodAutoscaler, or KEDA, including on consumer group lag and on the metrics Redpanda Connect emits. + +The Pipeline resource is in beta and requires an Enterprise Edition license. Enable it with the operator chart's `connectController.enabled=true` setting. + +* xref:manage:kubernetes/k-connect-pipelines.adoc[Redpanda Connect Pipelines in Kubernetes] -- Deploy pipelines, connect them to clusters, and inject secrets and cloud IAM identities. +* xref:manage:kubernetes/k-autoscale-connect-pipelines.adoc[Autoscale Redpanda Connect Pipelines] -- Step-by-step HPA and KEDA autoscaling. + == Redpanda Operator v26.2.x link:https://github.com/redpanda-data/redpanda-operator/blob/release/v26.2.x/operator/CHANGELOG.md[Changelog^] diff --git a/modules/manage/pages/kubernetes/k-autoscale-connect-pipelines.adoc b/modules/manage/pages/kubernetes/k-autoscale-connect-pipelines.adoc new file mode 100644 index 0000000000..d146788104 --- /dev/null +++ b/modules/manage/pages/kubernetes/k-autoscale-connect-pipelines.adoc @@ -0,0 +1,314 @@ += Autoscale Redpanda Connect Pipelines +:description: Scale Redpanda Connect pipelines manually or automatically with kubectl scale, a HorizontalPodAutoscaler, or KEDA. +:page-topic-type: how-to +:page-beta: true +:env-kubernetes: true +:page-categories: Management, Integration +:personas: platform_operator, platform_engineer +:learning-objective-1: Scale a pipeline manually through the scale subresource +:learning-objective-2: Autoscale a pipeline on CPU and memory with a HorizontalPodAutoscaler +:learning-objective-3: Autoscale a pipeline on consumer group lag or Connect metrics with KEDA + +// DRAFT (DOC-2275): Hold until the Pipeline resource ships in a released operator. + +Match pipeline capacity to your workload by scaling the number of pipeline replicas, either manually or automatically based on resource usage, consumer group lag, or the metrics that Redpanda Connect emits. + +After reading this page, you will be able to: + +* [ ] {learning-objective-1} +* [ ] {learning-objective-2} +* [ ] {learning-objective-3} + +The xref:manage:kubernetes/k-connect-pipelines.adoc[Pipeline resource] implements the Kubernetes https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/[scale subresource^], so anything that speaks the `/scale` API can drive `spec.replicas`: `kubectl scale`, a HorizontalPodAutoscaler (HPA), or a https://keda.sh/[KEDA^] ScaledObject. + +IMPORTANT: Always target the Pipeline resource, never its Deployment. The operator resets the Deployment's replica count to the Pipeline's value on every sync, so an autoscaler pointed at the Deployment is continuously fought and undone. + +== Prerequisites + +You must have the following: + +* A running pipeline managed by the Redpanda Operator with the Connect controller enabled. See xref:manage:kubernetes/k-connect-pipelines.adoc[]. +* For HPA on CPU or memory: https://github.com/kubernetes-sigs/metrics-server[metrics-server^] installed in the cluster. +* For KEDA: https://keda.sh/docs/latest/deploy/[KEDA^] installed in the cluster. +* For scaling on Redpanda Connect metrics: a Prometheus installation that scrapes pipeline Pods, for example https://github.com/prometheus-operator/kube-prometheus[kube-prometheus-stack^] together with the operator chart's `connectController.monitoring.enabled=true` setting. + +Configure only one autoscaler per pipeline. An HPA and a KEDA ScaledObject targeting the same Pipeline fight each other. + +== Scale a pipeline manually + +Scale the Pipeline directly with `kubectl scale`: + +[,bash] +---- +kubectl scale pipeline/ --replicas=3 --namespace +---- + +The operator updates the pipeline's Deployment to match, and the pipeline's consumer group rebalances its partitions across the new replicas. + +To confirm that the scale subresource is wired up, read it directly: + +[,bash] +---- +kubectl get --raw "/apis/cluster.redpanda.com/v1alpha2/namespaces//pipelines//scale" +---- + +The response reports `spec.replicas`, `status.replicas`, and `status.selector`, the label selector that autoscalers use to find the pipeline's Pods. To list a pipeline's Pods yourself, the `app.kubernetes.io/instance=` label is the simplest filter. + +[NOTE] +==== +When you scale a pipeline that consumes from Redpanda, replicas beyond the topic's partition count sit idle, because a consumer group assigns each partition to at most one member. Size `--replicas`, and any autoscaler's maximum, to the partition count of the input topic. +==== + +== Autoscale on CPU and memory with HPA + +CPU-based and memory-based HPAs work without any extra configuration, because pipeline Pods always carry resource requests: if you don't set `spec.resources`, the operator applies default requests of `100m` CPU and `256Mi` memory. + +. Ensure metrics-server is running. If `kubectl top pods` returns metrics, you're ready. Otherwise, install it: ++ +[,bash] +---- +kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml +---- + +. Set explicit resource requests on the pipeline, so utilization percentages are meaningful for your workload: ++ +[,yaml] +---- +spec: + resources: + requests: + cpu: 100m + memory: 256Mi + limits: + cpu: 300m + memory: 512Mi +---- + +. Create an HPA whose `scaleTargetRef` points at the Pipeline: ++ +.`pipeline-hpa.yaml` +[,yaml] +---- +apiVersion: autoscaling/v2 +kind: HorizontalPodAutoscaler +metadata: + name: orders-to-warehouse + namespace: +spec: + scaleTargetRef: + apiVersion: cluster.redpanda.com/v1alpha2 + kind: Pipeline + name: orders-to-warehouse + minReplicas: 1 + maxReplicas: 4 + behavior: + scaleUp: + stabilizationWindowSeconds: 0 + scaleDown: + stabilizationWindowSeconds: 60 + metrics: + - type: Resource + resource: + name: cpu + target: + type: Utilization + averageUtilization: 50 + - type: Resource + resource: + name: memory + target: + type: Utilization + averageUtilization: 80 +---- + +. Apply the manifest and watch the HPA react. Resource metrics lag by about a minute: ++ +[,bash] +---- +kubectl apply -f pipeline-hpa.yaml +kubectl get hpa orders-to-warehouse --namespace --watch +---- ++ +Example output while the pipeline is busy. CPU is at 300% of its request, above the 50% target, so the HPA scales out toward `maxReplicas`: ++ +[.no-copy] +---- +NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE +orders-to-warehouse Pipeline/orders-to-warehouse cpu: 300%/50%, memory: 30%/80% 1 4 4 2m +---- + +. Verify that the Pipeline followed: ++ +[,bash] +---- +kubectl get pipeline orders-to-warehouse --namespace +kubectl get pods -l app.kubernetes.io/instance=orders-to-warehouse --namespace +---- + +To scale an HPA on the metrics Redpanda Connect emits instead of CPU or memory, either expose them to the HPA as custom metrics through https://github.com/kubernetes-sigs/prometheus-adapter[prometheus-adapter^], or use KEDA's Prometheus trigger, described next, which queries Prometheus directly and needs no adapter. + +== Autoscale with KEDA + +KEDA drives the same scale subresource, but adds event-driven triggers, such as consumer group lag and arbitrary Prometheus queries, and can scale a pipeline to zero when it's idle. Under the hood, KEDA materializes and manages an HPA named `keda-hpa-` for you. + +Install KEDA if it isn't already present: + +[,bash] +---- +helm repo add kedacore https://kedacore.github.io/charts +helm upgrade --install keda kedacore/keda --namespace keda --create-namespace +---- + +=== Scale on consumer group lag + +For pipelines that consume from Redpanda, consumer group lag is usually the signal you actually want: it measures how far the pipeline is behind, not how busy its CPUs are. KEDA's `kafka` trigger works with Redpanda because Redpanda is Kafka API-compatible. + +. Create a ScaledObject that targets the Pipeline and watches the pipeline's consumer group: ++ +.`scaledobject-lag.yaml` +[,yaml] +---- +apiVersion: keda.sh/v1alpha1 +kind: ScaledObject +metadata: + name: orders-to-warehouse + namespace: +spec: + scaleTargetRef: + apiVersion: cluster.redpanda.com/v1alpha2 + kind: Pipeline + name: orders-to-warehouse + minReplicaCount: 1 + maxReplicaCount: 6 + pollingInterval: 10 # How often KEDA evaluates the trigger (seconds). + cooldownPeriod: 120 # Delay before scaling to zero (only with minReplicaCount: 0). + advanced: + horizontalPodAutoscalerConfig: + behavior: + scaleUp: + stabilizationWindowSeconds: 0 + scaleDown: + stabilizationWindowSeconds: 60 + policies: + - type: Pods + value: 2 + periodSeconds: 30 + triggers: + - type: kafka + metadata: + bootstrapServers: ..svc.cluster.local:9093 + consumerGroup: orders-to-warehouse-ingest + topic: orders + lagThreshold: "500" # Target lag per replica. + offsetResetPolicy: earliest + allowIdleConsumers: "false" # Never scale beyond the partition count. +---- ++ +Replace `..svc.cluster.local:9093` with the bootstrap address of your cluster's internal Kafka listener, and set `consumerGroup` and `topic` to match the pipeline's `input.redpanda` block. ++ +KEDA sizes the pipeline at roughly `total lag / lagThreshold` replicas, clamped between `minReplicaCount` and `maxReplicaCount`. With `allowIdleConsumers: "false"`, KEDA also caps replicas at the topic's partition count, matching consumer group parallelism. + +. If the listener requires TLS or SASL, add a https://keda.sh/docs/latest/scalers/apache-kafka/[TriggerAuthentication^] and reference it from the trigger: ++ +[,yaml] +---- +apiVersion: keda.sh/v1alpha1 +kind: TriggerAuthentication +metadata: + name: redpanda-kafka-auth + namespace: +spec: + secretTargetRef: + - parameter: sasl + name: keda-redpanda-auth + key: sasl # For example: scram_sha512 + - parameter: username + name: keda-redpanda-auth + key: username + - parameter: password + name: keda-redpanda-auth + key: password + - parameter: tls + name: keda-redpanda-auth + key: tls # enable + - parameter: ca + name: keda-redpanda-auth + key: ca.crt +---- ++ +Reference it in the ScaledObject trigger with `authenticationRef.name: redpanda-kafka-auth`. The referenced user needs `Describe` permissions on the topic and consumer group to read lag. + +. Apply the manifest and watch the pipeline scale as lag grows and drains: ++ +[,bash] +---- +kubectl apply -f scaledobject-lag.yaml +kubectl get scaledobject orders-to-warehouse --namespace +kubectl get pipeline orders-to-warehouse --namespace --watch +---- ++ +As producers outpace the pipeline, lag crosses the threshold and KEDA raises `spec.replicas`; the consumer group rebalances partitions across the new Pods, lag drains, and after the scale-down stabilization window KEDA scales the pipeline back in. + +=== Scale on Redpanda Connect metrics + +Redpanda Connect emits pipeline-level metrics such as `input_received`, `output_sent`, and latency timings on each Pod's `http` port (4195) at `/metrics`. Scaling on these lets you react to actual pipeline throughput even when the input isn't a Redpanda topic. + +. Get the metrics into Prometheus. With Prometheus Operator installed, set `connectController.monitoring.enabled=true` on the operator chart so it renders a PodMonitor for every pipeline: ++ +[,bash] +---- +helm upgrade --install redpanda-operator redpanda/operator \ + --namespace \ + --reuse-values \ + --set connectController.monitoring.enabled=true +kubectl get podmonitor --namespace +---- + +. Create a ScaledObject with a `prometheus` trigger. This example adds one replica for every 5,000 messages per second the pipeline receives: ++ +.`scaledobject-metrics.yaml` +[,yaml] +---- +apiVersion: keda.sh/v1alpha1 +kind: ScaledObject +metadata: + name: orders-to-warehouse-throughput + namespace: +spec: + scaleTargetRef: + apiVersion: cluster.redpanda.com/v1alpha2 + kind: Pipeline + name: orders-to-warehouse + minReplicaCount: 1 + maxReplicaCount: 4 + triggers: + - type: prometheus + metadata: + serverAddress: http://prometheus-operated.monitoring.svc:9090 + query: sum(rate(input_received{namespace="", pod=~"orders-to-warehouse-.*"}[2m])) + threshold: "5000" +---- + +. Apply the manifest and confirm that KEDA created its HPA against the Pipeline: ++ +[,bash] +---- +kubectl apply -f scaledobject-metrics.yaml +kubectl get hpa keda-hpa-orders-to-warehouse-throughput --namespace +---- + +=== Scale to zero + +Set `minReplicaCount: 0` on a ScaledObject to let KEDA stop an idle pipeline entirely and restart it when the trigger fires again, for example when lag reappears on the consumer group. KEDA manages the 0 to 1 transition itself, waiting `cooldownPeriod` seconds after the last trigger activity before scaling to zero. A plain HPA cannot do this. + +== How autoscaling interacts with the Pipeline spec + +* *`spec.paused` wins over the autoscaler.* A paused pipeline stays at zero replicas (phase `Stopped`) even while an HPA or KEDA keeps writing `spec.replicas`. When you unpause it, the pipeline resumes at the autoscaler-managed count. +* *An explicit `spec.replicas: 0` is preserved.* The operator does not re-default it to 1. Note that a plain HPA stops managing a target whose replica count is 0; this is standard HPA behavior. Scale back up manually, or use KEDA with `minReplicaCount: 0`, which manages the zero state itself. +* *Stop declaring `replicas` in manifests once an autoscaler owns it.* Every re-apply of a manifest that sets `spec.replicas` overwrites the autoscaler's value. Remove the field from the manifests you apply, or configure your GitOps tool to ignore it, for example with Argo CD's `ignoreDifferences`. + +== Next steps + +* xref:manage:kubernetes/k-connect-pipelines.adoc[] +* https://keda.sh/docs/latest/scalers/[KEDA scalers reference^] +* https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/[Kubernetes HPA documentation^] diff --git a/modules/manage/pages/kubernetes/k-connect-pipelines.adoc b/modules/manage/pages/kubernetes/k-connect-pipelines.adoc index 94d5acd2d1..bc3923941b 100644 --- a/modules/manage/pages/kubernetes/k-connect-pipelines.adoc +++ b/modules/manage/pages/kubernetes/k-connect-pipelines.adoc @@ -1,15 +1,26 @@ = Redpanda Connect Pipelines in Kubernetes :description: Run Redpanda Connect pipelines declaratively with the Redpanda Operator using the Pipeline resource. +:page-topic-type: how-to +:page-beta: true :env-kubernetes: true :page-categories: Management, Integration -:page-topic-type: concept +:personas: platform_operator, platform_engineer +:learning-objective-1: Enable the Connect controller on the Redpanda Operator +:learning-objective-2: Connect pipelines to Redpanda clusters with scoped credentials +:learning-objective-3: Inject secrets and cloud IAM identities into pipeline workloads -// DRAFT (DOC-2275): Hold until the Pipeline resource ships in a released operator -// and rp-connect-docs#457 is published. See that PR for the full guide this page routes to. +// DRAFT (DOC-2275): Hold until the Pipeline resource ships in a released operator. +// When rp-connect-docs#457 publishes, add a cross-link to connect:install:kubernetes-operator.adoc in Next steps. -Run xref:connect:home:index.adoc[Redpanda Connect] pipelines as Kubernetes resources managed by the Redpanda Operator. You define each pipeline in a `Pipeline` custom resource, and the operator deploys it, validates its configuration before startup, rolls it on every change, and reports its health in the resource status. Pipelines follow the same declarative workflow as your Topic and User resources, so you can build and operate streaming data pipelines with the same GitOps tooling you already use for cluster resources. This page explains when to choose the Pipeline resource over other options and where to find the deployment guide. +Run xref:connect:home:index.adoc[Redpanda Connect] pipelines as Kubernetes resources managed by the Redpanda Operator. You define each pipeline in a `Pipeline` custom resource, and the operator deploys it, validates its configuration before startup, rolls it on every change, and reports its health in the resource status. Pipelines follow the same declarative workflow as your Topic and User resources, so you can build and operate streaming data pipelines with the same GitOps tooling you already use for cluster resources. -This feature requires an xref:get-started:licensing/overview.adoc[Enterprise Edition license] and Redpanda Operator or later. +After reading this page, you will be able to: + +* [ ] {learning-objective-1} +* [ ] {learning-objective-2} +* [ ] {learning-objective-3} + +This feature is in beta. It requires an xref:get-started:licensing/overview.adoc[Enterprise Edition license] and Redpanda Operator or later. // TODO(DOC-2275): Set the real minimum operator version when the feature ships. == When to use the Pipeline resource @@ -23,8 +34,615 @@ Use the Pipeline resource when you want: For Kafka Connect, which is community-supported with Redpanda, see xref:manage:kubernetes/k-manage-connectors.adoc[]. To run Redpanda Connect on Kubernetes without the operator, see xref:connect:install:helm-chart.adoc[]. -== Get started +== Prerequisites + +You must have the following: + +* *Kubectl*: Ensure you have the https://kubernetes.io/docs/tasks/tools/#kubectl[`kubectl`^] command-line tool installed and configured to communicate with your cluster. +* *Redpanda Operator*: Ensure you have the xref:deploy:redpanda/kubernetes/k-production-deployment.adoc[Redpanda Operator] deployed in your Kubernetes cluster. +* *Enterprise Edition license*: The Connect controller is an enterprise feature. Without a valid license, pipelines are reconciled but never deployed. See <>. +* *Image access*: Your cluster must be able to pull the Redpanda Connect image (`docker.redpanda.com/redpandadata/connect` by default). + +== Enable the Connect controller + +The Connect controller ships inside the Redpanda Operator binary and is disabled by default. Enable it through the operator Helm chart, which configures both the controller and the enterprise license: + +. Create a Secret that holds your Enterprise Edition license: ++ +[,bash] +---- +kubectl create secret generic redpanda-license \ + --from-file=license= \ + --namespace +---- + +. Install or upgrade the operator with the Connect controller enabled: ++ +[,bash] +---- +helm upgrade --install redpanda-operator redpanda/operator \ + --namespace \ + --set connectController.enabled=true \ + --set enterprise.licenseSecretRef.name=redpanda-license \ + --set enterprise.licenseSecretRef.key=license +---- ++ +`connectController.enabled: true` adds the `--enable-connect` flag to the operator Deployment, and `enterprise.licenseSecretRef` mounts the license Secret into the operator Pod. To also create a PodMonitor for each pipeline, for example to scrape pipeline metrics with Prometheus Operator, set `connectController.monitoring.enabled=true`. + +. Verify that the Pipeline CRD is installed: ++ +[,bash] +---- +kubectl get crd pipelines.cluster.redpanda.com +---- + +== Deploy your first pipeline + +A pipeline is defined entirely in the `spec.configYaml` field, which holds a standard Redpanda Connect configuration. This minimal example generates data and writes it to the Pod's standard output, so you can try the workflow without a Redpanda cluster: + +. Create a file called `hello-pipeline.yaml`: ++ +.`hello-pipeline.yaml` +[,yaml] +---- +apiVersion: cluster.redpanda.com/v1alpha2 +kind: Pipeline +metadata: + name: hello + namespace: +spec: + configYaml: | + input: + generate: + interval: 1s + mapping: 'root.msg = "hello from connect"' + output: + stdout: {} +---- + +. Apply the manifest and watch the pipeline come up: ++ +[,bash] +---- +kubectl apply -f hello-pipeline.yaml +kubectl get pipeline hello --namespace --watch +---- ++ +Example output: ++ +[.no-copy] +---- +NAME READY PHASE REPLICAS AVAILABLE AGE +hello True Running 1 1 20s +---- + +. Inspect the pipeline's output: ++ +[,bash] +---- +kubectl logs deploy/hello -c connect --namespace -f +---- + +When you apply a Pipeline resource, the operator: + +. Renders a ConfigMap that holds the final `connect.yaml`. +. Renders a Deployment whose `lint` init container runs `redpanda-connect lint` against the configuration. The pipeline only starts if the configuration passes lint. A failing configuration surfaces as `ConfigValid: False` in the resource status. +. Stamps the Pod template with a `cluster.redpanda.com/config-checksum` annotation, so any later configuration change automatically rolls the Deployment. A second annotation, `cluster.redpanda.com/credentials-checksum`, tracks every referenced Secret and ConfigMap, so credential rotations also roll the pipeline. + +To stop a pipeline without deleting it, pause it. The workload scales to zero and the phase changes to `Stopped`: + +[,bash] +---- +kubectl patch pipeline hello --namespace --type merge -p '{"spec":{"paused":true}}' +---- + +== Connect a pipeline to a Redpanda cluster + +Production pipelines read from or write to a Redpanda cluster. Bind the pipeline to a Redpanda resource with `spec.cluster.clusterRef`, and give it SASL credentials with a xref:manage:kubernetes/security/authentication/k-user-controller.adoc[User resource] referenced through `spec.userRef`. The operator resolves the broker addresses, TLS material, and SASL credentials from the referenced resources and injects them into the rendered configuration, so your `configYaml` contains only the pipeline logic. + +This example deploys a pipeline that reads the `orders` topic and archives each order to S3. + +. Create a User resource for the pipeline. The user owns the pipeline's SASL identity, and its ACLs scope what the pipeline can access: ++ +.`orders-to-warehouse-user.yaml` +[,yaml] +---- +apiVersion: cluster.redpanda.com/v1alpha2 +kind: User +metadata: + name: orders-to-warehouse + namespace: +spec: + cluster: + clusterRef: + name: + authentication: + type: scram-sha-512 + password: + valueFrom: + secretKeyRef: + name: orders-to-warehouse-password + key: password + authorization: + acls: + - type: allow + resource: + type: topic + name: orders + patternType: literal + operations: [Read, Describe] + - type: allow + resource: + type: group + name: orders-to-warehouse-ingest + patternType: literal + operations: [Read, Describe] +---- ++ +[NOTE] +==== +Literal ACLs fit a single, static pipeline like this one. Pipelines whose configuration creates topics (for example, change data capture producing `mysql.*` topics) or consumes with dynamic consumer groups need `patternType: prefixed` grants instead. With a non-superuser pipeline user, a missing group ACL surfaces only as a consumer authorization error at runtime, so scope these grants up front. +==== + +. Create the Pipeline resource, referencing both the cluster and the user: ++ +.`orders-to-warehouse-pipeline.yaml` +[,yaml] +---- +apiVersion: cluster.redpanda.com/v1alpha2 +kind: Pipeline +metadata: + name: orders-to-warehouse + namespace: +spec: + cluster: + clusterRef: + name: # The operator resolves brokers and TLS. + userRef: + name: orders-to-warehouse # The operator resolves the password Secret and SASL mechanism. + valueSources: + - name: S3_SECRET_KEY + source: + secretKeyRef: + name: s3-creds + key: secret_access_key + configYaml: | + input: + redpanda: + # Only the pipeline logic. The operator injects seed_brokers, + # tls, and sasl from clusterRef and userRef. + topics: [orders] + consumer_group: orders-to-warehouse-ingest + pipeline: + processors: + - mapping: | + root = this + root.ingested_at = now() + output: + aws_s3: + bucket: warehouse-orders + region: us-east-2 + credentials: + secret: ${S3_SECRET_KEY} +---- + +. Apply both manifests and verify the pipeline: ++ +[,bash] +---- +kubectl apply -f orders-to-warehouse-user.yaml -f orders-to-warehouse-pipeline.yaml +kubectl get pipeline orders-to-warehouse --namespace +---- + +=== What the operator injects + +The operator merges the resolved connection into every `input.redpanda` and `output.redpanda` block in your configuration, and additionally renders the connection as the top-level `redpanda` block (the shared client), so `redpanda_common` components also work without inline credentials. The rendered configuration for the example above looks like this: + +[,yaml] +---- +input: + redpanda: + seed_brokers: # Injected + - redpanda-0.redpanda..svc.cluster.local.:9093 + - redpanda-1.redpanda..svc.cluster.local.:9093 + - redpanda-2.redpanda..svc.cluster.local.:9093 + tls: # Injected + enabled: true + root_cas_file: /etc/tls/certs/ca/ca.crt + sasl: # Injected + - mechanism: SCRAM-SHA-512 + username: ${REDPANDA_SASL_USERNAME} + password: ${REDPANDA_SASL_PASSWORD} + topics: [orders] # User-supplied + consumer_group: orders-to-warehouse-ingest # User-supplied +# ...plus the same connection rendered as the top-level redpanda block, +# and your pipeline and output sections, unchanged. +---- + +The SASL environment variables resolve from the User resource: the username is the User's name, the mechanism comes from `spec.authentication.type`, and the password is projected from the User's password Secret. + +Keys that you set yourself always win. For example, if you write `seed_brokers` inside `input.redpanda`, the operator leaves your value untouched and injects only the missing keys. This is the escape hatch for pipelines that point one input or output at a different cluster. Blocks for other connectors (`aws_s3`, `mongodb`, and so on) pass through untouched, so new Connect connectors work without operator changes. + +If you omit `userRef`, the operator falls back to the SASL identity of the cluster source (for `clusterRef`, the cluster's bootstrap user). Use a dedicated User with scoped ACLs in production so each pipeline authenticates with least-privilege credentials. + +== Connect to an external cluster + +To target a cluster that isn't managed by a Redpanda resource in the same namespace, such as a Redpanda Cloud cluster, a self-managed deployment outside Kubernetes, or another Kafka-compatible cluster, replace `clusterRef` with `cluster.staticConfiguration` and declare the connection inline. The injection contract is identical to `clusterRef`; only the source of the connection details differs. + +.`cross-region-mirror.yaml` +[,yaml] +---- +apiVersion: cluster.redpanda.com/v1alpha2 +kind: Pipeline +metadata: + name: cross-region-mirror + namespace: +spec: + cluster: + staticConfiguration: + kafka: + brokers: + - kafka.us-east.example.com:9094 + tls: + caCert: # Omit for publicly-issued certificates. + secretKeyRef: + name: external-kafka-ca + key: ca.crt + sasl: + mechanism: SCRAM-SHA-512 # PLAIN, SCRAM-SHA-256, or SCRAM-SHA-512 + username: pipeline-mirror-svc + password: + secretKeyRef: + name: external-kafka-creds + key: password + configYaml: | + input: + redpanda: + topics: [orders] + output: + redpanda: + topic: orders.mirrored +---- + +Note the following: + +* `staticConfiguration` and `userRef` are mutually exclusive: the static configuration carries its own SASL identity, so the API server rejects a Pipeline that sets both. `userRef` also requires `clusterRef`, because without a referenced cluster there is no cluster to authenticate against. +* The CA certificate can come from a Secret (`secretKeyRef`) or a ConfigMap (`configMapKeyRef`). For mutual TLS listeners, additionally set `tls.cert` and `tls.key` to present a client certificate. +* Setting any TLS certificate material enables TLS. `tls: {enabled: true}` alone requests TLS with publicly-issued certificates, and omitting the `tls` block connects without TLS. +* The SASL password also accepts an `inline` value. Avoid inline passwords outside development: the value is stored in plain text in the Pipeline spec. + +== Give a pipeline a cloud IAM identity + +Some connectors call cloud APIs directly, for example to read from Amazon RDS with IAM database authentication, or to write to S3 without static keys. Set `spec.serviceAccountName` to run the pipeline Pod under a dedicated Kubernetes ServiceAccount whose cloud IAM trust is scoped to that one pipeline: IAM Roles for Service Accounts (IRSA) on EKS, Workload Identity on GKE, or Pod Identity on AKS. The operator does not create the ServiceAccount; provision it, with its cloud IAM annotations, alongside your other infrastructure. + +The pipeline's cluster credentials (`cluster` and `userRef`) and its cloud identity (`serviceAccountName`) are independent trust boundaries: either can be set without the other. + +This example runs MySQL change data capture against an Amazon RDS database using IAM authentication, so no database password exists anywhere in the Pipeline, the Pod, or a Secret: + +. Create the ServiceAccount with the IAM role annotation. The IAM role's trust policy must be scoped to this exact ServiceAccount, and its policy must grant `rds-db:connect`: ++ +.`mysql-cdc-orders-sa.yaml` +[,yaml] +---- +apiVersion: v1 +kind: ServiceAccount +metadata: + name: mysql-cdc-orders-rds + namespace: + annotations: + eks.amazonaws.com/role-arn: arn:aws:iam:::role/mysql-cdc-orders-rds +---- + +. Create the Pipeline with `serviceAccountName` set: ++ +.`mysql-cdc-orders.yaml` +[,yaml] +---- +apiVersion: cluster.redpanda.com/v1alpha2 +kind: Pipeline +metadata: + name: mysql-cdc-orders + namespace: +spec: + cluster: + clusterRef: + name: + userRef: + name: mysql-cdc-orders-svc # SASL identity for the Redpanda output + serviceAccountName: mysql-cdc-orders-rds # Cloud IAM boundary for AWS calls + valueSources: + # No MYSQL_PASSWORD: IAM authentication supplies a short-lived token. + - name: MYSQL_HOST + source: + secretKeyRef: + name: mysql-cdc-creds + key: host + - name: MYSQL_USER + source: + secretKeyRef: + name: mysql-cdc-creds + key: username + configYaml: | + input: + mysql_cdc: + # Connect calls rds:GenerateDBAuthToken with the Pod's IAM role and + # uses the 15-minute token as the MySQL password. + dsn: "${MYSQL_USER}@tcp(${MYSQL_HOST}:3306)/shop?tls=skip-verify&allowCleartextPasswords=1" + aws: + enabled: true + region: us-east-2 + endpoint: "${MYSQL_HOST}:3306" + stream_snapshot: true + tables: [orders] + flavor: mysql + output: + redpanda: + topic: mysql.shop.orders + key: '${! @table }' +---- + +The Pod assumes only the annotated role, through the projected ServiceAccount token. `userRef` still gates access to Redpanda, while the ServiceAccount gates access to AWS. + +== Inject values and secrets with valueSources + +`spec.valueSources` projects named values into the pipeline Pod as environment variables, which your configuration references with `${NAME}` interpolation. Each entry is a single, typed pull from one of four sources, and you can mix sources freely in one list: + +* `inline`: A literal value in the Pipeline spec. +* `secretKeyRef`: One key from a Kubernetes Secret. +* `configMapKeyRef`: One key from a ConfigMap. +* `externalSecretRef`: A secret resolved through the https://external-secrets.io/[External Secrets Operator^], backed by a system such as Vault or 1Password. + +Because every value is a named, per-key pull, unused keys in a Secret never leak into the Pod environment, the environment variable name can differ from the Secret's key, and multiple pipelines can pull different keys from the same Secret. + +This example fans one topic out to MongoDB, Snowflake, and MySQL, drawing credentials from all four source types: + +.`orders-fanout.yaml` +[,yaml] +---- +apiVersion: cluster.redpanda.com/v1alpha2 +kind: Pipeline +metadata: + name: orders-fanout + namespace: +spec: + cluster: + clusterRef: + name: + userRef: + name: orders-fanout-svc + valueSources: + # MongoDB Atlas: full connection URI from a Secret. + - name: MONGO_URI + source: + secretKeyRef: + name: mongo-orders-atlas + key: connection_uri + # Snowflake key-pair auth: account and user inline, private key from a + # Secret, passphrase from an External Secrets Operator resource. + - name: SNOWFLAKE_ACCOUNT + source: + inline: "ab12345.us-east-1" + - name: SNOWFLAKE_USER + source: + inline: "PIPELINE_SVC" + - name: SNOWFLAKE_PRIVATE_KEY + source: + secretKeyRef: + name: snowflake-pipeline-svc + key: rsa_key.p8 + - name: SNOWFLAKE_PRIVATE_KEY_PASSPHRASE + source: + externalSecretRef: + name: snowflake-pipeline-svc-passphrase + # MySQL: host from a ConfigMap so the endpoint can rotate without + # touching the Pipeline resource. + - name: MYSQL_USER + source: + inline: "warehouse_writer" + - name: MYSQL_PASSWORD + source: + secretKeyRef: + name: mysql-warehouse-creds + key: password + - name: MYSQL_HOST + source: + configMapKeyRef: + name: warehouse-env + key: mysql_replica_host + configYaml: | + input: + redpanda: + topics: [orders] + consumer_group: orders-fanout + output: + broker: + outputs: + - mongodb: + url: ${MONGO_URI} + database: orders + collection: ingested + operation: insert-one + - snowflake_put: + account: ${SNOWFLAKE_ACCOUNT} + user: ${SNOWFLAKE_USER} + private_key: ${SNOWFLAKE_PRIVATE_KEY} + private_key_pass: ${SNOWFLAKE_PRIVATE_KEY_PASSPHRASE} + database: WAREHOUSE + schema: PUBLIC + stage: "@PIPELINE_STAGE" + - sql_insert: + driver: mysql + dsn: "${MYSQL_USER}:${MYSQL_PASSWORD}@tcp(${MYSQL_HOST}:3306)/warehouse?parseTime=true" + table: orders + columns: [order_id, ingested_at, payload] + args_mapping: | + root = [ this.id, this.ingested_at, this.format_json() ] +---- + +Resolution of every entry is reported by the `ValueSourcesResolved` status condition. + +== Customize the pipeline workload + +The Pipeline spec exposes standard workload settings: + +[,yaml] +---- +spec: + replicas: 3 # Default 1. Autoscalable through the scale subresource. + paused: true # Scale to zero without deleting the resource. + image: docker.redpanda.com/redpandadata/connect: # Override the Connect image. + resources: + requests: + cpu: 100m + memory: 256Mi + limits: + cpu: "1" + memory: 1Gi + serviceAccountName: my-pipeline-sa # Pod identity. See the cloud IAM section. + nodeSelector: + redpanda.com/node-pool: connect # Pin pipelines to a dedicated node pool. + tolerations: + - key: dedicated + operator: Equal + value: connect + effect: NoSchedule + affinity: {} # Standard Kubernetes affinity, merged with zones. + zones: ["us-east-2a", "us-east-2b"] # Spread Pods across availability zones. + budget: + maxUnavailable: 1 # Creates a PodDisruptionBudget. +---- + +The Connect image is resolved with the following precedence: `spec.image` on the Pipeline, then the operator chart's `connectController.image` values, then the default image baked into the operator binary. + +If a pipeline needs setup work before it starts, such as fetching certificates or waiting for a dependency, add `extraInitContainers`. They run to completion, in order, before the operator's built-in `lint` init container, so anything they stage into a shared volume is visible to both lint and the Connect runtime. Declare the backing volume in `extraVolumes` and mount it into the lint and Connect containers with `extraVolumeMounts`: + +[,yaml] +---- +spec: + extraVolumes: + - name: shared + emptyDir: {} + extraVolumeMounts: # Mounted into the lint and connect containers. + - name: shared + mountPath: /shared + readOnly: true + extraInitContainers: + - name: fetch-certs + image: curlimages/curl:8.11.0 + command: ["sh", "-c", "curl -fsSL $CERT_URL -o /shared/ca.pem"] + volumeMounts: + - name: shared + mountPath: /shared +---- + +The volume names `config`, `cluster-tls-ca`, and `cluster-tls-client` are reserved by the operator. + +== Autoscale pipelines + +The Pipeline resource implements the Kubernetes scale subresource, so `kubectl scale`, a HorizontalPodAutoscaler, or a KEDA ScaledObject can drive the replica count directly. For step-by-step HPA and KEDA examples, see xref:manage:kubernetes/k-autoscale-connect-pipelines.adoc[]. + +== Monitor pipelines + +Redpanda Connect serves its metrics on each pipeline Pod's `http` port (4195) at `/metrics`. If you run Prometheus Operator, set `connectController.monitoring.enabled=true` in the operator chart to render a PodMonitor for every pipeline, so metrics such as `input_received` and `output_sent` are scraped automatically. These metrics can also drive autoscaling. See xref:manage:kubernetes/k-autoscale-connect-pipelines.adoc[]. + +== License behavior + +The Connect controller requires a valid Enterprise Edition license to create or update pipeline workloads: + +* *No license, or expired license*: New Pipeline resources are reconciled but get no workload. The resource reports `License: False` in its status conditions and never reaches the `Running` phase. Pipelines that were already running keep processing data, but the operator stops applying spec updates to them until you configure a valid license. +* *License restored*: Reconciliation resumes automatically. You do not need to recreate Pipeline resources. + +The operator mirrors the license into each pipeline's namespace as a Pipeline-owned Secret named `-license`, injected into the Pod as `REDPANDA_LICENSE`, so the Connect runtime's own license checks pass without configuring the license twice. Anyone with permission to read Secrets in a namespace that hosts pipelines can read the license, so scope namespace RBAC accordingly. + +For how licensing works across Redpanda products, see xref:get-started:licensing/overview.adoc[]. + +== Check pipeline status + +[,bash] +---- +kubectl get pipeline --namespace -o yaml +---- + +The `.status.phase` field summarizes the pipeline state: + +[cols="1m,3a"] +|=== +| Phase | Meaning + +| Pending +| The resource is accepted but not yet acted on. + +| Provisioning +| The Deployment is created, but Pods are not yet ready. + +| Running +| The desired number of replicas is ready. + +| Stopped +| The pipeline is paused or scaled to zero. + +| Unknown +| The state could not be determined. +|=== + +The `.status.conditions` array reports each aspect of reconciliation: + +[cols="1m,3a"] +|=== +| Condition | True means + +| Ready +| The pipeline workload is healthy. This is the headline condition. + +| ConfigValid +| The Connect configuration passed lint. + +| ClusterRef +| The cluster source (`clusterRef` or `staticConfiguration`) was resolved. + +| UserRef +| The referenced User's credentials, including its password Secret, were resolved. + +| ValueSourcesResolved +| Every `valueSources` entry resolved to an existing object and key. + +| License +| The operator's enterprise license is valid and includes Redpanda Connect. +|=== + +Two operational guarantees are worth knowing: + +* *Pipeline Pods do not depend on the operator process.* The Deployment is owned by the Pipeline resource, not the operator, so operator restarts and upgrades never interrupt running pipelines. +* *Failures never tear down a running workload.* If a referenced Redpanda resource, User, Secret, ConfigMap, or the license becomes unresolvable, the controller reports the failing condition on the status and stops syncing, but the last-known-good Deployment keeps running. Only deleting the Pipeline resource removes its workload. + +== Troubleshoot + +[cols="1a,2a"] +|=== +| Symptom | Cause and resolution + +| `License: False` +| No valid license. Configure `enterprise.licenseSecretRef` in the operator chart. See <>. + +| `ConfigValid: False` +| The `lint` init container rejected the configuration. Run `kubectl logs -c lint` to see the lint error. The condition message carries a truncated copy. + +| `ClusterRef: False` +| The referenced Redpanda resource doesn't exist in the pipeline's namespace, or its CRD isn't installed. A `clusterRef` cannot point to another namespace. + +| Pod stuck in `Init` or `ImagePullBackOff` +| The cluster can't pull the Connect image or an `extraInitContainers` image. Check the image reference and pull secrets. + +| `redpanda_common` errors with `shared client not found` +| The pipeline isn't bound to a cluster. Set `cluster.clusterRef` (with `userRef` for SASL clusters) so the operator renders the top-level `redpanda` block. + +| `Ready: False` with reason `NameConflict` +| Another workload already owns an object with this pipeline's name, such as a ConfigMap or Deployment. Rename the pipeline or remove the conflicting object. The operator refuses to adopt objects it does not own. +|=== -For installation steps, pipeline examples, secrets handling, and lifecycle management, see xref:connect:install:kubernetes-operator.adoc[Run Pipelines with the Redpanda Operator] in the Redpanda Connect documentation. +== Next steps -For the Pipeline CRD source file, see the xref:reference:k-crd-index.adoc[CRD reference]. +* xref:manage:kubernetes/k-autoscale-connect-pipelines.adoc[] +* xref:reference:k-crd-index.adoc[] for the Pipeline CRD source definition +* xref:connect:home:index.adoc[] to explore Redpanda Connect components and configuration +* xref:get-started:licensing/overview.adoc[] diff --git a/modules/reference/pages/k-crd-index.adoc b/modules/reference/pages/k-crd-index.adoc index 9cf9373571..9a06401b5c 100644 --- a/modules/reference/pages/k-crd-index.adoc +++ b/modules/reference/pages/k-crd-index.adoc @@ -29,7 +29,7 @@ The CRD source definitions are maintained in the https://github.com/redpanda-dat | Pipeline | https://github.com/redpanda-data/redpanda-operator/blob/main/operator/config/crd/bases/cluster.redpanda.com_pipelines.yaml[`cluster.redpanda.com_pipelines.yaml`^] -| Defines Redpanda Connect pipeline resources. See xref:connect:install:kubernetes-operator.adoc[Run Pipelines with the Redpanda Operator]. +| Defines Redpanda Connect pipeline resources. See xref:manage:kubernetes/k-connect-pipelines.adoc[Redpanda Connect Pipelines in Kubernetes]. | Schema | https://github.com/redpanda-data/redpanda-operator/blob/main/operator/config/crd/bases/cluster.redpanda.com_schemas.yaml[`cluster.redpanda.com_schemas.yaml`^] From be7844c0b05c5afadac77a9c1ca8787a6529ea8e Mon Sep 17 00:00:00 2001 From: David Yu Date: Fri, 24 Jul 2026 11:12:32 -0700 Subject: [PATCH 04/14] docs: move the Pipeline what's-new entry into the v26.2.x section The feature ships with operator v26.2.x, not v26.3.x. Drop the provisional v26.3.x section and its TODO. Co-Authored-By: Claude Fable 5 --- .../pages/release-notes/operator.adoc | 27 +++++++------------ .../pages/kubernetes/k-connect-pipelines.adoc | 1 + 2 files changed, 11 insertions(+), 17 deletions(-) diff --git a/modules/get-started/pages/release-notes/operator.adoc b/modules/get-started/pages/release-notes/operator.adoc index b326189a38..795a21612c 100644 --- a/modules/get-started/pages/release-notes/operator.adoc +++ b/modules/get-started/pages/release-notes/operator.adoc @@ -10,12 +10,17 @@ See also: * xref:upgrade:k-rolling-upgrade.adoc[] -// TODO(DOC-2275): Confirm the operator version that ships the Pipeline resource -// before un-drafting. The feature is currently expected in v26.3.x; if it lands -// in a v26.2.x release instead, fold this section into v26.2.x below. -== Redpanda Operator v26.3.x +== Redpanda Operator v26.2.x + +link:https://github.com/redpanda-data/redpanda-operator/blob/release/v26.2.x/operator/CHANGELOG.md[Changelog^] -link:https://github.com/redpanda-data/redpanda-operator/blob/release/v26.3.x/operator/CHANGELOG.md[Changelog^] +=== Stretch Clusters + +The Redpanda Operator introduces Stretch Clusters: a single logical Redpanda cluster distributed across multiple Kubernetes clusters for multi-region high availability. Operators in each Kubernetes cluster coordinate through Raft consensus, and the new StretchCluster and RedpandaBrokerPool custom resources define the cluster and its per-cluster broker groups. + +The release also adds the xref:reference:rpk/rpk-k8s/rpk-k8s.adoc[`rpk k8s` plugin], with xref:reference:rpk/rpk-k8s/rpk-k8s-multicluster.adoc[`rpk k8s multicluster`] commands to bootstrap TLS certificates and peer configuration (`bootstrap`), check deployment health (`status`), and collect diagnostics bundles (`bundle`). + +For deployment steps, see xref:deploy:redpanda/kubernetes/k-stretch-clusters.adoc[]. === Redpanda Connect pipelines (beta) @@ -28,18 +33,6 @@ The Pipeline resource is in beta and requires an Enterprise Edition license. Ena * xref:manage:kubernetes/k-connect-pipelines.adoc[Redpanda Connect Pipelines in Kubernetes] -- Deploy pipelines, connect them to clusters, and inject secrets and cloud IAM identities. * xref:manage:kubernetes/k-autoscale-connect-pipelines.adoc[Autoscale Redpanda Connect Pipelines] -- Step-by-step HPA and KEDA autoscaling. -== Redpanda Operator v26.2.x - -link:https://github.com/redpanda-data/redpanda-operator/blob/release/v26.2.x/operator/CHANGELOG.md[Changelog^] - -=== Stretch Clusters - -The Redpanda Operator introduces Stretch Clusters: a single logical Redpanda cluster distributed across multiple Kubernetes clusters for multi-region high availability. Operators in each Kubernetes cluster coordinate through Raft consensus, and the new StretchCluster and RedpandaBrokerPool custom resources define the cluster and its per-cluster broker groups. - -The release also adds the xref:reference:rpk/rpk-k8s/rpk-k8s.adoc[`rpk k8s` plugin], with xref:reference:rpk/rpk-k8s/rpk-k8s-multicluster.adoc[`rpk k8s multicluster`] commands to bootstrap TLS certificates and peer configuration (`bootstrap`), check deployment health (`status`), and collect diagnostics bundles (`bundle`). - -For deployment steps, see xref:deploy:redpanda/kubernetes/k-stretch-clusters.adoc[]. - === Gateway API support for Redpanda Console (beta) The Console custom resource and the Redpanda Console Helm chart can expose Redpanda Console through a Kubernetes Gateway API HTTPRoute as an alternative to Ingress. You reference an existing Gateway in `gateway.parentRefs`, and the operator or chart renders and manages the HTTPRoute for you. You can enable only one of `gateway` or `ingress` at a time. diff --git a/modules/manage/pages/kubernetes/k-connect-pipelines.adoc b/modules/manage/pages/kubernetes/k-connect-pipelines.adoc index bc3923941b..8e12d0c16c 100644 --- a/modules/manage/pages/kubernetes/k-connect-pipelines.adoc +++ b/modules/manage/pages/kubernetes/k-connect-pipelines.adoc @@ -22,6 +22,7 @@ After reading this page, you will be able to: This feature is in beta. It requires an xref:get-started:licensing/overview.adoc[Enterprise Edition license] and Redpanda Operator or later. // TODO(DOC-2275): Set the real minimum operator version when the feature ships. +// Ships with the v26.2.x line; fill in the exact first release (expected v26.2.1). == When to use the Pipeline resource From 99760efb424d6f7ad774995af499e338e5bc16a9 Mon Sep 17 00:00:00 2001 From: David Yu Date: Fri, 24 Jul 2026 11:32:33 -0700 Subject: [PATCH 05/14] docs: add Redpanda Operator enterprise features section to the licensing overview Give the operator its own enterprise-features table, matching the Redpanda / Console / Connect sections: Stretch Clusters (multicluster operator refuses to start without a valid license; LicenseValid condition + status.licenseStatus on running clusters) and Connect pipelines (moved out of the Connect table). Add a 'Manage licenses for the Redpanda Operator' section covering enterprise.licenseSecretRef and per-cluster licensing via the Redpanda and StretchCluster resources. Co-Authored-By: Claude Fable 5 --- .../get-started/pages/licensing/overview.adoc | 42 +++++++++++++++++-- 1 file changed, 38 insertions(+), 4 deletions(-) diff --git a/modules/get-started/pages/licensing/overview.adoc b/modules/get-started/pages/licensing/overview.adoc index b5c82af8b9..50952ad3cf 100644 --- a/modules/get-started/pages/licensing/overview.adoc +++ b/modules/get-started/pages/licensing/overview.adoc @@ -243,10 +243,6 @@ The following enterprise features are available with a valid Enterprise Edition | Run Redpanda Connect using a FIPS-compliant version of `rpk`, the Redpanda command-line interface (CLI). | No change. -| xref:manage:kubernetes/k-connect-pipelines.adoc[Pipelines in Kubernetes] (Pipeline resource) -| Run Redpanda Connect pipelines declaratively with the Redpanda Operator. The operator's Connect controller requires a valid license, configured through the operator's `enterprise.licenseSecretRef` Helm chart values. -| When the license is missing, expired, or invalid, the operator stops creating and updating pipeline workloads: new Pipeline resources report `License: False` in their status and are never deployed, and existing pipelines keep running with their last-applied configuration but no longer receive updates. Applying a valid license resumes reconciliation automatically, without recreating any Pipeline resources. - | xref:connect:components:redpanda/about.adoc[Redpanda Connect configuration service] | A configuration block that you can use to send logs and status events to a topic on a Redpanda cluster. | No change. @@ -257,6 +253,26 @@ The following enterprise features are available with a valid Enterprise Edition |=== +[[operator]] +=== Redpanda Operator enterprise features + +The Redpanda Operator gates the following Kubernetes features on a valid Enterprise Edition license. In both cases, the operator reads the license from a Secret referenced by the `enterprise.licenseSecretRef` values in the operator Helm chart. + +.Enterprise features in the Redpanda Operator +[cols="1a,2a,2a"] +|=== +| Feature | Description | Restrictions Without Valid License + +| xref:deploy:redpanda/kubernetes/k-stretch-clusters.adoc[Stretch Clusters] (StretchCluster resource) +| A single logical Redpanda cluster distributed across multiple Kubernetes clusters for multi-region high availability, managed by the multicluster Redpanda Operator. +| The multicluster operator requires a valid license to start, so stretch clusters cannot be deployed or managed without one. On a running stretch cluster, the operator continuously reports license state through the StretchCluster resource's `LicenseValid` status condition and `status.licenseStatus`, including the expiration date and any enterprise features in violation. When the license expires, the stretch cluster keeps running and its brokers' enterprise features follow the restrictions in <>. + +| xref:manage:kubernetes/k-connect-pipelines.adoc[Redpanda Connect pipelines] (Pipeline resource) +| Run Redpanda Connect pipelines declaratively through the operator's Connect controller. +| When the license is missing, expired, or invalid, the operator stops creating and updating pipeline workloads: new Pipeline resources report `License: False` in their status and are never deployed, and existing pipelines keep running with their last-applied configuration but no longer receive updates. Applying a valid license resumes reconciliation automatically, without recreating any Pipeline resources. + +|=== + == Manage licenses for Redpanda Redpanda offers multiple ways to manage your license depending on your deployment method. Proper license management ensures access to advanced enterprise features and avoids restrictions when licenses are invalid or expired. @@ -275,6 +291,24 @@ Redpanda Console offers two methods for applying or updating a license, dependin Redpanda Connect offers multiple ways to apply or update your license. See xref:connect:get-started:licensing.adoc#apply-a-license-key-to-redpanda-connect[Apply a license key to Redpanda Connect]. +== Manage licenses for the Redpanda Operator + +The Redpanda Operator reads its license from a Kubernetes Secret referenced by the `enterprise.licenseSecretRef` values in the operator Helm chart: + +[source,bash] +---- +kubectl create secret generic redpanda-license \ + --from-file=license= \ + --namespace + +helm upgrade --install redpanda-operator redpanda/operator \ + --namespace \ + --set enterprise.licenseSecretRef.name=redpanda-license \ + --set enterprise.licenseSecretRef.key=license +---- + +This operator-level license gates the operator's own enterprise features, such as the <>. The clusters that the operator manages are licensed independently: set the `enterprise.license` or `enterprise.licenseSecretRef` field in a Redpanda or StretchCluster resource, and the operator applies that license to the cluster for you. See xref:get-started:licensing/add-license-redpanda/kubernetes.adoc[]. + == Next steps - xref:get-started:licensing/add-license-redpanda/index.adoc[] From 982e93a0cac01f9200c4174c44013d527714413e Mon Sep 17 00:00:00 2001 From: David Yu Date: Fri, 24 Jul 2026 11:35:11 -0700 Subject: [PATCH 06/14] docs: use the standard enterprise-license notice and require the Connect product entitlement MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Replace the bespoke license sentence on the pipelines page with the shared enterprise-license partial, and state everywhere the license is discussed (notice, prerequisites, license behavior, troubleshooting, licensing overview table) that the license must include the Redpanda Connect product — matching the operator's IncludesProduct(ProductConnect) check. Co-Authored-By: Claude Fable 5 --- modules/get-started/pages/licensing/overview.adoc | 4 ++-- .../pages/kubernetes/k-connect-pipelines.adoc | 14 +++++++++----- 2 files changed, 11 insertions(+), 7 deletions(-) diff --git a/modules/get-started/pages/licensing/overview.adoc b/modules/get-started/pages/licensing/overview.adoc index 50952ad3cf..2a9fd729c3 100644 --- a/modules/get-started/pages/licensing/overview.adoc +++ b/modules/get-started/pages/licensing/overview.adoc @@ -268,8 +268,8 @@ The Redpanda Operator gates the following Kubernetes features on a valid Enterpr | The multicluster operator requires a valid license to start, so stretch clusters cannot be deployed or managed without one. On a running stretch cluster, the operator continuously reports license state through the StretchCluster resource's `LicenseValid` status condition and `status.licenseStatus`, including the expiration date and any enterprise features in violation. When the license expires, the stretch cluster keeps running and its brokers' enterprise features follow the restrictions in <>. | xref:manage:kubernetes/k-connect-pipelines.adoc[Redpanda Connect pipelines] (Pipeline resource) -| Run Redpanda Connect pipelines declaratively through the operator's Connect controller. -| When the license is missing, expired, or invalid, the operator stops creating and updating pipeline workloads: new Pipeline resources report `License: False` in their status and are never deployed, and existing pipelines keep running with their last-applied configuration but no longer receive updates. Applying a valid license resumes reconciliation automatically, without recreating any Pipeline resources. +| Run Redpanda Connect pipelines declaratively through the operator's Connect controller. Requires a license that includes the Redpanda Connect product. +| When the license is missing, expired, invalid, or does not include the Redpanda Connect product, the operator stops creating and updating pipeline workloads: new Pipeline resources report `License: False` in their status and are never deployed, and existing pipelines keep running with their last-applied configuration but no longer receive updates. Applying a valid license resumes reconciliation automatically, without recreating any Pipeline resources. |=== diff --git a/modules/manage/pages/kubernetes/k-connect-pipelines.adoc b/modules/manage/pages/kubernetes/k-connect-pipelines.adoc index 8e12d0c16c..0f50aad798 100644 --- a/modules/manage/pages/kubernetes/k-connect-pipelines.adoc +++ b/modules/manage/pages/kubernetes/k-connect-pipelines.adoc @@ -20,10 +20,14 @@ After reading this page, you will be able to: * [ ] {learning-objective-2} * [ ] {learning-objective-3} -This feature is in beta. It requires an xref:get-started:licensing/overview.adoc[Enterprise Edition license] and Redpanda Operator or later. +This feature is in beta and requires Redpanda Operator or later. // TODO(DOC-2275): Set the real minimum operator version when the feature ships. // Ships with the v26.2.x line; fill in the exact first release (expected v26.2.1). +include::shared:partial$enterprise-license.adoc[] + +Your license must include the Redpanda Connect product. The operator treats a license without this entitlement like a missing license and does not deploy Pipeline resources. See <>. + == When to use the Pipeline resource Use the Pipeline resource when you want: @@ -41,7 +45,7 @@ You must have the following: * *Kubectl*: Ensure you have the https://kubernetes.io/docs/tasks/tools/#kubectl[`kubectl`^] command-line tool installed and configured to communicate with your cluster. * *Redpanda Operator*: Ensure you have the xref:deploy:redpanda/kubernetes/k-production-deployment.adoc[Redpanda Operator] deployed in your Kubernetes cluster. -* *Enterprise Edition license*: The Connect controller is an enterprise feature. Without a valid license, pipelines are reconciled but never deployed. See <>. +* *Enterprise Edition license*: The Connect controller is an enterprise feature and requires a license that includes the Redpanda Connect product. Without one, pipelines are reconciled but never deployed. See <>. * *Image access*: Your cluster must be able to pull the Redpanda Connect image (`docker.redpanda.com/redpandadata/connect` by default). == Enable the Connect controller @@ -548,9 +552,9 @@ Redpanda Connect serves its metrics on each pipeline Pod's `http` port (4195) at == License behavior -The Connect controller requires a valid Enterprise Edition license to create or update pipeline workloads: +The Connect controller requires a valid Enterprise Edition license that includes the Redpanda Connect product to create or update pipeline workloads: -* *No license, or expired license*: New Pipeline resources are reconciled but get no workload. The resource reports `License: False` in its status conditions and never reaches the `Running` phase. Pipelines that were already running keep processing data, but the operator stops applying spec updates to them until you configure a valid license. +* *No license, expired license, or a license without the Redpanda Connect product*: New Pipeline resources are reconciled but get no workload. The resource reports `License: False` in its status conditions and never reaches the `Running` phase. Pipelines that were already running keep processing data, but the operator stops applying spec updates to them until you configure a valid license. * *License restored*: Reconciliation resumes automatically. You do not need to recreate Pipeline resources. The operator mirrors the license into each pipeline's namespace as a Pipeline-owned Secret named `-license`, injected into the Pod as `REDPANDA_LICENSE`, so the Connect runtime's own license checks pass without configuring the license twice. Anyone with permission to read Secrets in a namespace that hosts pipelines can read the license, so scope namespace RBAC accordingly. @@ -623,7 +627,7 @@ Two operational guarantees are worth knowing: | Symptom | Cause and resolution | `License: False` -| No valid license. Configure `enterprise.licenseSecretRef` in the operator chart. See <>. +| No valid license, or the license does not include the Redpanda Connect product. Configure `enterprise.licenseSecretRef` in the operator chart. See <>. | `ConfigValid: False` | The `lint` init container rejected the configuration. Run `kubectl logs -c lint` to see the lint error. The condition message carries a truncated copy. From e8ffdbf96ff9c113751bdd4123293693848ead9a Mon Sep 17 00:00:00 2001 From: David Yu Date: Fri, 24 Jul 2026 14:16:23 -0700 Subject: [PATCH 07/14] docs: pin the Pipeline feature's minimum operator version to v26.2.1 Co-Authored-By: Claude Fable 5 --- modules/manage/pages/kubernetes/k-connect-pipelines.adoc | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/modules/manage/pages/kubernetes/k-connect-pipelines.adoc b/modules/manage/pages/kubernetes/k-connect-pipelines.adoc index 0f50aad798..64fc7599ca 100644 --- a/modules/manage/pages/kubernetes/k-connect-pipelines.adoc +++ b/modules/manage/pages/kubernetes/k-connect-pipelines.adoc @@ -20,9 +20,7 @@ After reading this page, you will be able to: * [ ] {learning-objective-2} * [ ] {learning-objective-3} -This feature is in beta and requires Redpanda Operator or later. -// TODO(DOC-2275): Set the real minimum operator version when the feature ships. -// Ships with the v26.2.x line; fill in the exact first release (expected v26.2.1). +This feature is in beta and requires Redpanda Operator v26.2.1 or later. include::shared:partial$enterprise-license.adoc[] From bacf10666eb76d2e7dfd942332aa9de58582e0ca Mon Sep 17 00:00:00 2001 From: JakeSCahill Date: Mon, 27 Jul 2026 08:43:14 +0100 Subject: [PATCH 08/14] docs: prepare Pipeline docs for 26.2 GA as a beta feature The Pipeline resource ships as a beta feature in the 26.2 GA operator (redpanda-operator#1677 merged to main on July 22), so remove the draft-hold markers and keep the release-note section under v26.2.x. Also apply standards review fixes: - Rename the page to 'Run Redpanda Connect Pipelines in Kubernetes' (verb phrase for a how-to) - Use the standard beta notice on both pipeline pages, including the autoscaling page, which set page-beta without an in-body disclosure - Mark the feature (beta) at its other discovery surfaces: the CRD index row, the licensing overview row, and the Kafka Connect page - Replace list-bullet double hyphens with colons in the release notes - Split semicolon-joined sentences, fix a passive section opener, avoid starting a sentence with inline code, spell out IAM on first use, and add lead-in sentences before the status command and the troubleshooting table - Cross-link the Redpanda Connect operator install guide from Next steps (resolves when rp-connect-docs#457 merges) --- .../get-started/pages/licensing/overview.adoc | 2 +- .../pages/release-notes/operator.adoc | 20 +++++++++---------- .../k-autoscale-connect-pipelines.adoc | 7 ++++--- .../pages/kubernetes/k-connect-pipelines.adoc | 19 ++++++++++-------- .../pages/kubernetes/k-manage-connectors.adoc | 2 +- modules/reference/pages/k-crd-index.adoc | 2 +- 6 files changed, 28 insertions(+), 24 deletions(-) diff --git a/modules/get-started/pages/licensing/overview.adoc b/modules/get-started/pages/licensing/overview.adoc index 2a9fd729c3..c33c6c6c67 100644 --- a/modules/get-started/pages/licensing/overview.adoc +++ b/modules/get-started/pages/licensing/overview.adoc @@ -267,7 +267,7 @@ The Redpanda Operator gates the following Kubernetes features on a valid Enterpr | A single logical Redpanda cluster distributed across multiple Kubernetes clusters for multi-region high availability, managed by the multicluster Redpanda Operator. | The multicluster operator requires a valid license to start, so stretch clusters cannot be deployed or managed without one. On a running stretch cluster, the operator continuously reports license state through the StretchCluster resource's `LicenseValid` status condition and `status.licenseStatus`, including the expiration date and any enterprise features in violation. When the license expires, the stretch cluster keeps running and its brokers' enterprise features follow the restrictions in <>. -| xref:manage:kubernetes/k-connect-pipelines.adoc[Redpanda Connect pipelines] (Pipeline resource) +| xref:manage:kubernetes/k-connect-pipelines.adoc[Redpanda Connect pipelines] (Pipeline resource, beta) | Run Redpanda Connect pipelines declaratively through the operator's Connect controller. Requires a license that includes the Redpanda Connect product. | When the license is missing, expired, invalid, or does not include the Redpanda Connect product, the operator stops creating and updating pipeline workloads: new Pipeline resources report `License: False` in their status and are never deployed, and existing pipelines keep running with their last-applied configuration but no longer receive updates. Applying a valid license resumes reconciliation automatically, without recreating any Pipeline resources. diff --git a/modules/get-started/pages/release-notes/operator.adoc b/modules/get-started/pages/release-notes/operator.adoc index 795a21612c..71598f7a32 100644 --- a/modules/get-started/pages/release-notes/operator.adoc +++ b/modules/get-started/pages/release-notes/operator.adoc @@ -14,14 +14,6 @@ See also: link:https://github.com/redpanda-data/redpanda-operator/blob/release/v26.2.x/operator/CHANGELOG.md[Changelog^] -=== Stretch Clusters - -The Redpanda Operator introduces Stretch Clusters: a single logical Redpanda cluster distributed across multiple Kubernetes clusters for multi-region high availability. Operators in each Kubernetes cluster coordinate through Raft consensus, and the new StretchCluster and RedpandaBrokerPool custom resources define the cluster and its per-cluster broker groups. - -The release also adds the xref:reference:rpk/rpk-k8s/rpk-k8s.adoc[`rpk k8s` plugin], with xref:reference:rpk/rpk-k8s/rpk-k8s-multicluster.adoc[`rpk k8s multicluster`] commands to bootstrap TLS certificates and peer configuration (`bootstrap`), check deployment health (`status`), and collect diagnostics bundles (`bundle`). - -For deployment steps, see xref:deploy:redpanda/kubernetes/k-stretch-clusters.adoc[]. - === Redpanda Connect pipelines (beta) The Redpanda Operator introduces the Pipeline custom resource, which runs xref:connect:home:index.adoc[Redpanda Connect] pipelines as declaratively managed Kubernetes workloads. You define the pipeline configuration in the resource, and the operator deploys it, lints it before startup, rolls it automatically on configuration and credential changes, and reports health through status conditions. A pipeline can reference a Redpanda resource and a User resource, and the operator injects broker addresses, TLS material, and SASL credentials into the rendered configuration for you. @@ -30,8 +22,16 @@ Pipelines also implement the Kubernetes scale subresource, so you can scale them The Pipeline resource is in beta and requires an Enterprise Edition license. Enable it with the operator chart's `connectController.enabled=true` setting. -* xref:manage:kubernetes/k-connect-pipelines.adoc[Redpanda Connect Pipelines in Kubernetes] -- Deploy pipelines, connect them to clusters, and inject secrets and cloud IAM identities. -* xref:manage:kubernetes/k-autoscale-connect-pipelines.adoc[Autoscale Redpanda Connect Pipelines] -- Step-by-step HPA and KEDA autoscaling. +* xref:manage:kubernetes/k-connect-pipelines.adoc[Run Redpanda Connect Pipelines in Kubernetes]: Deploy pipelines, connect them to clusters, and inject secrets and cloud IAM identities. +* xref:manage:kubernetes/k-autoscale-connect-pipelines.adoc[Autoscale Redpanda Connect Pipelines]: Step-by-step HPA and KEDA autoscaling. + +=== Stretch Clusters + +The Redpanda Operator introduces Stretch Clusters: a single logical Redpanda cluster distributed across multiple Kubernetes clusters for multi-region high availability. Operators in each Kubernetes cluster coordinate through Raft consensus, and the new StretchCluster and RedpandaBrokerPool custom resources define the cluster and its per-cluster broker groups. + +The release also adds the xref:reference:rpk/rpk-k8s/rpk-k8s.adoc[`rpk k8s` plugin], with xref:reference:rpk/rpk-k8s/rpk-k8s-multicluster.adoc[`rpk k8s multicluster`] commands to bootstrap TLS certificates and peer configuration (`bootstrap`), check deployment health (`status`), and collect diagnostics bundles (`bundle`). + +For deployment steps, see xref:deploy:redpanda/kubernetes/k-stretch-clusters.adoc[]. === Gateway API support for Redpanda Console (beta) diff --git a/modules/manage/pages/kubernetes/k-autoscale-connect-pipelines.adoc b/modules/manage/pages/kubernetes/k-autoscale-connect-pipelines.adoc index d146788104..6d754ab0ea 100644 --- a/modules/manage/pages/kubernetes/k-autoscale-connect-pipelines.adoc +++ b/modules/manage/pages/kubernetes/k-autoscale-connect-pipelines.adoc @@ -9,7 +9,6 @@ :learning-objective-2: Autoscale a pipeline on CPU and memory with a HorizontalPodAutoscaler :learning-objective-3: Autoscale a pipeline on consumer group lag or Connect metrics with KEDA -// DRAFT (DOC-2275): Hold until the Pipeline resource ships in a released operator. Match pipeline capacity to your workload by scaling the number of pipeline replicas, either manually or automatically based on resource usage, consumer group lag, or the metrics that Redpanda Connect emits. @@ -19,6 +18,8 @@ After reading this page, you will be able to: * [ ] {learning-objective-2} * [ ] {learning-objective-3} +This feature is in beta and requires Redpanda Operator 26.2 or later. Beta features are not recommended for production environments. + The xref:manage:kubernetes/k-connect-pipelines.adoc[Pipeline resource] implements the Kubernetes https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/[scale subresource^], so anything that speaks the `/scale` API can drive `spec.replicas`: `kubectl scale`, a HorizontalPodAutoscaler (HPA), or a https://keda.sh/[KEDA^] ScaledObject. IMPORTANT: Always target the Pipeline resource, never its Deployment. The operator resets the Deployment's replica count to the Pipeline's value on every sync, so an autoscaler pointed at the Deployment is continuously fought and undone. @@ -247,7 +248,7 @@ kubectl get scaledobject orders-to-warehouse --namespace kubectl get pipeline orders-to-warehouse --namespace --watch ---- + -As producers outpace the pipeline, lag crosses the threshold and KEDA raises `spec.replicas`; the consumer group rebalances partitions across the new Pods, lag drains, and after the scale-down stabilization window KEDA scales the pipeline back in. +As producers outpace the pipeline, lag crosses the threshold and KEDA raises `spec.replicas`. The consumer group rebalances partitions across the new Pods, lag drains, and after the scale-down stabilization window KEDA scales the pipeline back in. === Scale on Redpanda Connect metrics @@ -304,7 +305,7 @@ Set `minReplicaCount: 0` on a ScaledObject to let KEDA stop an idle pipeline ent == How autoscaling interacts with the Pipeline spec * *`spec.paused` wins over the autoscaler.* A paused pipeline stays at zero replicas (phase `Stopped`) even while an HPA or KEDA keeps writing `spec.replicas`. When you unpause it, the pipeline resumes at the autoscaler-managed count. -* *An explicit `spec.replicas: 0` is preserved.* The operator does not re-default it to 1. Note that a plain HPA stops managing a target whose replica count is 0; this is standard HPA behavior. Scale back up manually, or use KEDA with `minReplicaCount: 0`, which manages the zero state itself. +* *An explicit `spec.replicas: 0` is preserved.* The operator does not re-default it to 1. Note that a plain HPA stops managing a target whose replica count is 0. This is standard HPA behavior. Scale back up manually, or use KEDA with `minReplicaCount: 0`, which manages the zero state itself. * *Stop declaring `replicas` in manifests once an autoscaler owns it.* Every re-apply of a manifest that sets `spec.replicas` overwrites the autoscaler's value. Remove the field from the manifests you apply, or configure your GitOps tool to ignore it, for example with Argo CD's `ignoreDifferences`. == Next steps diff --git a/modules/manage/pages/kubernetes/k-connect-pipelines.adoc b/modules/manage/pages/kubernetes/k-connect-pipelines.adoc index 64fc7599ca..af3eedcca5 100644 --- a/modules/manage/pages/kubernetes/k-connect-pipelines.adoc +++ b/modules/manage/pages/kubernetes/k-connect-pipelines.adoc @@ -1,4 +1,4 @@ -= Redpanda Connect Pipelines in Kubernetes += Run Redpanda Connect Pipelines in Kubernetes :description: Run Redpanda Connect pipelines declaratively with the Redpanda Operator using the Pipeline resource. :page-topic-type: how-to :page-beta: true @@ -9,8 +9,6 @@ :learning-objective-2: Connect pipelines to Redpanda clusters with scoped credentials :learning-objective-3: Inject secrets and cloud IAM identities into pipeline workloads -// DRAFT (DOC-2275): Hold until the Pipeline resource ships in a released operator. -// When rp-connect-docs#457 publishes, add a cross-link to connect:install:kubernetes-operator.adoc in Next steps. Run xref:connect:home:index.adoc[Redpanda Connect] pipelines as Kubernetes resources managed by the Redpanda Operator. You define each pipeline in a `Pipeline` custom resource, and the operator deploys it, validates its configuration before startup, rolls it on every change, and reports its health in the resource status. Pipelines follow the same declarative workflow as your Topic and User resources, so you can build and operate streaming data pipelines with the same GitOps tooling you already use for cluster resources. @@ -20,7 +18,7 @@ After reading this page, you will be able to: * [ ] {learning-objective-2} * [ ] {learning-objective-3} -This feature is in beta and requires Redpanda Operator v26.2.1 or later. +This feature is in beta and requires Redpanda Operator 26.2 or later. Beta features are not recommended for production environments. To give feedback, reach out to the Redpanda team in https://redpanda.com/slack[Redpanda Community Slack^]. include::shared:partial$enterprise-license.adoc[] @@ -81,7 +79,7 @@ kubectl get crd pipelines.cluster.redpanda.com == Deploy your first pipeline -A pipeline is defined entirely in the `spec.configYaml` field, which holds a standard Redpanda Connect configuration. This minimal example generates data and writes it to the Pod's standard output, so you can try the workflow without a Redpanda cluster: +You define a pipeline entirely in the `spec.configYaml` field, which holds a standard Redpanda Connect configuration. This minimal example generates data and writes it to the Pod's standard output, so you can try the workflow without a Redpanda cluster: . Create a file called `hello-pipeline.yaml`: + @@ -270,7 +268,7 @@ If you omit `userRef`, the operator falls back to the SASL identity of the clust == Connect to an external cluster -To target a cluster that isn't managed by a Redpanda resource in the same namespace, such as a Redpanda Cloud cluster, a self-managed deployment outside Kubernetes, or another Kafka-compatible cluster, replace `clusterRef` with `cluster.staticConfiguration` and declare the connection inline. The injection contract is identical to `clusterRef`; only the source of the connection details differs. +To target a cluster that isn't managed by a Redpanda resource in the same namespace, such as a Redpanda Cloud cluster, a self-managed deployment outside Kubernetes, or another Kafka-compatible cluster, replace `clusterRef` with `cluster.staticConfiguration` and declare the connection inline. The injection contract is identical to `clusterRef`. Only the source of the connection details differs. .`cross-region-mirror.yaml` [,yaml] @@ -316,7 +314,7 @@ Note the following: == Give a pipeline a cloud IAM identity -Some connectors call cloud APIs directly, for example to read from Amazon RDS with IAM database authentication, or to write to S3 without static keys. Set `spec.serviceAccountName` to run the pipeline Pod under a dedicated Kubernetes ServiceAccount whose cloud IAM trust is scoped to that one pipeline: IAM Roles for Service Accounts (IRSA) on EKS, Workload Identity on GKE, or Pod Identity on AKS. The operator does not create the ServiceAccount; provision it, with its cloud IAM annotations, alongside your other infrastructure. +Some connectors call cloud APIs directly, for example to read from Amazon RDS with Identity and Access Management (IAM) database authentication, or to write to S3 without static keys. Set `spec.serviceAccountName` to run the pipeline Pod under a dedicated Kubernetes ServiceAccount whose cloud IAM trust is scoped to that one pipeline: IAM Roles for Service Accounts (IRSA) on EKS, Workload Identity on GKE, or Pod Identity on AKS. The operator does not create the ServiceAccount. Provision it, with its cloud IAM annotations, alongside your other infrastructure. The pipeline's cluster credentials (`cluster` and `userRef`) and its cloud identity (`serviceAccountName`) are independent trust boundaries: either can be set without the other. @@ -388,7 +386,7 @@ The Pod assumes only the annotated role, through the projected ServiceAccount to == Inject values and secrets with valueSources -`spec.valueSources` projects named values into the pipeline Pod as environment variables, which your configuration references with `${NAME}` interpolation. Each entry is a single, typed pull from one of four sources, and you can mix sources freely in one list: +Use `spec.valueSources` to project named values into the pipeline Pod as environment variables, which your configuration references with `${NAME}` interpolation. Each entry is a single, typed pull from one of four sources, and you can mix sources freely in one list: * `inline`: A literal value in the Pipeline spec. * `secretKeyRef`: One key from a Kubernetes Secret. @@ -561,6 +559,8 @@ For how licensing works across Redpanda products, see xref:get-started:licensing == Check pipeline status +Inspect a pipeline's phase and conditions with the following command: + [,bash] ---- kubectl get pipeline --namespace -o yaml @@ -620,6 +620,8 @@ Two operational guarantees are worth knowing: == Troubleshoot +Use this table to diagnose common failure conditions: + [cols="1a,2a"] |=== | Symptom | Cause and resolution @@ -646,6 +648,7 @@ Two operational guarantees are worth knowing: == Next steps * xref:manage:kubernetes/k-autoscale-connect-pipelines.adoc[] +* xref:connect:install:kubernetes-operator.adoc[] for the full pipeline configuration reference in the Redpanda Connect docs * xref:reference:k-crd-index.adoc[] for the Pipeline CRD source definition * xref:connect:home:index.adoc[] to explore Redpanda Connect components and configuration * xref:get-started:licensing/overview.adoc[] diff --git a/modules/manage/pages/kubernetes/k-manage-connectors.adoc b/modules/manage/pages/kubernetes/k-manage-connectors.adoc index 2424cd24fc..3211c6cb9c 100644 --- a/modules/manage/pages/kubernetes/k-manage-connectors.adoc +++ b/modules/manage/pages/kubernetes/k-manage-connectors.adoc @@ -4,7 +4,7 @@ :page-categories: Management, Integration :env-kubernetes: true -When you have Kafka Connect deployed, you can create and manage connectors using Redpanda Console or the Kafka Connect REST API. To build streaming data pipelines with enterprise support and without Kafka Connect infrastructure, consider running xref:manage:kubernetes/k-connect-pipelines.adoc[Redpanda Connect pipelines] with the Redpanda Operator instead. +When you have Kafka Connect deployed, you can create and manage connectors using Redpanda Console or the Kafka Connect REST API. To build streaming data pipelines with enterprise support and without Kafka Connect infrastructure, consider running xref:manage:kubernetes/k-connect-pipelines.adoc[Redpanda Connect pipelines] (beta) with the Redpanda Operator instead. include::shared:partial$community-supported-connectors.adoc[] diff --git a/modules/reference/pages/k-crd-index.adoc b/modules/reference/pages/k-crd-index.adoc index 9a06401b5c..a00166bc93 100644 --- a/modules/reference/pages/k-crd-index.adoc +++ b/modules/reference/pages/k-crd-index.adoc @@ -27,7 +27,7 @@ The CRD source definitions are maintained in the https://github.com/redpanda-dat | https://github.com/redpanda-data/redpanda-operator/blob/main/operator/config/crd/bases/cluster.redpanda.com_consoles.yaml[`cluster.redpanda.com_consoles.yaml`^] | Defines Redpanda Console resources. -| Pipeline +| Pipeline (beta) | https://github.com/redpanda-data/redpanda-operator/blob/main/operator/config/crd/bases/cluster.redpanda.com_pipelines.yaml[`cluster.redpanda.com_pipelines.yaml`^] | Defines Redpanda Connect pipeline resources. See xref:manage:kubernetes/k-connect-pipelines.adoc[Redpanda Connect Pipelines in Kubernetes]. From 5e53143b43be6cb6d4853cd66cc443bf149bf1f8 Mon Sep 17 00:00:00 2001 From: JakeSCahill Date: Mon, 27 Jul 2026 08:48:58 +0100 Subject: [PATCH 09/14] docs: use the badge macro for section-level beta markers Register the shared badge inline macro in the playbook (already used by cloud-docs; the docs playbook never registered it, which is why no page here could use it) and apply badge::[label=beta] at the places where beta status applies to a feature mention inside a GA page: the CRD index row, the licensing overview row, and the Kafka Connect page cross-link. Whole-page beta status stays on the page-beta attribute. --- local-antora-playbook.yml | 1 + modules/get-started/pages/licensing/overview.adoc | 2 +- modules/manage/pages/kubernetes/k-manage-connectors.adoc | 2 +- modules/reference/pages/k-crd-index.adoc | 2 +- 4 files changed, 4 insertions(+), 3 deletions(-) diff --git a/local-antora-playbook.yml b/local-antora-playbook.yml index 1636d5f304..46666cce8d 100644 --- a/local-antora-playbook.yml +++ b/local-antora-playbook.yml @@ -39,6 +39,7 @@ asciidoc: - '@redpanda-data/docs-extensions-and-macros/macros/glossary' - '@redpanda-data/docs-extensions-and-macros/macros/config-ref' - '@redpanda-data/docs-extensions-and-macros/macros/helm-ref' + - '@redpanda-data/docs-extensions-and-macros/macros/badge' - '@redpanda-data/docs-extensions-and-macros/asciidoc-extensions/add-line-numbers-highlights' antora: extensions: diff --git a/modules/get-started/pages/licensing/overview.adoc b/modules/get-started/pages/licensing/overview.adoc index c33c6c6c67..b649d3ece5 100644 --- a/modules/get-started/pages/licensing/overview.adoc +++ b/modules/get-started/pages/licensing/overview.adoc @@ -267,7 +267,7 @@ The Redpanda Operator gates the following Kubernetes features on a valid Enterpr | A single logical Redpanda cluster distributed across multiple Kubernetes clusters for multi-region high availability, managed by the multicluster Redpanda Operator. | The multicluster operator requires a valid license to start, so stretch clusters cannot be deployed or managed without one. On a running stretch cluster, the operator continuously reports license state through the StretchCluster resource's `LicenseValid` status condition and `status.licenseStatus`, including the expiration date and any enterprise features in violation. When the license expires, the stretch cluster keeps running and its brokers' enterprise features follow the restrictions in <>. -| xref:manage:kubernetes/k-connect-pipelines.adoc[Redpanda Connect pipelines] (Pipeline resource, beta) +| xref:manage:kubernetes/k-connect-pipelines.adoc[Redpanda Connect pipelines] (Pipeline resource) badge::[label=beta] | Run Redpanda Connect pipelines declaratively through the operator's Connect controller. Requires a license that includes the Redpanda Connect product. | When the license is missing, expired, invalid, or does not include the Redpanda Connect product, the operator stops creating and updating pipeline workloads: new Pipeline resources report `License: False` in their status and are never deployed, and existing pipelines keep running with their last-applied configuration but no longer receive updates. Applying a valid license resumes reconciliation automatically, without recreating any Pipeline resources. diff --git a/modules/manage/pages/kubernetes/k-manage-connectors.adoc b/modules/manage/pages/kubernetes/k-manage-connectors.adoc index 3211c6cb9c..df76dda920 100644 --- a/modules/manage/pages/kubernetes/k-manage-connectors.adoc +++ b/modules/manage/pages/kubernetes/k-manage-connectors.adoc @@ -4,7 +4,7 @@ :page-categories: Management, Integration :env-kubernetes: true -When you have Kafka Connect deployed, you can create and manage connectors using Redpanda Console or the Kafka Connect REST API. To build streaming data pipelines with enterprise support and without Kafka Connect infrastructure, consider running xref:manage:kubernetes/k-connect-pipelines.adoc[Redpanda Connect pipelines] (beta) with the Redpanda Operator instead. +When you have Kafka Connect deployed, you can create and manage connectors using Redpanda Console or the Kafka Connect REST API. To build streaming data pipelines with enterprise support and without Kafka Connect infrastructure, consider running xref:manage:kubernetes/k-connect-pipelines.adoc[Redpanda Connect pipelines] badge::[label=beta] with the Redpanda Operator instead. include::shared:partial$community-supported-connectors.adoc[] diff --git a/modules/reference/pages/k-crd-index.adoc b/modules/reference/pages/k-crd-index.adoc index a00166bc93..da49ffed69 100644 --- a/modules/reference/pages/k-crd-index.adoc +++ b/modules/reference/pages/k-crd-index.adoc @@ -27,7 +27,7 @@ The CRD source definitions are maintained in the https://github.com/redpanda-dat | https://github.com/redpanda-data/redpanda-operator/blob/main/operator/config/crd/bases/cluster.redpanda.com_consoles.yaml[`cluster.redpanda.com_consoles.yaml`^] | Defines Redpanda Console resources. -| Pipeline (beta) +| Pipeline badge::[label=beta] | https://github.com/redpanda-data/redpanda-operator/blob/main/operator/config/crd/bases/cluster.redpanda.com_pipelines.yaml[`cluster.redpanda.com_pipelines.yaml`^] | Defines Redpanda Connect pipeline resources. See xref:manage:kubernetes/k-connect-pipelines.adoc[Redpanda Connect Pipelines in Kubernetes]. From ab65d4752acf0f615dd05b2a13f0685c09222b74 Mon Sep 17 00:00:00 2001 From: JakeSCahill Date: Mon, 27 Jul 2026 09:21:30 +0100 Subject: [PATCH 10/14] docs: make the Redpanda Connect docs canonical for Pipeline how-to content The connect repo's Run Pipelines with the Redpanda Operator page (rp-connect-docs#457) is the canonical Pipeline guide. Slim this component's page to a discovery surface: when to use the Pipeline resource, enabling the Connect controller, a first pipeline, and license behavior, with the deep workflows linked in the Redpanda Connect docs. - Move the external-cluster, cloud IAM, license-behavior, and troubleshooting sections to the connect page - Move the autoscaling page to the connect component and remove it here, including its nav entry - Retarget the release-note links --- modules/ROOT/nav.adoc | 1 - .../pages/release-notes/operator.adoc | 5 +- .../k-autoscale-connect-pipelines.adoc | 315 ----------- .../pages/kubernetes/k-connect-pipelines.adoc | 511 +----------------- 4 files changed, 10 insertions(+), 822 deletions(-) delete mode 100644 modules/manage/pages/kubernetes/k-autoscale-connect-pipelines.adoc diff --git a/modules/ROOT/nav.adoc b/modules/ROOT/nav.adoc index 864b10e471..30dc0e4f05 100644 --- a/modules/ROOT/nav.adoc +++ b/modules/ROOT/nav.adoc @@ -111,7 +111,6 @@ *** xref:manage:kubernetes/k-manage-topics.adoc[Manage Topics] *** xref:manage:kubernetes/k-manage-connectors.adoc[Manage Kafka Connect] *** xref:manage:kubernetes/k-connect-pipelines.adoc[Redpanda Connect Pipelines] -**** xref:manage:kubernetes/k-autoscale-connect-pipelines.adoc[Autoscaling] *** xref:manage:kubernetes/storage/index.adoc[Storage] **** xref:manage:kubernetes/storage/k-volume-types.adoc[Volume Types] **** xref:manage:kubernetes/storage/k-configure-storage.adoc[Configure Storage] diff --git a/modules/get-started/pages/release-notes/operator.adoc b/modules/get-started/pages/release-notes/operator.adoc index 71598f7a32..9bf2445ffe 100644 --- a/modules/get-started/pages/release-notes/operator.adoc +++ b/modules/get-started/pages/release-notes/operator.adoc @@ -22,8 +22,9 @@ Pipelines also implement the Kubernetes scale subresource, so you can scale them The Pipeline resource is in beta and requires an Enterprise Edition license. Enable it with the operator chart's `connectController.enabled=true` setting. -* xref:manage:kubernetes/k-connect-pipelines.adoc[Run Redpanda Connect Pipelines in Kubernetes]: Deploy pipelines, connect them to clusters, and inject secrets and cloud IAM identities. -* xref:manage:kubernetes/k-autoscale-connect-pipelines.adoc[Autoscale Redpanda Connect Pipelines]: Step-by-step HPA and KEDA autoscaling. +* xref:manage:kubernetes/k-connect-pipelines.adoc[Run Redpanda Connect Pipelines in Kubernetes]: Enable the Connect controller and deploy your first pipeline. +* xref:connect:install:kubernetes-operator.adoc[Run Pipelines with the Redpanda Operator]: The full Pipeline guide in the Redpanda Connect docs. +* xref:connect:install:kubernetes-operator-autoscale.adoc[Autoscale Pipelines]: Step-by-step HPA and KEDA autoscaling in the Redpanda Connect docs. === Stretch Clusters diff --git a/modules/manage/pages/kubernetes/k-autoscale-connect-pipelines.adoc b/modules/manage/pages/kubernetes/k-autoscale-connect-pipelines.adoc deleted file mode 100644 index 6d754ab0ea..0000000000 --- a/modules/manage/pages/kubernetes/k-autoscale-connect-pipelines.adoc +++ /dev/null @@ -1,315 +0,0 @@ -= Autoscale Redpanda Connect Pipelines -:description: Scale Redpanda Connect pipelines manually or automatically with kubectl scale, a HorizontalPodAutoscaler, or KEDA. -:page-topic-type: how-to -:page-beta: true -:env-kubernetes: true -:page-categories: Management, Integration -:personas: platform_operator, platform_engineer -:learning-objective-1: Scale a pipeline manually through the scale subresource -:learning-objective-2: Autoscale a pipeline on CPU and memory with a HorizontalPodAutoscaler -:learning-objective-3: Autoscale a pipeline on consumer group lag or Connect metrics with KEDA - - -Match pipeline capacity to your workload by scaling the number of pipeline replicas, either manually or automatically based on resource usage, consumer group lag, or the metrics that Redpanda Connect emits. - -After reading this page, you will be able to: - -* [ ] {learning-objective-1} -* [ ] {learning-objective-2} -* [ ] {learning-objective-3} - -This feature is in beta and requires Redpanda Operator 26.2 or later. Beta features are not recommended for production environments. - -The xref:manage:kubernetes/k-connect-pipelines.adoc[Pipeline resource] implements the Kubernetes https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/[scale subresource^], so anything that speaks the `/scale` API can drive `spec.replicas`: `kubectl scale`, a HorizontalPodAutoscaler (HPA), or a https://keda.sh/[KEDA^] ScaledObject. - -IMPORTANT: Always target the Pipeline resource, never its Deployment. The operator resets the Deployment's replica count to the Pipeline's value on every sync, so an autoscaler pointed at the Deployment is continuously fought and undone. - -== Prerequisites - -You must have the following: - -* A running pipeline managed by the Redpanda Operator with the Connect controller enabled. See xref:manage:kubernetes/k-connect-pipelines.adoc[]. -* For HPA on CPU or memory: https://github.com/kubernetes-sigs/metrics-server[metrics-server^] installed in the cluster. -* For KEDA: https://keda.sh/docs/latest/deploy/[KEDA^] installed in the cluster. -* For scaling on Redpanda Connect metrics: a Prometheus installation that scrapes pipeline Pods, for example https://github.com/prometheus-operator/kube-prometheus[kube-prometheus-stack^] together with the operator chart's `connectController.monitoring.enabled=true` setting. - -Configure only one autoscaler per pipeline. An HPA and a KEDA ScaledObject targeting the same Pipeline fight each other. - -== Scale a pipeline manually - -Scale the Pipeline directly with `kubectl scale`: - -[,bash] ----- -kubectl scale pipeline/ --replicas=3 --namespace ----- - -The operator updates the pipeline's Deployment to match, and the pipeline's consumer group rebalances its partitions across the new replicas. - -To confirm that the scale subresource is wired up, read it directly: - -[,bash] ----- -kubectl get --raw "/apis/cluster.redpanda.com/v1alpha2/namespaces//pipelines//scale" ----- - -The response reports `spec.replicas`, `status.replicas`, and `status.selector`, the label selector that autoscalers use to find the pipeline's Pods. To list a pipeline's Pods yourself, the `app.kubernetes.io/instance=` label is the simplest filter. - -[NOTE] -==== -When you scale a pipeline that consumes from Redpanda, replicas beyond the topic's partition count sit idle, because a consumer group assigns each partition to at most one member. Size `--replicas`, and any autoscaler's maximum, to the partition count of the input topic. -==== - -== Autoscale on CPU and memory with HPA - -CPU-based and memory-based HPAs work without any extra configuration, because pipeline Pods always carry resource requests: if you don't set `spec.resources`, the operator applies default requests of `100m` CPU and `256Mi` memory. - -. Ensure metrics-server is running. If `kubectl top pods` returns metrics, you're ready. Otherwise, install it: -+ -[,bash] ----- -kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml ----- - -. Set explicit resource requests on the pipeline, so utilization percentages are meaningful for your workload: -+ -[,yaml] ----- -spec: - resources: - requests: - cpu: 100m - memory: 256Mi - limits: - cpu: 300m - memory: 512Mi ----- - -. Create an HPA whose `scaleTargetRef` points at the Pipeline: -+ -.`pipeline-hpa.yaml` -[,yaml] ----- -apiVersion: autoscaling/v2 -kind: HorizontalPodAutoscaler -metadata: - name: orders-to-warehouse - namespace: -spec: - scaleTargetRef: - apiVersion: cluster.redpanda.com/v1alpha2 - kind: Pipeline - name: orders-to-warehouse - minReplicas: 1 - maxReplicas: 4 - behavior: - scaleUp: - stabilizationWindowSeconds: 0 - scaleDown: - stabilizationWindowSeconds: 60 - metrics: - - type: Resource - resource: - name: cpu - target: - type: Utilization - averageUtilization: 50 - - type: Resource - resource: - name: memory - target: - type: Utilization - averageUtilization: 80 ----- - -. Apply the manifest and watch the HPA react. Resource metrics lag by about a minute: -+ -[,bash] ----- -kubectl apply -f pipeline-hpa.yaml -kubectl get hpa orders-to-warehouse --namespace --watch ----- -+ -Example output while the pipeline is busy. CPU is at 300% of its request, above the 50% target, so the HPA scales out toward `maxReplicas`: -+ -[.no-copy] ----- -NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE -orders-to-warehouse Pipeline/orders-to-warehouse cpu: 300%/50%, memory: 30%/80% 1 4 4 2m ----- - -. Verify that the Pipeline followed: -+ -[,bash] ----- -kubectl get pipeline orders-to-warehouse --namespace -kubectl get pods -l app.kubernetes.io/instance=orders-to-warehouse --namespace ----- - -To scale an HPA on the metrics Redpanda Connect emits instead of CPU or memory, either expose them to the HPA as custom metrics through https://github.com/kubernetes-sigs/prometheus-adapter[prometheus-adapter^], or use KEDA's Prometheus trigger, described next, which queries Prometheus directly and needs no adapter. - -== Autoscale with KEDA - -KEDA drives the same scale subresource, but adds event-driven triggers, such as consumer group lag and arbitrary Prometheus queries, and can scale a pipeline to zero when it's idle. Under the hood, KEDA materializes and manages an HPA named `keda-hpa-` for you. - -Install KEDA if it isn't already present: - -[,bash] ----- -helm repo add kedacore https://kedacore.github.io/charts -helm upgrade --install keda kedacore/keda --namespace keda --create-namespace ----- - -=== Scale on consumer group lag - -For pipelines that consume from Redpanda, consumer group lag is usually the signal you actually want: it measures how far the pipeline is behind, not how busy its CPUs are. KEDA's `kafka` trigger works with Redpanda because Redpanda is Kafka API-compatible. - -. Create a ScaledObject that targets the Pipeline and watches the pipeline's consumer group: -+ -.`scaledobject-lag.yaml` -[,yaml] ----- -apiVersion: keda.sh/v1alpha1 -kind: ScaledObject -metadata: - name: orders-to-warehouse - namespace: -spec: - scaleTargetRef: - apiVersion: cluster.redpanda.com/v1alpha2 - kind: Pipeline - name: orders-to-warehouse - minReplicaCount: 1 - maxReplicaCount: 6 - pollingInterval: 10 # How often KEDA evaluates the trigger (seconds). - cooldownPeriod: 120 # Delay before scaling to zero (only with minReplicaCount: 0). - advanced: - horizontalPodAutoscalerConfig: - behavior: - scaleUp: - stabilizationWindowSeconds: 0 - scaleDown: - stabilizationWindowSeconds: 60 - policies: - - type: Pods - value: 2 - periodSeconds: 30 - triggers: - - type: kafka - metadata: - bootstrapServers: ..svc.cluster.local:9093 - consumerGroup: orders-to-warehouse-ingest - topic: orders - lagThreshold: "500" # Target lag per replica. - offsetResetPolicy: earliest - allowIdleConsumers: "false" # Never scale beyond the partition count. ----- -+ -Replace `..svc.cluster.local:9093` with the bootstrap address of your cluster's internal Kafka listener, and set `consumerGroup` and `topic` to match the pipeline's `input.redpanda` block. -+ -KEDA sizes the pipeline at roughly `total lag / lagThreshold` replicas, clamped between `minReplicaCount` and `maxReplicaCount`. With `allowIdleConsumers: "false"`, KEDA also caps replicas at the topic's partition count, matching consumer group parallelism. - -. If the listener requires TLS or SASL, add a https://keda.sh/docs/latest/scalers/apache-kafka/[TriggerAuthentication^] and reference it from the trigger: -+ -[,yaml] ----- -apiVersion: keda.sh/v1alpha1 -kind: TriggerAuthentication -metadata: - name: redpanda-kafka-auth - namespace: -spec: - secretTargetRef: - - parameter: sasl - name: keda-redpanda-auth - key: sasl # For example: scram_sha512 - - parameter: username - name: keda-redpanda-auth - key: username - - parameter: password - name: keda-redpanda-auth - key: password - - parameter: tls - name: keda-redpanda-auth - key: tls # enable - - parameter: ca - name: keda-redpanda-auth - key: ca.crt ----- -+ -Reference it in the ScaledObject trigger with `authenticationRef.name: redpanda-kafka-auth`. The referenced user needs `Describe` permissions on the topic and consumer group to read lag. - -. Apply the manifest and watch the pipeline scale as lag grows and drains: -+ -[,bash] ----- -kubectl apply -f scaledobject-lag.yaml -kubectl get scaledobject orders-to-warehouse --namespace -kubectl get pipeline orders-to-warehouse --namespace --watch ----- -+ -As producers outpace the pipeline, lag crosses the threshold and KEDA raises `spec.replicas`. The consumer group rebalances partitions across the new Pods, lag drains, and after the scale-down stabilization window KEDA scales the pipeline back in. - -=== Scale on Redpanda Connect metrics - -Redpanda Connect emits pipeline-level metrics such as `input_received`, `output_sent`, and latency timings on each Pod's `http` port (4195) at `/metrics`. Scaling on these lets you react to actual pipeline throughput even when the input isn't a Redpanda topic. - -. Get the metrics into Prometheus. With Prometheus Operator installed, set `connectController.monitoring.enabled=true` on the operator chart so it renders a PodMonitor for every pipeline: -+ -[,bash] ----- -helm upgrade --install redpanda-operator redpanda/operator \ - --namespace \ - --reuse-values \ - --set connectController.monitoring.enabled=true -kubectl get podmonitor --namespace ----- - -. Create a ScaledObject with a `prometheus` trigger. This example adds one replica for every 5,000 messages per second the pipeline receives: -+ -.`scaledobject-metrics.yaml` -[,yaml] ----- -apiVersion: keda.sh/v1alpha1 -kind: ScaledObject -metadata: - name: orders-to-warehouse-throughput - namespace: -spec: - scaleTargetRef: - apiVersion: cluster.redpanda.com/v1alpha2 - kind: Pipeline - name: orders-to-warehouse - minReplicaCount: 1 - maxReplicaCount: 4 - triggers: - - type: prometheus - metadata: - serverAddress: http://prometheus-operated.monitoring.svc:9090 - query: sum(rate(input_received{namespace="", pod=~"orders-to-warehouse-.*"}[2m])) - threshold: "5000" ----- - -. Apply the manifest and confirm that KEDA created its HPA against the Pipeline: -+ -[,bash] ----- -kubectl apply -f scaledobject-metrics.yaml -kubectl get hpa keda-hpa-orders-to-warehouse-throughput --namespace ----- - -=== Scale to zero - -Set `minReplicaCount: 0` on a ScaledObject to let KEDA stop an idle pipeline entirely and restart it when the trigger fires again, for example when lag reappears on the consumer group. KEDA manages the 0 to 1 transition itself, waiting `cooldownPeriod` seconds after the last trigger activity before scaling to zero. A plain HPA cannot do this. - -== How autoscaling interacts with the Pipeline spec - -* *`spec.paused` wins over the autoscaler.* A paused pipeline stays at zero replicas (phase `Stopped`) even while an HPA or KEDA keeps writing `spec.replicas`. When you unpause it, the pipeline resumes at the autoscaler-managed count. -* *An explicit `spec.replicas: 0` is preserved.* The operator does not re-default it to 1. Note that a plain HPA stops managing a target whose replica count is 0. This is standard HPA behavior. Scale back up manually, or use KEDA with `minReplicaCount: 0`, which manages the zero state itself. -* *Stop declaring `replicas` in manifests once an autoscaler owns it.* Every re-apply of a manifest that sets `spec.replicas` overwrites the autoscaler's value. Remove the field from the manifests you apply, or configure your GitOps tool to ignore it, for example with Argo CD's `ignoreDifferences`. - -== Next steps - -* xref:manage:kubernetes/k-connect-pipelines.adoc[] -* https://keda.sh/docs/latest/scalers/[KEDA scalers reference^] -* https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/[Kubernetes HPA documentation^] diff --git a/modules/manage/pages/kubernetes/k-connect-pipelines.adoc b/modules/manage/pages/kubernetes/k-connect-pipelines.adoc index af3eedcca5..7db4fecf8f 100644 --- a/modules/manage/pages/kubernetes/k-connect-pipelines.adoc +++ b/modules/manage/pages/kubernetes/k-connect-pipelines.adoc @@ -6,8 +6,8 @@ :page-categories: Management, Integration :personas: platform_operator, platform_engineer :learning-objective-1: Enable the Connect controller on the Redpanda Operator -:learning-objective-2: Connect pipelines to Redpanda clusters with scoped credentials -:learning-objective-3: Inject secrets and cloud IAM identities into pipeline workloads +:learning-objective-2: Deploy a pipeline and verify that it runs +:learning-objective-3: Describe how Pipeline resources behave when the license is missing or expired Run xref:connect:home:index.adoc[Redpanda Connect] pipelines as Kubernetes resources managed by the Redpanda Operator. You define each pipeline in a `Pipeline` custom resource, and the operator deploys it, validates its configuration before startup, rolls it on every change, and reports its health in the resource status. Pipelines follow the same declarative workflow as your Topic and User resources, so you can build and operate streaming data pipelines with the same GitOps tooling you already use for cluster resources. @@ -23,7 +23,6 @@ This feature is in beta and requires Redpanda Operator 26.2 or later. Beta featu include::shared:partial$enterprise-license.adoc[] Your license must include the Redpanda Connect product. The operator treats a license without this entitlement like a missing license and does not deploy Pipeline resources. See <>. - == When to use the Pipeline resource Use the Pipeline resource when you want: @@ -34,7 +33,6 @@ Use the Pipeline resource when you want: * *Built-in connectors*: Redpanda Connect ships hundreds of connectors and processors, including AI integrations, without the JVM or plugin management that Kafka Connect requires. For Kafka Connect, which is community-supported with Redpanda, see xref:manage:kubernetes/k-manage-connectors.adoc[]. To run Redpanda Connect on Kubernetes without the operator, see xref:connect:install:helm-chart.adoc[]. - == Prerequisites You must have the following: @@ -43,7 +41,6 @@ You must have the following: * *Redpanda Operator*: Ensure you have the xref:deploy:redpanda/kubernetes/k-production-deployment.adoc[Redpanda Operator] deployed in your Kubernetes cluster. * *Enterprise Edition license*: The Connect controller is an enterprise feature and requires a license that includes the Redpanda Connect product. Without one, pipelines are reconciled but never deployed. See <>. * *Image access*: Your cluster must be able to pull the Redpanda Connect image (`docker.redpanda.com/redpandadata/connect` by default). - == Enable the Connect controller The Connect controller ships inside the Redpanda Operator binary and is disabled by default. Enable it through the operator Helm chart, which configures both the controller and the enterprise license: @@ -76,7 +73,6 @@ helm upgrade --install redpanda-operator redpanda/operator \ ---- kubectl get crd pipelines.cluster.redpanda.com ---- - == Deploy your first pipeline You define a pipeline entirely in the `spec.configYaml` field, which holds a standard Redpanda Connect configuration. This minimal example generates data and writes it to the Pod's standard output, so you can try the workflow without a Redpanda cluster: @@ -136,415 +132,11 @@ To stop a pipeline without deleting it, pause it. The workload scales to zero an ---- kubectl patch pipeline hello --namespace --type merge -p '{"spec":{"paused":true}}' ---- +== Configure pipelines -== Connect a pipeline to a Redpanda cluster - -Production pipelines read from or write to a Redpanda cluster. Bind the pipeline to a Redpanda resource with `spec.cluster.clusterRef`, and give it SASL credentials with a xref:manage:kubernetes/security/authentication/k-user-controller.adoc[User resource] referenced through `spec.userRef`. The operator resolves the broker addresses, TLS material, and SASL credentials from the referenced resources and injects them into the rendered configuration, so your `configYaml` contains only the pipeline logic. - -This example deploys a pipeline that reads the `orders` topic and archives each order to S3. - -. Create a User resource for the pipeline. The user owns the pipeline's SASL identity, and its ACLs scope what the pipeline can access: -+ -.`orders-to-warehouse-user.yaml` -[,yaml] ----- -apiVersion: cluster.redpanda.com/v1alpha2 -kind: User -metadata: - name: orders-to-warehouse - namespace: -spec: - cluster: - clusterRef: - name: - authentication: - type: scram-sha-512 - password: - valueFrom: - secretKeyRef: - name: orders-to-warehouse-password - key: password - authorization: - acls: - - type: allow - resource: - type: topic - name: orders - patternType: literal - operations: [Read, Describe] - - type: allow - resource: - type: group - name: orders-to-warehouse-ingest - patternType: literal - operations: [Read, Describe] ----- -+ -[NOTE] -==== -Literal ACLs fit a single, static pipeline like this one. Pipelines whose configuration creates topics (for example, change data capture producing `mysql.*` topics) or consumes with dynamic consumer groups need `patternType: prefixed` grants instead. With a non-superuser pipeline user, a missing group ACL surfaces only as a consumer authorization error at runtime, so scope these grants up front. -==== - -. Create the Pipeline resource, referencing both the cluster and the user: -+ -.`orders-to-warehouse-pipeline.yaml` -[,yaml] ----- -apiVersion: cluster.redpanda.com/v1alpha2 -kind: Pipeline -metadata: - name: orders-to-warehouse - namespace: -spec: - cluster: - clusterRef: - name: # The operator resolves brokers and TLS. - userRef: - name: orders-to-warehouse # The operator resolves the password Secret and SASL mechanism. - valueSources: - - name: S3_SECRET_KEY - source: - secretKeyRef: - name: s3-creds - key: secret_access_key - configYaml: | - input: - redpanda: - # Only the pipeline logic. The operator injects seed_brokers, - # tls, and sasl from clusterRef and userRef. - topics: [orders] - consumer_group: orders-to-warehouse-ingest - pipeline: - processors: - - mapping: | - root = this - root.ingested_at = now() - output: - aws_s3: - bucket: warehouse-orders - region: us-east-2 - credentials: - secret: ${S3_SECRET_KEY} ----- - -. Apply both manifests and verify the pipeline: -+ -[,bash] ----- -kubectl apply -f orders-to-warehouse-user.yaml -f orders-to-warehouse-pipeline.yaml -kubectl get pipeline orders-to-warehouse --namespace ----- - -=== What the operator injects - -The operator merges the resolved connection into every `input.redpanda` and `output.redpanda` block in your configuration, and additionally renders the connection as the top-level `redpanda` block (the shared client), so `redpanda_common` components also work without inline credentials. The rendered configuration for the example above looks like this: - -[,yaml] ----- -input: - redpanda: - seed_brokers: # Injected - - redpanda-0.redpanda..svc.cluster.local.:9093 - - redpanda-1.redpanda..svc.cluster.local.:9093 - - redpanda-2.redpanda..svc.cluster.local.:9093 - tls: # Injected - enabled: true - root_cas_file: /etc/tls/certs/ca/ca.crt - sasl: # Injected - - mechanism: SCRAM-SHA-512 - username: ${REDPANDA_SASL_USERNAME} - password: ${REDPANDA_SASL_PASSWORD} - topics: [orders] # User-supplied - consumer_group: orders-to-warehouse-ingest # User-supplied -# ...plus the same connection rendered as the top-level redpanda block, -# and your pipeline and output sections, unchanged. ----- - -The SASL environment variables resolve from the User resource: the username is the User's name, the mechanism comes from `spec.authentication.type`, and the password is projected from the User's password Secret. - -Keys that you set yourself always win. For example, if you write `seed_brokers` inside `input.redpanda`, the operator leaves your value untouched and injects only the missing keys. This is the escape hatch for pipelines that point one input or output at a different cluster. Blocks for other connectors (`aws_s3`, `mongodb`, and so on) pass through untouched, so new Connect connectors work without operator changes. - -If you omit `userRef`, the operator falls back to the SASL identity of the cluster source (for `clusterRef`, the cluster's bootstrap user). Use a dedicated User with scoped ACLs in production so each pipeline authenticates with least-privilege credentials. - -== Connect to an external cluster - -To target a cluster that isn't managed by a Redpanda resource in the same namespace, such as a Redpanda Cloud cluster, a self-managed deployment outside Kubernetes, or another Kafka-compatible cluster, replace `clusterRef` with `cluster.staticConfiguration` and declare the connection inline. The injection contract is identical to `clusterRef`. Only the source of the connection details differs. - -.`cross-region-mirror.yaml` -[,yaml] ----- -apiVersion: cluster.redpanda.com/v1alpha2 -kind: Pipeline -metadata: - name: cross-region-mirror - namespace: -spec: - cluster: - staticConfiguration: - kafka: - brokers: - - kafka.us-east.example.com:9094 - tls: - caCert: # Omit for publicly-issued certificates. - secretKeyRef: - name: external-kafka-ca - key: ca.crt - sasl: - mechanism: SCRAM-SHA-512 # PLAIN, SCRAM-SHA-256, or SCRAM-SHA-512 - username: pipeline-mirror-svc - password: - secretKeyRef: - name: external-kafka-creds - key: password - configYaml: | - input: - redpanda: - topics: [orders] - output: - redpanda: - topic: orders.mirrored ----- - -Note the following: - -* `staticConfiguration` and `userRef` are mutually exclusive: the static configuration carries its own SASL identity, so the API server rejects a Pipeline that sets both. `userRef` also requires `clusterRef`, because without a referenced cluster there is no cluster to authenticate against. -* The CA certificate can come from a Secret (`secretKeyRef`) or a ConfigMap (`configMapKeyRef`). For mutual TLS listeners, additionally set `tls.cert` and `tls.key` to present a client certificate. -* Setting any TLS certificate material enables TLS. `tls: {enabled: true}` alone requests TLS with publicly-issued certificates, and omitting the `tls` block connects without TLS. -* The SASL password also accepts an `inline` value. Avoid inline passwords outside development: the value is stored in plain text in the Pipeline spec. - -== Give a pipeline a cloud IAM identity - -Some connectors call cloud APIs directly, for example to read from Amazon RDS with Identity and Access Management (IAM) database authentication, or to write to S3 without static keys. Set `spec.serviceAccountName` to run the pipeline Pod under a dedicated Kubernetes ServiceAccount whose cloud IAM trust is scoped to that one pipeline: IAM Roles for Service Accounts (IRSA) on EKS, Workload Identity on GKE, or Pod Identity on AKS. The operator does not create the ServiceAccount. Provision it, with its cloud IAM annotations, alongside your other infrastructure. - -The pipeline's cluster credentials (`cluster` and `userRef`) and its cloud identity (`serviceAccountName`) are independent trust boundaries: either can be set without the other. - -This example runs MySQL change data capture against an Amazon RDS database using IAM authentication, so no database password exists anywhere in the Pipeline, the Pod, or a Secret: - -. Create the ServiceAccount with the IAM role annotation. The IAM role's trust policy must be scoped to this exact ServiceAccount, and its policy must grant `rds-db:connect`: -+ -.`mysql-cdc-orders-sa.yaml` -[,yaml] ----- -apiVersion: v1 -kind: ServiceAccount -metadata: - name: mysql-cdc-orders-rds - namespace: - annotations: - eks.amazonaws.com/role-arn: arn:aws:iam:::role/mysql-cdc-orders-rds ----- - -. Create the Pipeline with `serviceAccountName` set: -+ -.`mysql-cdc-orders.yaml` -[,yaml] ----- -apiVersion: cluster.redpanda.com/v1alpha2 -kind: Pipeline -metadata: - name: mysql-cdc-orders - namespace: -spec: - cluster: - clusterRef: - name: - userRef: - name: mysql-cdc-orders-svc # SASL identity for the Redpanda output - serviceAccountName: mysql-cdc-orders-rds # Cloud IAM boundary for AWS calls - valueSources: - # No MYSQL_PASSWORD: IAM authentication supplies a short-lived token. - - name: MYSQL_HOST - source: - secretKeyRef: - name: mysql-cdc-creds - key: host - - name: MYSQL_USER - source: - secretKeyRef: - name: mysql-cdc-creds - key: username - configYaml: | - input: - mysql_cdc: - # Connect calls rds:GenerateDBAuthToken with the Pod's IAM role and - # uses the 15-minute token as the MySQL password. - dsn: "${MYSQL_USER}@tcp(${MYSQL_HOST}:3306)/shop?tls=skip-verify&allowCleartextPasswords=1" - aws: - enabled: true - region: us-east-2 - endpoint: "${MYSQL_HOST}:3306" - stream_snapshot: true - tables: [orders] - flavor: mysql - output: - redpanda: - topic: mysql.shop.orders - key: '${! @table }' ----- - -The Pod assumes only the annotated role, through the projected ServiceAccount token. `userRef` still gates access to Redpanda, while the ServiceAccount gates access to AWS. - -== Inject values and secrets with valueSources - -Use `spec.valueSources` to project named values into the pipeline Pod as environment variables, which your configuration references with `${NAME}` interpolation. Each entry is a single, typed pull from one of four sources, and you can mix sources freely in one list: - -* `inline`: A literal value in the Pipeline spec. -* `secretKeyRef`: One key from a Kubernetes Secret. -* `configMapKeyRef`: One key from a ConfigMap. -* `externalSecretRef`: A secret resolved through the https://external-secrets.io/[External Secrets Operator^], backed by a system such as Vault or 1Password. - -Because every value is a named, per-key pull, unused keys in a Secret never leak into the Pod environment, the environment variable name can differ from the Secret's key, and multiple pipelines can pull different keys from the same Secret. - -This example fans one topic out to MongoDB, Snowflake, and MySQL, drawing credentials from all four source types: - -.`orders-fanout.yaml` -[,yaml] ----- -apiVersion: cluster.redpanda.com/v1alpha2 -kind: Pipeline -metadata: - name: orders-fanout - namespace: -spec: - cluster: - clusterRef: - name: - userRef: - name: orders-fanout-svc - valueSources: - # MongoDB Atlas: full connection URI from a Secret. - - name: MONGO_URI - source: - secretKeyRef: - name: mongo-orders-atlas - key: connection_uri - # Snowflake key-pair auth: account and user inline, private key from a - # Secret, passphrase from an External Secrets Operator resource. - - name: SNOWFLAKE_ACCOUNT - source: - inline: "ab12345.us-east-1" - - name: SNOWFLAKE_USER - source: - inline: "PIPELINE_SVC" - - name: SNOWFLAKE_PRIVATE_KEY - source: - secretKeyRef: - name: snowflake-pipeline-svc - key: rsa_key.p8 - - name: SNOWFLAKE_PRIVATE_KEY_PASSPHRASE - source: - externalSecretRef: - name: snowflake-pipeline-svc-passphrase - # MySQL: host from a ConfigMap so the endpoint can rotate without - # touching the Pipeline resource. - - name: MYSQL_USER - source: - inline: "warehouse_writer" - - name: MYSQL_PASSWORD - source: - secretKeyRef: - name: mysql-warehouse-creds - key: password - - name: MYSQL_HOST - source: - configMapKeyRef: - name: warehouse-env - key: mysql_replica_host - configYaml: | - input: - redpanda: - topics: [orders] - consumer_group: orders-fanout - output: - broker: - outputs: - - mongodb: - url: ${MONGO_URI} - database: orders - collection: ingested - operation: insert-one - - snowflake_put: - account: ${SNOWFLAKE_ACCOUNT} - user: ${SNOWFLAKE_USER} - private_key: ${SNOWFLAKE_PRIVATE_KEY} - private_key_pass: ${SNOWFLAKE_PRIVATE_KEY_PASSPHRASE} - database: WAREHOUSE - schema: PUBLIC - stage: "@PIPELINE_STAGE" - - sql_insert: - driver: mysql - dsn: "${MYSQL_USER}:${MYSQL_PASSWORD}@tcp(${MYSQL_HOST}:3306)/warehouse?parseTime=true" - table: orders - columns: [order_id, ingested_at, payload] - args_mapping: | - root = [ this.id, this.ingested_at, this.format_json() ] ----- - -Resolution of every entry is reported by the `ValueSourcesResolved` status condition. - -== Customize the pipeline workload - -The Pipeline spec exposes standard workload settings: - -[,yaml] ----- -spec: - replicas: 3 # Default 1. Autoscalable through the scale subresource. - paused: true # Scale to zero without deleting the resource. - image: docker.redpanda.com/redpandadata/connect: # Override the Connect image. - resources: - requests: - cpu: 100m - memory: 256Mi - limits: - cpu: "1" - memory: 1Gi - serviceAccountName: my-pipeline-sa # Pod identity. See the cloud IAM section. - nodeSelector: - redpanda.com/node-pool: connect # Pin pipelines to a dedicated node pool. - tolerations: - - key: dedicated - operator: Equal - value: connect - effect: NoSchedule - affinity: {} # Standard Kubernetes affinity, merged with zones. - zones: ["us-east-2a", "us-east-2b"] # Spread Pods across availability zones. - budget: - maxUnavailable: 1 # Creates a PodDisruptionBudget. ----- - -The Connect image is resolved with the following precedence: `spec.image` on the Pipeline, then the operator chart's `connectController.image` values, then the default image baked into the operator binary. - -If a pipeline needs setup work before it starts, such as fetching certificates or waiting for a dependency, add `extraInitContainers`. They run to completion, in order, before the operator's built-in `lint` init container, so anything they stage into a shared volume is visible to both lint and the Connect runtime. Declare the backing volume in `extraVolumes` and mount it into the lint and Connect containers with `extraVolumeMounts`: - -[,yaml] ----- -spec: - extraVolumes: - - name: shared - emptyDir: {} - extraVolumeMounts: # Mounted into the lint and connect containers. - - name: shared - mountPath: /shared - readOnly: true - extraInitContainers: - - name: fetch-certs - image: curlimages/curl:8.11.0 - command: ["sh", "-c", "curl -fsSL $CERT_URL -o /shared/ca.pem"] - volumeMounts: - - name: shared - mountPath: /shared ----- - -The volume names `config`, `cluster-tls-ca`, and `cluster-tls-client` are reserved by the operator. - -== Autoscale pipelines - -The Pipeline resource implements the Kubernetes scale subresource, so `kubectl scale`, a HorizontalPodAutoscaler, or a KEDA ScaledObject can drive the replica count directly. For step-by-step HPA and KEDA examples, see xref:manage:kubernetes/k-autoscale-connect-pipelines.adoc[]. +The Redpanda Connect documentation covers the full Pipeline workflow, including connecting pipelines to Redpanda clusters with scoped credentials, targeting external clusters, injecting secrets and cloud IAM identities, customizing and spreading the workload, monitoring, and troubleshooting. See xref:connect:install:kubernetes-operator.adoc[]. -== Monitor pipelines - -Redpanda Connect serves its metrics on each pipeline Pod's `http` port (4195) at `/metrics`. If you run Prometheus Operator, set `connectController.monitoring.enabled=true` in the operator chart to render a PodMonitor for every pipeline, so metrics such as `input_received` and `output_sent` are scraped automatically. These metrics can also drive autoscaling. See xref:manage:kubernetes/k-autoscale-connect-pipelines.adoc[]. +Pipelines implement the Kubernetes scale subresource, so `kubectl scale`, a HorizontalPodAutoscaler, or a KEDA ScaledObject can drive the replica count, including on consumer group lag. See xref:connect:install:kubernetes-operator-autoscale.adoc[]. == License behavior @@ -557,98 +149,9 @@ The operator mirrors the license into each pipeline's namespace as a Pipeline-ow For how licensing works across Redpanda products, see xref:get-started:licensing/overview.adoc[]. -== Check pipeline status - -Inspect a pipeline's phase and conditions with the following command: - -[,bash] ----- -kubectl get pipeline --namespace -o yaml ----- - -The `.status.phase` field summarizes the pipeline state: - -[cols="1m,3a"] -|=== -| Phase | Meaning - -| Pending -| The resource is accepted but not yet acted on. - -| Provisioning -| The Deployment is created, but Pods are not yet ready. - -| Running -| The desired number of replicas is ready. - -| Stopped -| The pipeline is paused or scaled to zero. - -| Unknown -| The state could not be determined. -|=== - -The `.status.conditions` array reports each aspect of reconciliation: - -[cols="1m,3a"] -|=== -| Condition | True means - -| Ready -| The pipeline workload is healthy. This is the headline condition. - -| ConfigValid -| The Connect configuration passed lint. - -| ClusterRef -| The cluster source (`clusterRef` or `staticConfiguration`) was resolved. - -| UserRef -| The referenced User's credentials, including its password Secret, were resolved. - -| ValueSourcesResolved -| Every `valueSources` entry resolved to an existing object and key. - -| License -| The operator's enterprise license is valid and includes Redpanda Connect. -|=== - -Two operational guarantees are worth knowing: - -* *Pipeline Pods do not depend on the operator process.* The Deployment is owned by the Pipeline resource, not the operator, so operator restarts and upgrades never interrupt running pipelines. -* *Failures never tear down a running workload.* If a referenced Redpanda resource, User, Secret, ConfigMap, or the license becomes unresolvable, the controller reports the failing condition on the status and stops syncing, but the last-known-good Deployment keeps running. Only deleting the Pipeline resource removes its workload. - -== Troubleshoot - -Use this table to diagnose common failure conditions: - -[cols="1a,2a"] -|=== -| Symptom | Cause and resolution - -| `License: False` -| No valid license, or the license does not include the Redpanda Connect product. Configure `enterprise.licenseSecretRef` in the operator chart. See <>. - -| `ConfigValid: False` -| The `lint` init container rejected the configuration. Run `kubectl logs -c lint` to see the lint error. The condition message carries a truncated copy. - -| `ClusterRef: False` -| The referenced Redpanda resource doesn't exist in the pipeline's namespace, or its CRD isn't installed. A `clusterRef` cannot point to another namespace. - -| Pod stuck in `Init` or `ImagePullBackOff` -| The cluster can't pull the Connect image or an `extraInitContainers` image. Check the image reference and pull secrets. - -| `redpanda_common` errors with `shared client not found` -| The pipeline isn't bound to a cluster. Set `cluster.clusterRef` (with `userRef` for SASL clusters) so the operator renders the top-level `redpanda` block. - -| `Ready: False` with reason `NameConflict` -| Another workload already owns an object with this pipeline's name, such as a ConfigMap or Deployment. Rename the pipeline or remove the conflicting object. The operator refuses to adopt objects it does not own. -|=== - == Next steps -* xref:manage:kubernetes/k-autoscale-connect-pipelines.adoc[] -* xref:connect:install:kubernetes-operator.adoc[] for the full pipeline configuration reference in the Redpanda Connect docs +* xref:connect:install:kubernetes-operator.adoc[] for the full Pipeline guide in the Redpanda Connect docs +* xref:connect:install:kubernetes-operator-autoscale.adoc[] to autoscale pipelines with an HPA or KEDA * xref:reference:k-crd-index.adoc[] for the Pipeline CRD source definition -* xref:connect:home:index.adoc[] to explore Redpanda Connect components and configuration * xref:get-started:licensing/overview.adoc[] From ea0b49f90ca1a5bdca01537be10e63b9ac813420 Mon Sep 17 00:00:00 2001 From: JakeSCahill Date: Mon, 27 Jul 2026 11:12:38 +0100 Subject: [PATCH 11/14] docs: keep the Self-Managed docs canonical for the Pipeline guide Restore the full guide and the autoscaling page. The Pipeline resource is an operator CRD, and every other operator CRD guide lives here alongside the generated CRD reference, so this component owns the canonical how-to. The Redpanda Connect install section keeps a discovery page that links here (rp-connect-docs#457). Link the Pipeline CRD reference anchor directly (lands with #1830). --- modules/ROOT/nav.adoc | 1 + .../pages/release-notes/operator.adoc | 5 +- .../k-autoscale-connect-pipelines.adoc | 315 +++++++++++ .../pages/kubernetes/k-connect-pipelines.adoc | 512 +++++++++++++++++- 4 files changed, 822 insertions(+), 11 deletions(-) create mode 100644 modules/manage/pages/kubernetes/k-autoscale-connect-pipelines.adoc diff --git a/modules/ROOT/nav.adoc b/modules/ROOT/nav.adoc index 30dc0e4f05..864b10e471 100644 --- a/modules/ROOT/nav.adoc +++ b/modules/ROOT/nav.adoc @@ -111,6 +111,7 @@ *** xref:manage:kubernetes/k-manage-topics.adoc[Manage Topics] *** xref:manage:kubernetes/k-manage-connectors.adoc[Manage Kafka Connect] *** xref:manage:kubernetes/k-connect-pipelines.adoc[Redpanda Connect Pipelines] +**** xref:manage:kubernetes/k-autoscale-connect-pipelines.adoc[Autoscaling] *** xref:manage:kubernetes/storage/index.adoc[Storage] **** xref:manage:kubernetes/storage/k-volume-types.adoc[Volume Types] **** xref:manage:kubernetes/storage/k-configure-storage.adoc[Configure Storage] diff --git a/modules/get-started/pages/release-notes/operator.adoc b/modules/get-started/pages/release-notes/operator.adoc index 9bf2445ffe..71598f7a32 100644 --- a/modules/get-started/pages/release-notes/operator.adoc +++ b/modules/get-started/pages/release-notes/operator.adoc @@ -22,9 +22,8 @@ Pipelines also implement the Kubernetes scale subresource, so you can scale them The Pipeline resource is in beta and requires an Enterprise Edition license. Enable it with the operator chart's `connectController.enabled=true` setting. -* xref:manage:kubernetes/k-connect-pipelines.adoc[Run Redpanda Connect Pipelines in Kubernetes]: Enable the Connect controller and deploy your first pipeline. -* xref:connect:install:kubernetes-operator.adoc[Run Pipelines with the Redpanda Operator]: The full Pipeline guide in the Redpanda Connect docs. -* xref:connect:install:kubernetes-operator-autoscale.adoc[Autoscale Pipelines]: Step-by-step HPA and KEDA autoscaling in the Redpanda Connect docs. +* xref:manage:kubernetes/k-connect-pipelines.adoc[Run Redpanda Connect Pipelines in Kubernetes]: Deploy pipelines, connect them to clusters, and inject secrets and cloud IAM identities. +* xref:manage:kubernetes/k-autoscale-connect-pipelines.adoc[Autoscale Redpanda Connect Pipelines]: Step-by-step HPA and KEDA autoscaling. === Stretch Clusters diff --git a/modules/manage/pages/kubernetes/k-autoscale-connect-pipelines.adoc b/modules/manage/pages/kubernetes/k-autoscale-connect-pipelines.adoc new file mode 100644 index 0000000000..6d754ab0ea --- /dev/null +++ b/modules/manage/pages/kubernetes/k-autoscale-connect-pipelines.adoc @@ -0,0 +1,315 @@ += Autoscale Redpanda Connect Pipelines +:description: Scale Redpanda Connect pipelines manually or automatically with kubectl scale, a HorizontalPodAutoscaler, or KEDA. +:page-topic-type: how-to +:page-beta: true +:env-kubernetes: true +:page-categories: Management, Integration +:personas: platform_operator, platform_engineer +:learning-objective-1: Scale a pipeline manually through the scale subresource +:learning-objective-2: Autoscale a pipeline on CPU and memory with a HorizontalPodAutoscaler +:learning-objective-3: Autoscale a pipeline on consumer group lag or Connect metrics with KEDA + + +Match pipeline capacity to your workload by scaling the number of pipeline replicas, either manually or automatically based on resource usage, consumer group lag, or the metrics that Redpanda Connect emits. + +After reading this page, you will be able to: + +* [ ] {learning-objective-1} +* [ ] {learning-objective-2} +* [ ] {learning-objective-3} + +This feature is in beta and requires Redpanda Operator 26.2 or later. Beta features are not recommended for production environments. + +The xref:manage:kubernetes/k-connect-pipelines.adoc[Pipeline resource] implements the Kubernetes https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/[scale subresource^], so anything that speaks the `/scale` API can drive `spec.replicas`: `kubectl scale`, a HorizontalPodAutoscaler (HPA), or a https://keda.sh/[KEDA^] ScaledObject. + +IMPORTANT: Always target the Pipeline resource, never its Deployment. The operator resets the Deployment's replica count to the Pipeline's value on every sync, so an autoscaler pointed at the Deployment is continuously fought and undone. + +== Prerequisites + +You must have the following: + +* A running pipeline managed by the Redpanda Operator with the Connect controller enabled. See xref:manage:kubernetes/k-connect-pipelines.adoc[]. +* For HPA on CPU or memory: https://github.com/kubernetes-sigs/metrics-server[metrics-server^] installed in the cluster. +* For KEDA: https://keda.sh/docs/latest/deploy/[KEDA^] installed in the cluster. +* For scaling on Redpanda Connect metrics: a Prometheus installation that scrapes pipeline Pods, for example https://github.com/prometheus-operator/kube-prometheus[kube-prometheus-stack^] together with the operator chart's `connectController.monitoring.enabled=true` setting. + +Configure only one autoscaler per pipeline. An HPA and a KEDA ScaledObject targeting the same Pipeline fight each other. + +== Scale a pipeline manually + +Scale the Pipeline directly with `kubectl scale`: + +[,bash] +---- +kubectl scale pipeline/ --replicas=3 --namespace +---- + +The operator updates the pipeline's Deployment to match, and the pipeline's consumer group rebalances its partitions across the new replicas. + +To confirm that the scale subresource is wired up, read it directly: + +[,bash] +---- +kubectl get --raw "/apis/cluster.redpanda.com/v1alpha2/namespaces//pipelines//scale" +---- + +The response reports `spec.replicas`, `status.replicas`, and `status.selector`, the label selector that autoscalers use to find the pipeline's Pods. To list a pipeline's Pods yourself, the `app.kubernetes.io/instance=` label is the simplest filter. + +[NOTE] +==== +When you scale a pipeline that consumes from Redpanda, replicas beyond the topic's partition count sit idle, because a consumer group assigns each partition to at most one member. Size `--replicas`, and any autoscaler's maximum, to the partition count of the input topic. +==== + +== Autoscale on CPU and memory with HPA + +CPU-based and memory-based HPAs work without any extra configuration, because pipeline Pods always carry resource requests: if you don't set `spec.resources`, the operator applies default requests of `100m` CPU and `256Mi` memory. + +. Ensure metrics-server is running. If `kubectl top pods` returns metrics, you're ready. Otherwise, install it: ++ +[,bash] +---- +kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml +---- + +. Set explicit resource requests on the pipeline, so utilization percentages are meaningful for your workload: ++ +[,yaml] +---- +spec: + resources: + requests: + cpu: 100m + memory: 256Mi + limits: + cpu: 300m + memory: 512Mi +---- + +. Create an HPA whose `scaleTargetRef` points at the Pipeline: ++ +.`pipeline-hpa.yaml` +[,yaml] +---- +apiVersion: autoscaling/v2 +kind: HorizontalPodAutoscaler +metadata: + name: orders-to-warehouse + namespace: +spec: + scaleTargetRef: + apiVersion: cluster.redpanda.com/v1alpha2 + kind: Pipeline + name: orders-to-warehouse + minReplicas: 1 + maxReplicas: 4 + behavior: + scaleUp: + stabilizationWindowSeconds: 0 + scaleDown: + stabilizationWindowSeconds: 60 + metrics: + - type: Resource + resource: + name: cpu + target: + type: Utilization + averageUtilization: 50 + - type: Resource + resource: + name: memory + target: + type: Utilization + averageUtilization: 80 +---- + +. Apply the manifest and watch the HPA react. Resource metrics lag by about a minute: ++ +[,bash] +---- +kubectl apply -f pipeline-hpa.yaml +kubectl get hpa orders-to-warehouse --namespace --watch +---- ++ +Example output while the pipeline is busy. CPU is at 300% of its request, above the 50% target, so the HPA scales out toward `maxReplicas`: ++ +[.no-copy] +---- +NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE +orders-to-warehouse Pipeline/orders-to-warehouse cpu: 300%/50%, memory: 30%/80% 1 4 4 2m +---- + +. Verify that the Pipeline followed: ++ +[,bash] +---- +kubectl get pipeline orders-to-warehouse --namespace +kubectl get pods -l app.kubernetes.io/instance=orders-to-warehouse --namespace +---- + +To scale an HPA on the metrics Redpanda Connect emits instead of CPU or memory, either expose them to the HPA as custom metrics through https://github.com/kubernetes-sigs/prometheus-adapter[prometheus-adapter^], or use KEDA's Prometheus trigger, described next, which queries Prometheus directly and needs no adapter. + +== Autoscale with KEDA + +KEDA drives the same scale subresource, but adds event-driven triggers, such as consumer group lag and arbitrary Prometheus queries, and can scale a pipeline to zero when it's idle. Under the hood, KEDA materializes and manages an HPA named `keda-hpa-` for you. + +Install KEDA if it isn't already present: + +[,bash] +---- +helm repo add kedacore https://kedacore.github.io/charts +helm upgrade --install keda kedacore/keda --namespace keda --create-namespace +---- + +=== Scale on consumer group lag + +For pipelines that consume from Redpanda, consumer group lag is usually the signal you actually want: it measures how far the pipeline is behind, not how busy its CPUs are. KEDA's `kafka` trigger works with Redpanda because Redpanda is Kafka API-compatible. + +. Create a ScaledObject that targets the Pipeline and watches the pipeline's consumer group: ++ +.`scaledobject-lag.yaml` +[,yaml] +---- +apiVersion: keda.sh/v1alpha1 +kind: ScaledObject +metadata: + name: orders-to-warehouse + namespace: +spec: + scaleTargetRef: + apiVersion: cluster.redpanda.com/v1alpha2 + kind: Pipeline + name: orders-to-warehouse + minReplicaCount: 1 + maxReplicaCount: 6 + pollingInterval: 10 # How often KEDA evaluates the trigger (seconds). + cooldownPeriod: 120 # Delay before scaling to zero (only with minReplicaCount: 0). + advanced: + horizontalPodAutoscalerConfig: + behavior: + scaleUp: + stabilizationWindowSeconds: 0 + scaleDown: + stabilizationWindowSeconds: 60 + policies: + - type: Pods + value: 2 + periodSeconds: 30 + triggers: + - type: kafka + metadata: + bootstrapServers: ..svc.cluster.local:9093 + consumerGroup: orders-to-warehouse-ingest + topic: orders + lagThreshold: "500" # Target lag per replica. + offsetResetPolicy: earliest + allowIdleConsumers: "false" # Never scale beyond the partition count. +---- ++ +Replace `..svc.cluster.local:9093` with the bootstrap address of your cluster's internal Kafka listener, and set `consumerGroup` and `topic` to match the pipeline's `input.redpanda` block. ++ +KEDA sizes the pipeline at roughly `total lag / lagThreshold` replicas, clamped between `minReplicaCount` and `maxReplicaCount`. With `allowIdleConsumers: "false"`, KEDA also caps replicas at the topic's partition count, matching consumer group parallelism. + +. If the listener requires TLS or SASL, add a https://keda.sh/docs/latest/scalers/apache-kafka/[TriggerAuthentication^] and reference it from the trigger: ++ +[,yaml] +---- +apiVersion: keda.sh/v1alpha1 +kind: TriggerAuthentication +metadata: + name: redpanda-kafka-auth + namespace: +spec: + secretTargetRef: + - parameter: sasl + name: keda-redpanda-auth + key: sasl # For example: scram_sha512 + - parameter: username + name: keda-redpanda-auth + key: username + - parameter: password + name: keda-redpanda-auth + key: password + - parameter: tls + name: keda-redpanda-auth + key: tls # enable + - parameter: ca + name: keda-redpanda-auth + key: ca.crt +---- ++ +Reference it in the ScaledObject trigger with `authenticationRef.name: redpanda-kafka-auth`. The referenced user needs `Describe` permissions on the topic and consumer group to read lag. + +. Apply the manifest and watch the pipeline scale as lag grows and drains: ++ +[,bash] +---- +kubectl apply -f scaledobject-lag.yaml +kubectl get scaledobject orders-to-warehouse --namespace +kubectl get pipeline orders-to-warehouse --namespace --watch +---- ++ +As producers outpace the pipeline, lag crosses the threshold and KEDA raises `spec.replicas`. The consumer group rebalances partitions across the new Pods, lag drains, and after the scale-down stabilization window KEDA scales the pipeline back in. + +=== Scale on Redpanda Connect metrics + +Redpanda Connect emits pipeline-level metrics such as `input_received`, `output_sent`, and latency timings on each Pod's `http` port (4195) at `/metrics`. Scaling on these lets you react to actual pipeline throughput even when the input isn't a Redpanda topic. + +. Get the metrics into Prometheus. With Prometheus Operator installed, set `connectController.monitoring.enabled=true` on the operator chart so it renders a PodMonitor for every pipeline: ++ +[,bash] +---- +helm upgrade --install redpanda-operator redpanda/operator \ + --namespace \ + --reuse-values \ + --set connectController.monitoring.enabled=true +kubectl get podmonitor --namespace +---- + +. Create a ScaledObject with a `prometheus` trigger. This example adds one replica for every 5,000 messages per second the pipeline receives: ++ +.`scaledobject-metrics.yaml` +[,yaml] +---- +apiVersion: keda.sh/v1alpha1 +kind: ScaledObject +metadata: + name: orders-to-warehouse-throughput + namespace: +spec: + scaleTargetRef: + apiVersion: cluster.redpanda.com/v1alpha2 + kind: Pipeline + name: orders-to-warehouse + minReplicaCount: 1 + maxReplicaCount: 4 + triggers: + - type: prometheus + metadata: + serverAddress: http://prometheus-operated.monitoring.svc:9090 + query: sum(rate(input_received{namespace="", pod=~"orders-to-warehouse-.*"}[2m])) + threshold: "5000" +---- + +. Apply the manifest and confirm that KEDA created its HPA against the Pipeline: ++ +[,bash] +---- +kubectl apply -f scaledobject-metrics.yaml +kubectl get hpa keda-hpa-orders-to-warehouse-throughput --namespace +---- + +=== Scale to zero + +Set `minReplicaCount: 0` on a ScaledObject to let KEDA stop an idle pipeline entirely and restart it when the trigger fires again, for example when lag reappears on the consumer group. KEDA manages the 0 to 1 transition itself, waiting `cooldownPeriod` seconds after the last trigger activity before scaling to zero. A plain HPA cannot do this. + +== How autoscaling interacts with the Pipeline spec + +* *`spec.paused` wins over the autoscaler.* A paused pipeline stays at zero replicas (phase `Stopped`) even while an HPA or KEDA keeps writing `spec.replicas`. When you unpause it, the pipeline resumes at the autoscaler-managed count. +* *An explicit `spec.replicas: 0` is preserved.* The operator does not re-default it to 1. Note that a plain HPA stops managing a target whose replica count is 0. This is standard HPA behavior. Scale back up manually, or use KEDA with `minReplicaCount: 0`, which manages the zero state itself. +* *Stop declaring `replicas` in manifests once an autoscaler owns it.* Every re-apply of a manifest that sets `spec.replicas` overwrites the autoscaler's value. Remove the field from the manifests you apply, or configure your GitOps tool to ignore it, for example with Argo CD's `ignoreDifferences`. + +== Next steps + +* xref:manage:kubernetes/k-connect-pipelines.adoc[] +* https://keda.sh/docs/latest/scalers/[KEDA scalers reference^] +* https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/[Kubernetes HPA documentation^] diff --git a/modules/manage/pages/kubernetes/k-connect-pipelines.adoc b/modules/manage/pages/kubernetes/k-connect-pipelines.adoc index 7db4fecf8f..0286ba58a8 100644 --- a/modules/manage/pages/kubernetes/k-connect-pipelines.adoc +++ b/modules/manage/pages/kubernetes/k-connect-pipelines.adoc @@ -6,8 +6,8 @@ :page-categories: Management, Integration :personas: platform_operator, platform_engineer :learning-objective-1: Enable the Connect controller on the Redpanda Operator -:learning-objective-2: Deploy a pipeline and verify that it runs -:learning-objective-3: Describe how Pipeline resources behave when the license is missing or expired +:learning-objective-2: Connect pipelines to Redpanda clusters with scoped credentials +:learning-objective-3: Inject secrets and cloud IAM identities into pipeline workloads Run xref:connect:home:index.adoc[Redpanda Connect] pipelines as Kubernetes resources managed by the Redpanda Operator. You define each pipeline in a `Pipeline` custom resource, and the operator deploys it, validates its configuration before startup, rolls it on every change, and reports its health in the resource status. Pipelines follow the same declarative workflow as your Topic and User resources, so you can build and operate streaming data pipelines with the same GitOps tooling you already use for cluster resources. @@ -23,6 +23,7 @@ This feature is in beta and requires Redpanda Operator 26.2 or later. Beta featu include::shared:partial$enterprise-license.adoc[] Your license must include the Redpanda Connect product. The operator treats a license without this entitlement like a missing license and does not deploy Pipeline resources. See <>. + == When to use the Pipeline resource Use the Pipeline resource when you want: @@ -33,6 +34,7 @@ Use the Pipeline resource when you want: * *Built-in connectors*: Redpanda Connect ships hundreds of connectors and processors, including AI integrations, without the JVM or plugin management that Kafka Connect requires. For Kafka Connect, which is community-supported with Redpanda, see xref:manage:kubernetes/k-manage-connectors.adoc[]. To run Redpanda Connect on Kubernetes without the operator, see xref:connect:install:helm-chart.adoc[]. + == Prerequisites You must have the following: @@ -41,6 +43,7 @@ You must have the following: * *Redpanda Operator*: Ensure you have the xref:deploy:redpanda/kubernetes/k-production-deployment.adoc[Redpanda Operator] deployed in your Kubernetes cluster. * *Enterprise Edition license*: The Connect controller is an enterprise feature and requires a license that includes the Redpanda Connect product. Without one, pipelines are reconciled but never deployed. See <>. * *Image access*: Your cluster must be able to pull the Redpanda Connect image (`docker.redpanda.com/redpandadata/connect` by default). + == Enable the Connect controller The Connect controller ships inside the Redpanda Operator binary and is disabled by default. Enable it through the operator Helm chart, which configures both the controller and the enterprise license: @@ -73,6 +76,7 @@ helm upgrade --install redpanda-operator redpanda/operator \ ---- kubectl get crd pipelines.cluster.redpanda.com ---- + == Deploy your first pipeline You define a pipeline entirely in the `spec.configYaml` field, which holds a standard Redpanda Connect configuration. This minimal example generates data and writes it to the Pod's standard output, so you can try the workflow without a Redpanda cluster: @@ -132,11 +136,415 @@ To stop a pipeline without deleting it, pause it. The workload scales to zero an ---- kubectl patch pipeline hello --namespace --type merge -p '{"spec":{"paused":true}}' ---- -== Configure pipelines -The Redpanda Connect documentation covers the full Pipeline workflow, including connecting pipelines to Redpanda clusters with scoped credentials, targeting external clusters, injecting secrets and cloud IAM identities, customizing and spreading the workload, monitoring, and troubleshooting. See xref:connect:install:kubernetes-operator.adoc[]. +== Connect a pipeline to a Redpanda cluster + +Production pipelines read from or write to a Redpanda cluster. Bind the pipeline to a Redpanda resource with `spec.cluster.clusterRef`, and give it SASL credentials with a xref:manage:kubernetes/security/authentication/k-user-controller.adoc[User resource] referenced through `spec.userRef`. The operator resolves the broker addresses, TLS material, and SASL credentials from the referenced resources and injects them into the rendered configuration, so your `configYaml` contains only the pipeline logic. + +This example deploys a pipeline that reads the `orders` topic and archives each order to S3. + +. Create a User resource for the pipeline. The user owns the pipeline's SASL identity, and its ACLs scope what the pipeline can access: ++ +.`orders-to-warehouse-user.yaml` +[,yaml] +---- +apiVersion: cluster.redpanda.com/v1alpha2 +kind: User +metadata: + name: orders-to-warehouse + namespace: +spec: + cluster: + clusterRef: + name: + authentication: + type: scram-sha-512 + password: + valueFrom: + secretKeyRef: + name: orders-to-warehouse-password + key: password + authorization: + acls: + - type: allow + resource: + type: topic + name: orders + patternType: literal + operations: [Read, Describe] + - type: allow + resource: + type: group + name: orders-to-warehouse-ingest + patternType: literal + operations: [Read, Describe] +---- ++ +[NOTE] +==== +Literal ACLs fit a single, static pipeline like this one. Pipelines whose configuration creates topics (for example, change data capture producing `mysql.*` topics) or consumes with dynamic consumer groups need `patternType: prefixed` grants instead. With a non-superuser pipeline user, a missing group ACL surfaces only as a consumer authorization error at runtime, so scope these grants up front. +==== + +. Create the Pipeline resource, referencing both the cluster and the user: ++ +.`orders-to-warehouse-pipeline.yaml` +[,yaml] +---- +apiVersion: cluster.redpanda.com/v1alpha2 +kind: Pipeline +metadata: + name: orders-to-warehouse + namespace: +spec: + cluster: + clusterRef: + name: # The operator resolves brokers and TLS. + userRef: + name: orders-to-warehouse # The operator resolves the password Secret and SASL mechanism. + valueSources: + - name: S3_SECRET_KEY + source: + secretKeyRef: + name: s3-creds + key: secret_access_key + configYaml: | + input: + redpanda: + # Only the pipeline logic. The operator injects seed_brokers, + # tls, and sasl from clusterRef and userRef. + topics: [orders] + consumer_group: orders-to-warehouse-ingest + pipeline: + processors: + - mapping: | + root = this + root.ingested_at = now() + output: + aws_s3: + bucket: warehouse-orders + region: us-east-2 + credentials: + secret: ${S3_SECRET_KEY} +---- + +. Apply both manifests and verify the pipeline: ++ +[,bash] +---- +kubectl apply -f orders-to-warehouse-user.yaml -f orders-to-warehouse-pipeline.yaml +kubectl get pipeline orders-to-warehouse --namespace +---- + +=== What the operator injects + +The operator merges the resolved connection into every `input.redpanda` and `output.redpanda` block in your configuration, and additionally renders the connection as the top-level `redpanda` block (the shared client), so `redpanda_common` components also work without inline credentials. The rendered configuration for the example above looks like this: + +[,yaml] +---- +input: + redpanda: + seed_brokers: # Injected + - redpanda-0.redpanda..svc.cluster.local.:9093 + - redpanda-1.redpanda..svc.cluster.local.:9093 + - redpanda-2.redpanda..svc.cluster.local.:9093 + tls: # Injected + enabled: true + root_cas_file: /etc/tls/certs/ca/ca.crt + sasl: # Injected + - mechanism: SCRAM-SHA-512 + username: ${REDPANDA_SASL_USERNAME} + password: ${REDPANDA_SASL_PASSWORD} + topics: [orders] # User-supplied + consumer_group: orders-to-warehouse-ingest # User-supplied +# ...plus the same connection rendered as the top-level redpanda block, +# and your pipeline and output sections, unchanged. +---- + +The SASL environment variables resolve from the User resource: the username is the User's name, the mechanism comes from `spec.authentication.type`, and the password is projected from the User's password Secret. + +Keys that you set yourself always win. For example, if you write `seed_brokers` inside `input.redpanda`, the operator leaves your value untouched and injects only the missing keys. This is the escape hatch for pipelines that point one input or output at a different cluster. Blocks for other connectors (`aws_s3`, `mongodb`, and so on) pass through untouched, so new Connect connectors work without operator changes. + +If you omit `userRef`, the operator falls back to the SASL identity of the cluster source (for `clusterRef`, the cluster's bootstrap user). Use a dedicated User with scoped ACLs in production so each pipeline authenticates with least-privilege credentials. + +== Connect to an external cluster + +To target a cluster that isn't managed by a Redpanda resource in the same namespace, such as a Redpanda Cloud cluster, a self-managed deployment outside Kubernetes, or another Kafka-compatible cluster, replace `clusterRef` with `cluster.staticConfiguration` and declare the connection inline. The injection contract is identical to `clusterRef`. Only the source of the connection details differs. + +.`cross-region-mirror.yaml` +[,yaml] +---- +apiVersion: cluster.redpanda.com/v1alpha2 +kind: Pipeline +metadata: + name: cross-region-mirror + namespace: +spec: + cluster: + staticConfiguration: + kafka: + brokers: + - kafka.us-east.example.com:9094 + tls: + caCert: # Omit for publicly-issued certificates. + secretKeyRef: + name: external-kafka-ca + key: ca.crt + sasl: + mechanism: SCRAM-SHA-512 # PLAIN, SCRAM-SHA-256, or SCRAM-SHA-512 + username: pipeline-mirror-svc + password: + secretKeyRef: + name: external-kafka-creds + key: password + configYaml: | + input: + redpanda: + topics: [orders] + output: + redpanda: + topic: orders.mirrored +---- + +Note the following: + +* `staticConfiguration` and `userRef` are mutually exclusive: the static configuration carries its own SASL identity, so the API server rejects a Pipeline that sets both. `userRef` also requires `clusterRef`, because without a referenced cluster there is no cluster to authenticate against. +* The CA certificate can come from a Secret (`secretKeyRef`) or a ConfigMap (`configMapKeyRef`). For mutual TLS listeners, additionally set `tls.cert` and `tls.key` to present a client certificate. +* Setting any TLS certificate material enables TLS. `tls: {enabled: true}` alone requests TLS with publicly-issued certificates, and omitting the `tls` block connects without TLS. +* The SASL password also accepts an `inline` value. Avoid inline passwords outside development: the value is stored in plain text in the Pipeline spec. + +== Give a pipeline a cloud IAM identity + +Some connectors call cloud APIs directly, for example to read from Amazon RDS with Identity and Access Management (IAM) database authentication, or to write to S3 without static keys. Set `spec.serviceAccountName` to run the pipeline Pod under a dedicated Kubernetes ServiceAccount whose cloud IAM trust is scoped to that one pipeline: IAM Roles for Service Accounts (IRSA) on EKS, Workload Identity on GKE, or Pod Identity on AKS. The operator does not create the ServiceAccount. Provision it, with its cloud IAM annotations, alongside your other infrastructure. + +The pipeline's cluster credentials (`cluster` and `userRef`) and its cloud identity (`serviceAccountName`) are independent trust boundaries: either can be set without the other. + +This example runs MySQL change data capture against an Amazon RDS database using IAM authentication, so no database password exists anywhere in the Pipeline, the Pod, or a Secret: + +. Create the ServiceAccount with the IAM role annotation. The IAM role's trust policy must be scoped to this exact ServiceAccount, and its policy must grant `rds-db:connect`: ++ +.`mysql-cdc-orders-sa.yaml` +[,yaml] +---- +apiVersion: v1 +kind: ServiceAccount +metadata: + name: mysql-cdc-orders-rds + namespace: + annotations: + eks.amazonaws.com/role-arn: arn:aws:iam:::role/mysql-cdc-orders-rds +---- + +. Create the Pipeline with `serviceAccountName` set: ++ +.`mysql-cdc-orders.yaml` +[,yaml] +---- +apiVersion: cluster.redpanda.com/v1alpha2 +kind: Pipeline +metadata: + name: mysql-cdc-orders + namespace: +spec: + cluster: + clusterRef: + name: + userRef: + name: mysql-cdc-orders-svc # SASL identity for the Redpanda output + serviceAccountName: mysql-cdc-orders-rds # Cloud IAM boundary for AWS calls + valueSources: + # No MYSQL_PASSWORD: IAM authentication supplies a short-lived token. + - name: MYSQL_HOST + source: + secretKeyRef: + name: mysql-cdc-creds + key: host + - name: MYSQL_USER + source: + secretKeyRef: + name: mysql-cdc-creds + key: username + configYaml: | + input: + mysql_cdc: + # Connect calls rds:GenerateDBAuthToken with the Pod's IAM role and + # uses the 15-minute token as the MySQL password. + dsn: "${MYSQL_USER}@tcp(${MYSQL_HOST}:3306)/shop?tls=skip-verify&allowCleartextPasswords=1" + aws: + enabled: true + region: us-east-2 + endpoint: "${MYSQL_HOST}:3306" + stream_snapshot: true + tables: [orders] + flavor: mysql + output: + redpanda: + topic: mysql.shop.orders + key: '${! @table }' +---- + +The Pod assumes only the annotated role, through the projected ServiceAccount token. `userRef` still gates access to Redpanda, while the ServiceAccount gates access to AWS. + +== Inject values and secrets with valueSources + +Use `spec.valueSources` to project named values into the pipeline Pod as environment variables, which your configuration references with `${NAME}` interpolation. Each entry is a single, typed pull from one of four sources, and you can mix sources freely in one list: + +* `inline`: A literal value in the Pipeline spec. +* `secretKeyRef`: One key from a Kubernetes Secret. +* `configMapKeyRef`: One key from a ConfigMap. +* `externalSecretRef`: A secret resolved through the https://external-secrets.io/[External Secrets Operator^], backed by a system such as Vault or 1Password. + +Because every value is a named, per-key pull, unused keys in a Secret never leak into the Pod environment, the environment variable name can differ from the Secret's key, and multiple pipelines can pull different keys from the same Secret. + +This example fans one topic out to MongoDB, Snowflake, and MySQL, drawing credentials from all four source types: + +.`orders-fanout.yaml` +[,yaml] +---- +apiVersion: cluster.redpanda.com/v1alpha2 +kind: Pipeline +metadata: + name: orders-fanout + namespace: +spec: + cluster: + clusterRef: + name: + userRef: + name: orders-fanout-svc + valueSources: + # MongoDB Atlas: full connection URI from a Secret. + - name: MONGO_URI + source: + secretKeyRef: + name: mongo-orders-atlas + key: connection_uri + # Snowflake key-pair auth: account and user inline, private key from a + # Secret, passphrase from an External Secrets Operator resource. + - name: SNOWFLAKE_ACCOUNT + source: + inline: "ab12345.us-east-1" + - name: SNOWFLAKE_USER + source: + inline: "PIPELINE_SVC" + - name: SNOWFLAKE_PRIVATE_KEY + source: + secretKeyRef: + name: snowflake-pipeline-svc + key: rsa_key.p8 + - name: SNOWFLAKE_PRIVATE_KEY_PASSPHRASE + source: + externalSecretRef: + name: snowflake-pipeline-svc-passphrase + # MySQL: host from a ConfigMap so the endpoint can rotate without + # touching the Pipeline resource. + - name: MYSQL_USER + source: + inline: "warehouse_writer" + - name: MYSQL_PASSWORD + source: + secretKeyRef: + name: mysql-warehouse-creds + key: password + - name: MYSQL_HOST + source: + configMapKeyRef: + name: warehouse-env + key: mysql_replica_host + configYaml: | + input: + redpanda: + topics: [orders] + consumer_group: orders-fanout + output: + broker: + outputs: + - mongodb: + url: ${MONGO_URI} + database: orders + collection: ingested + operation: insert-one + - snowflake_put: + account: ${SNOWFLAKE_ACCOUNT} + user: ${SNOWFLAKE_USER} + private_key: ${SNOWFLAKE_PRIVATE_KEY} + private_key_pass: ${SNOWFLAKE_PRIVATE_KEY_PASSPHRASE} + database: WAREHOUSE + schema: PUBLIC + stage: "@PIPELINE_STAGE" + - sql_insert: + driver: mysql + dsn: "${MYSQL_USER}:${MYSQL_PASSWORD}@tcp(${MYSQL_HOST}:3306)/warehouse?parseTime=true" + table: orders + columns: [order_id, ingested_at, payload] + args_mapping: | + root = [ this.id, this.ingested_at, this.format_json() ] +---- + +Resolution of every entry is reported by the `ValueSourcesResolved` status condition. + +== Customize the pipeline workload + +The Pipeline spec exposes standard workload settings: + +[,yaml] +---- +spec: + replicas: 3 # Default 1. Autoscalable through the scale subresource. + paused: true # Scale to zero without deleting the resource. + image: docker.redpanda.com/redpandadata/connect: # Override the Connect image. + resources: + requests: + cpu: 100m + memory: 256Mi + limits: + cpu: "1" + memory: 1Gi + serviceAccountName: my-pipeline-sa # Pod identity. See the cloud IAM section. + nodeSelector: + redpanda.com/node-pool: connect # Pin pipelines to a dedicated node pool. + tolerations: + - key: dedicated + operator: Equal + value: connect + effect: NoSchedule + affinity: {} # Standard Kubernetes affinity, merged with zones. + zones: ["us-east-2a", "us-east-2b"] # Spread Pods across availability zones. + budget: + maxUnavailable: 1 # Creates a PodDisruptionBudget. +---- + +The Connect image is resolved with the following precedence: `spec.image` on the Pipeline, then the operator chart's `connectController.image` values, then the default image baked into the operator binary. + +If a pipeline needs setup work before it starts, such as fetching certificates or waiting for a dependency, add `extraInitContainers`. They run to completion, in order, before the operator's built-in `lint` init container, so anything they stage into a shared volume is visible to both lint and the Connect runtime. Declare the backing volume in `extraVolumes` and mount it into the lint and Connect containers with `extraVolumeMounts`: + +[,yaml] +---- +spec: + extraVolumes: + - name: shared + emptyDir: {} + extraVolumeMounts: # Mounted into the lint and connect containers. + - name: shared + mountPath: /shared + readOnly: true + extraInitContainers: + - name: fetch-certs + image: curlimages/curl:8.11.0 + command: ["sh", "-c", "curl -fsSL $CERT_URL -o /shared/ca.pem"] + volumeMounts: + - name: shared + mountPath: /shared +---- + +The volume names `config`, `cluster-tls-ca`, and `cluster-tls-client` are reserved by the operator. + +== Autoscale pipelines + +The Pipeline resource implements the Kubernetes scale subresource, so `kubectl scale`, a HorizontalPodAutoscaler, or a KEDA ScaledObject can drive the replica count directly. For step-by-step HPA and KEDA examples, see xref:manage:kubernetes/k-autoscale-connect-pipelines.adoc[]. -Pipelines implement the Kubernetes scale subresource, so `kubectl scale`, a HorizontalPodAutoscaler, or a KEDA ScaledObject can drive the replica count, including on consumer group lag. See xref:connect:install:kubernetes-operator-autoscale.adoc[]. +== Monitor pipelines + +Redpanda Connect serves its metrics on each pipeline Pod's `http` port (4195) at `/metrics`. If you run Prometheus Operator, set `connectController.monitoring.enabled=true` in the operator chart to render a PodMonitor for every pipeline, so metrics such as `input_received` and `output_sent` are scraped automatically. These metrics can also drive autoscaling. See xref:manage:kubernetes/k-autoscale-connect-pipelines.adoc[]. == License behavior @@ -149,9 +557,97 @@ The operator mirrors the license into each pipeline's namespace as a Pipeline-ow For how licensing works across Redpanda products, see xref:get-started:licensing/overview.adoc[]. +== Check pipeline status + +Inspect a pipeline's phase and conditions with the following command: + +[,bash] +---- +kubectl get pipeline --namespace -o yaml +---- + +The `.status.phase` field summarizes the pipeline state: + +[cols="1m,3a"] +|=== +| Phase | Meaning + +| Pending +| The resource is accepted but not yet acted on. + +| Provisioning +| The Deployment is created, but Pods are not yet ready. + +| Running +| The desired number of replicas is ready. + +| Stopped +| The pipeline is paused or scaled to zero. + +| Unknown +| The state could not be determined. +|=== + +The `.status.conditions` array reports each aspect of reconciliation: + +[cols="1m,3a"] +|=== +| Condition | True means + +| Ready +| The pipeline workload is healthy. This is the headline condition. + +| ConfigValid +| The Connect configuration passed lint. + +| ClusterRef +| The cluster source (`clusterRef` or `staticConfiguration`) was resolved. + +| UserRef +| The referenced User's credentials, including its password Secret, were resolved. + +| ValueSourcesResolved +| Every `valueSources` entry resolved to an existing object and key. + +| License +| The operator's enterprise license is valid and includes Redpanda Connect. +|=== + +Two operational guarantees are worth knowing: + +* *Pipeline Pods do not depend on the operator process.* The Deployment is owned by the Pipeline resource, not the operator, so operator restarts and upgrades never interrupt running pipelines. +* *Failures never tear down a running workload.* If a referenced Redpanda resource, User, Secret, ConfigMap, or the license becomes unresolvable, the controller reports the failing condition on the status and stops syncing, but the last-known-good Deployment keeps running. Only deleting the Pipeline resource removes its workload. + +== Troubleshoot + +Use this table to diagnose common failure conditions: + +[cols="1a,2a"] +|=== +| Symptom | Cause and resolution + +| `License: False` +| No valid license, or the license does not include the Redpanda Connect product. Configure `enterprise.licenseSecretRef` in the operator chart. See <>. + +| `ConfigValid: False` +| The `lint` init container rejected the configuration. Run `kubectl logs -c lint` to see the lint error. The condition message carries a truncated copy. + +| `ClusterRef: False` +| The referenced Redpanda resource doesn't exist in the pipeline's namespace, or its CRD isn't installed. A `clusterRef` cannot point to another namespace. + +| Pod stuck in `Init` or `ImagePullBackOff` +| The cluster can't pull the Connect image or an `extraInitContainers` image. Check the image reference and pull secrets. + +| `redpanda_common` errors with `shared client not found` +| The pipeline isn't bound to a cluster. Set `cluster.clusterRef` (with `userRef` for SASL clusters) so the operator renders the top-level `redpanda` block. + +| `Ready: False` with reason `NameConflict` +| Another workload already owns an object with this pipeline's name, such as a ConfigMap or Deployment. Rename the pipeline or remove the conflicting object. The operator refuses to adopt objects it does not own. +|=== + == Next steps -* xref:connect:install:kubernetes-operator.adoc[] for the full Pipeline guide in the Redpanda Connect docs -* xref:connect:install:kubernetes-operator-autoscale.adoc[] to autoscale pipelines with an HPA or KEDA -* xref:reference:k-crd-index.adoc[] for the Pipeline CRD source definition +* xref:manage:kubernetes/k-autoscale-connect-pipelines.adoc[] +* xref:reference:k-crd.adoc#k8s-api-github-com-redpanda-data-redpanda-operator-operator-api-redpanda-v1alpha2-pipeline[Pipeline CRD reference] for all Pipeline fields +* xref:connect:home:index.adoc[] to explore Redpanda Connect components and configuration * xref:get-started:licensing/overview.adoc[] From 95c962a9533a82f7cd140c6e87786c647af6bbdd Mon Sep 17 00:00:00 2001 From: JakeSCahill Date: Mon, 27 Jul 2026 11:14:24 +0100 Subject: [PATCH 12/14] docs: escape inline interpolation tokens Wrap ${NAME}-style tokens in inline passthroughs so Asciidoctor does not treat them as attribute references and log missing-attribute warnings. --- modules/manage/pages/kubernetes/k-connect-pipelines.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/manage/pages/kubernetes/k-connect-pipelines.adoc b/modules/manage/pages/kubernetes/k-connect-pipelines.adoc index 0286ba58a8..6261c78e60 100644 --- a/modules/manage/pages/kubernetes/k-connect-pipelines.adoc +++ b/modules/manage/pages/kubernetes/k-connect-pipelines.adoc @@ -386,7 +386,7 @@ The Pod assumes only the annotated role, through the projected ServiceAccount to == Inject values and secrets with valueSources -Use `spec.valueSources` to project named values into the pipeline Pod as environment variables, which your configuration references with `${NAME}` interpolation. Each entry is a single, typed pull from one of four sources, and you can mix sources freely in one list: +Use `spec.valueSources` to project named values into the pipeline Pod as environment variables, which your configuration references with `+${NAME}+` interpolation. Each entry is a single, typed pull from one of four sources, and you can mix sources freely in one list: * `inline`: A literal value in the Pipeline spec. * `secretKeyRef`: One key from a Kubernetes Secret. From 46c638a0165535bc55cf39400d805d2663cf05ac Mon Sep 17 00:00:00 2001 From: JakeSCahill Date: Mon, 27 Jul 2026 11:26:31 +0100 Subject: [PATCH 13/14] docs: clarify that the Connect controller is part of the Redpanda Operator Readers who hear 'Redpanda Connect operator' may expect a separate component to install. State up front that the Connect controller is a feature of the existing Redpanda Operator, enabled with one Helm setting. --- modules/manage/pages/kubernetes/k-connect-pipelines.adoc | 2 ++ 1 file changed, 2 insertions(+) diff --git a/modules/manage/pages/kubernetes/k-connect-pipelines.adoc b/modules/manage/pages/kubernetes/k-connect-pipelines.adoc index 6261c78e60..935a727ac9 100644 --- a/modules/manage/pages/kubernetes/k-connect-pipelines.adoc +++ b/modules/manage/pages/kubernetes/k-connect-pipelines.adoc @@ -26,6 +26,8 @@ Your license must include the Redpanda Connect product. The operator treats a li == When to use the Pipeline resource +NOTE: There is no separate operator to install. The Connect controller is part of the Redpanda Operator, and you enable it with a single Helm setting on your existing installation. See <>. + Use the Pipeline resource when you want: * *Declarative, GitOps-friendly management*: Pipelines are plain Kubernetes manifests that you can version, review, and deploy with tools such as Argo CD or Flux. From 0ed5f658ce753caac7fe879fb92845e9b78c123f Mon Sep 17 00:00:00 2001 From: JakeSCahill Date: Tue, 28 Jul 2026 10:16:24 +0100 Subject: [PATCH 14/14] docs: avoid starting sentence with 'There is' per style guide --- modules/manage/pages/kubernetes/k-connect-pipelines.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/manage/pages/kubernetes/k-connect-pipelines.adoc b/modules/manage/pages/kubernetes/k-connect-pipelines.adoc index 935a727ac9..1d802df5f0 100644 --- a/modules/manage/pages/kubernetes/k-connect-pipelines.adoc +++ b/modules/manage/pages/kubernetes/k-connect-pipelines.adoc @@ -26,7 +26,7 @@ Your license must include the Redpanda Connect product. The operator treats a li == When to use the Pipeline resource -NOTE: There is no separate operator to install. The Connect controller is part of the Redpanda Operator, and you enable it with a single Helm setting on your existing installation. See <>. +NOTE: You don't install a separate operator. The Connect controller is part of the Redpanda Operator, and you enable it with a single Helm setting on your existing installation. See <>. Use the Pipeline resource when you want: