docs: surface Redpanda Connect pipelines for operator users (DOC-2275) - #1808
Open
JakeSCahill wants to merge 13 commits into
Open
docs: surface Redpanda Connect pipelines for operator users (DOC-2275)#1808JakeSCahill wants to merge 13 commits into
JakeSCahill wants to merge 13 commits into
Conversation
✅ Deploy Preview for redpanda-docs-preview ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
Contributor
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
Author
|
Update: the Redpanda Connect operator support (Pipeline CRD, redpanda-operator#1677) is expected to slip past 26.2, most likely to 26.3. This PR stays in draft until the CRD ships in a released operator. Do not merge for the 26.2 GA on July 28. |
Contributor
|
PR is merged redpanda-data/redpanda-operator#1677, this will go out as Beta for the 26.2 date. Could you ensure that examples are provided with examples of how to use HPA and KEDA for autoscaling? |
david-yu
marked this pull request as ready for review
July 24, 2026 18:09
david-yu
approved these changes
Jul 24, 2026
david-yu
force-pushed
the
doc-2275-connect-pipelines-discovery
branch
from
July 24, 2026 20:37
44b1fe4 to
ddb3fc5
Compare
This was referenced Jul 27, 2026
Contributor
Author
|
Structure decision reversed after discussion: this component is canonical for the Pipeline how-to. The Pipeline resource is an operator CRD, and every other operator CRD guide (Topic, User, ShadowLink) lives here alongside the generated CRD reference, so the full guide and the autoscaling page are restored on this branch (77db658). rp-connect-docs#457 is now a thin discovery page in the Connect install section that links here, with zero duplicated procedure. The full guide links the Pipeline CRD reference anchor, which lands with #1830. Merge order: #1830 → this PR (the CRD-reference deep link) and #457 in any order after. |
Three discovery surfaces for the 26.2 Pipeline CRD (DOC-2275): - new Manage > Kubernetes page routing to the full guide in the Connect docs, positioned next to Manage Kafka Connect - Pipeline row in the CRD reference index - TIP on Manage Kafka Connect pointing at the first-class alternative
…censing, and what's new (DOC-2275) Turn the discovery page into a full operator-side guide covering every flow from redpanda-operator#1677: enable the Connect controller, first pipeline, clusterRef+userRef cluster binding (with the rendered-config injection contract), staticConfiguration for external clusters, per-pipeline cloud IAM via serviceAccountName (IRSA worked example), and valueSources with all four source types. Adds status/condition reference and troubleshooting. Add a dedicated autoscaling page with step-by-step HPA (cpu/memory) and KEDA (consumer-group lag, Connect metrics via Prometheus, scale-to-zero) examples, all grounded in manifests validated end-to-end on EKS. Document the no-license/expired-license behavior for pipelines in the licensing overview, and add the operator what's-new entry (version section marked TODO until the ship vehicle is confirmed). Route the CRD index row to the local guide instead of the unpublished connect-docs page, which removes the two expected broken-xref warnings. Co-Authored-By: Claude Fable 5 <[email protected]>
The feature ships with operator v26.2.x, not v26.3.x. Drop the provisional v26.3.x section and its TODO. Co-Authored-By: Claude Fable 5 <[email protected]>
…ing overview Give the operator its own enterprise-features table, matching the Redpanda / Console / Connect sections: Stretch Clusters (multicluster operator refuses to start without a valid license; LicenseValid condition + status.licenseStatus on running clusters) and Connect pipelines (moved out of the Connect table). Add a 'Manage licenses for the Redpanda Operator' section covering enterprise.licenseSecretRef and per-cluster licensing via the Redpanda and StretchCluster resources. Co-Authored-By: Claude Fable 5 <[email protected]>
…ect product entitlement Replace the bespoke license sentence on the pipelines page with the shared enterprise-license partial, and state everywhere the license is discussed (notice, prerequisites, license behavior, troubleshooting, licensing overview table) that the license must include the Redpanda Connect product — matching the operator's IncludesProduct(ProductConnect) check. Co-Authored-By: Claude Fable 5 <[email protected]>
Co-Authored-By: Claude Fable 5 <[email protected]>
The Pipeline resource ships as a beta feature in the 26.2 GA operator (redpanda-operator#1677 merged to main on July 22), so remove the draft-hold markers and keep the release-note section under v26.2.x. Also apply standards review fixes: - Rename the page to 'Run Redpanda Connect Pipelines in Kubernetes' (verb phrase for a how-to) - Use the standard beta notice on both pipeline pages, including the autoscaling page, which set page-beta without an in-body disclosure - Mark the feature (beta) at its other discovery surfaces: the CRD index row, the licensing overview row, and the Kafka Connect page - Replace list-bullet double hyphens with colons in the release notes - Split semicolon-joined sentences, fix a passive section opener, avoid starting a sentence with inline code, spell out IAM on first use, and add lead-in sentences before the status command and the troubleshooting table - Cross-link the Redpanda Connect operator install guide from Next steps (resolves when rp-connect-docs#457 merges)
Register the shared badge inline macro in the playbook (already used by cloud-docs; the docs playbook never registered it, which is why no page here could use it) and apply badge::[label=beta] at the places where beta status applies to a feature mention inside a GA page: the CRD index row, the licensing overview row, and the Kafka Connect page cross-link. Whole-page beta status stays on the page-beta attribute.
…ntent The connect repo's Run Pipelines with the Redpanda Operator page (rp-connect-docs#457) is the canonical Pipeline guide. Slim this component's page to a discovery surface: when to use the Pipeline resource, enabling the Connect controller, a first pipeline, and license behavior, with the deep workflows linked in the Redpanda Connect docs. - Move the external-cluster, cloud IAM, license-behavior, and troubleshooting sections to the connect page - Move the autoscaling page to the connect component and remove it here, including its nav entry - Retarget the release-note links
Restore the full guide and the autoscaling page. The Pipeline resource is an operator CRD, and every other operator CRD guide lives here alongside the generated CRD reference, so this component owns the canonical how-to. The Redpanda Connect install section keeps a discovery page that links here (rp-connect-docs#457). Link the Pipeline CRD reference anchor directly (lands with #1830).
Wrap ${NAME}-style tokens in inline passthroughs so Asciidoctor does
not treat them as attribute references and log missing-attribute
warnings.
…rator Readers who hear 'Redpanda Connect operator' may expect a separate component to install. State up front that the Connect controller is a feature of the existing Redpanda Operator, enabled with one Helm setting.
JakeSCahill
force-pushed
the
doc-2275-connect-pipelines-discovery
branch
from
July 27, 2026 12:13
e9ccdb1 to
46c638a
Compare
david-yu
approved these changes
Jul 28, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The Pipeline CRD ships as a beta feature in the 26.2 GA operator (redpanda-operator#1677 merged to main July 22). Merge together with redpanda-data/rp-connect-docs#457: the Next steps cross-link to
connect:install:kubernetes-operator.adocresolves once #457 publishes.Documents the Pipeline resource (redpanda-operator#1677) for operator users. Originally a discovery-only PR; now expanded into the full operator-side guide, per review feedback, covering every flow in the eng PR description with worked, e2e-validated examples.
Pages
manage:kubernetes/k-connect-pipelines.adoc(nav: "Redpanda Connect Pipelines") — full guide:connectController.enabled+enterprise.licenseSecretRef)configYaml), lint-gated startup, config/credential checksum rollouts, pausecluster.clusterRef+userRefwith a scopedUserCR, the rendered-config injection contract (seed_brokers/tls/sasl+ top-level shared client), user-keys-win escape hatch, ACL-scoping guidancecluster.staticConfigurationwith Secret/ConfigMap-backed TLS CA, inline SASL, CEL exclusivity withuserRefserviceAccountName+ IRSA worked example (RDS IAM auth, no DB password anywhere)inline/secretKeyRef/configMapKeyRef/externalSecretRef) in one fan-out exampleextraInitContainers), monitoring (PodMonitor), license behavior, status/conditions reference, troubleshooting. Uses the standardshared:partial$enterprise-license.adocnotice and states that the license must include the Redpanda Connect product (matches the operator'sIncludesProduct(ProductConnect)check).manage:kubernetes/k-autoscale-connect-pipelines.adoc(new, nav: nested "Autoscaling") — step-by-step autoscaling through the scale subresource, grounded in manifests validated e2e on EKS (HPA cpu 1→4; KEDA kafka-lag 1→6; KEDA prometheusinput_received1→3):kubectl scale+ raw/scalesmoke testkubectl get hpaoutput)connectController.monitoring.enabled+ PodMonitor), scale-to-zeropausedprecedence, explicitreplicas: 0, GitOps ownership ofreplicas, one autoscaler per pipelineget-started:licensing/overview.adoc): new "Redpanda Operator enterprise features" section with its own table (matching the Redpanda/Console/Connect sections) covering both operator-gated features — Stretch Clusters (multicluster operator requires a valid license to start;LicenseValidcondition +status.licenseStatusreport expiry/violations on running clusters) and Connect pipelines (missing/expired license or missing Connect product entitlement → new pipelines reportLicense: Falseand get no workload; running pipelines keep processing but stop receiving updates; recovery is automatic once a valid license is applied). Also adds a "Manage licenses for the Redpanda Operator" section (enterprise.licenseSecretRefchart values + per-cluster licensing via the Redpanda/StretchClusterenterprisefield).get-started:release-notes/operator.adoc): "Redpanda Connect pipelines (beta)" entry under thev26.2.xsection.Build note
The two previously-expected broken-xref warnings for
connect:install:kubernetes-operator.adocare gone: all routing now targets the local pages.origin/betahas been merged into the branch.Before un-drafting (GA batch)
Fill in the exact minimum operator version— done: the page states Redpanda Operator v26.2.1 or laterconnect:install:kubernetes-operator.adoc(comment markers in place)Part of DOC-2275; eng feature is redpanda-operator#1677.
Preview pages
🤖 Generated with Claude Code