From 59471ae23fe3a221ddb577df4f9f0b4e1f0a857d Mon Sep 17 00:00:00 2001 From: micheleRP Date: Mon, 27 Jul 2026 16:05:43 -0600 Subject: [PATCH 1/3] Document Connect pipeline egress allowlist for BYOC clusters Add a Networking how-to for the Redpanda Connect egress allowlist (redpanda_connect.allowed_destination_cidr_ports), including the Cloud API flow, which had no coverage, and document the default pipeline egress behavior where readers look for firewall boundaries. - Add networking:byoc/connect-egress-allowlist.adoc, plus a shared partial for the rule fields and limits - Document default-allowed, default-blocked, and never-allowlistable pipeline destinations in cloud-security-network.adoc - Reuse the shared partial in terraform-provider.adoc and correct the minimum provider version to 2.1.2 - Broaden the What's New entry from Terraform-only to the feature DOC-2069 Co-Authored-By: Claude Opus 5 (1M context) --- modules/ROOT/nav.adoc | 1 + .../get-started/pages/whats-new-cloud.adoc | 4 +- modules/manage/pages/terraform-provider.adoc | 8 +- .../pages/byoc/connect-egress-allowlist.adoc | 158 ++++++++++++++++++ .../pages/cloud-security-network.adoc | 8 + .../connect-egress-allowlist-rules.adoc | 23 +++ 6 files changed, 195 insertions(+), 7 deletions(-) create mode 100644 modules/networking/pages/byoc/connect-egress-allowlist.adoc create mode 100644 modules/networking/partials/connect-egress-allowlist-rules.adoc diff --git a/modules/ROOT/nav.adoc b/modules/ROOT/nav.adoc index 2109a99b4..5e529927f 100644 --- a/modules/ROOT/nav.adoc +++ b/modules/ROOT/nav.adoc @@ -477,6 +477,7 @@ **** xref:networking:serverless/aws/privatelink-ui.adoc[Configure PrivateLink in the Cloud Console] **** xref:networking:serverless/aws/privatelink-api.adoc[Configure PrivateLink with the Cloud API] ** xref:networking:byoc/index.adoc[BYOC] +*** xref:networking:byoc/connect-egress-allowlist.adoc[Configure Redpanda Connect Pipeline Egress] *** xref:networking:byoc/aws/index.adoc[AWS] **** xref:networking:byoc/aws/vpc-peering-aws.adoc[Add a Peering Connection] **** xref:networking:configure-privatelink-in-cloud-ui.adoc[Configure PrivateLink in the Cloud Console] diff --git a/modules/get-started/pages/whats-new-cloud.adoc b/modules/get-started/pages/whats-new-cloud.adoc index f94dfbd2c..f1f44c690 100644 --- a/modules/get-started/pages/whats-new-cloud.adoc +++ b/modules/get-started/pages/whats-new-cloud.adoc @@ -8,9 +8,9 @@ This page lists new features added to Redpanda Cloud. == July 2026 -=== Terraform provider: Egress allowlist for Redpanda Connect pipelines +=== Egress allowlist for Redpanda Connect pipelines -The Redpanda Terraform provider (v2.1.1+) now supports allowing Redpanda Connect pipelines on BYOC clusters to open outbound connections to custom destinations, such as a database with a private address in a peered VPC. Use the `redpanda_connect.allowed_destination_cidr_ports` attribute on a `redpanda_cluster` resource to allow egress to up to 16 CIDR and port range combinations. See xref:manage:terraform-provider.adoc#enable-egress-to-custom-destinations[Enable egress to custom destinations]. +You can now allow Redpanda Connect pipelines on BYOC clusters to open outbound connections to destinations that the data plane firewall blocks by default. Add up to 16 CIDR and port range combinations to a cluster with the `redpanda_connect.allowed_destination_cidr_ports` field in the Cloud API, or with the same attribute on a `redpanda_cluster` resource in the Redpanda Terraform provider (v2.1.2+). See xref:networking:byoc/connect-egress-allowlist.adoc[Configure Egress for Redpanda Connect Pipelines]. === New AWS regions for BYOC diff --git a/modules/manage/pages/terraform-provider.adoc b/modules/manage/pages/terraform-provider.adoc index e5ac4a685..fae8a3e3c 100644 --- a/modules/manage/pages/terraform-provider.adoc +++ b/modules/manage/pages/terraform-provider.adoc @@ -383,7 +383,7 @@ For the full schema and import syntax, see the https://registry.terraform.io/pro On BYOC clusters, Redpanda Connect pipelines run inside the Redpanda data plane VPC in your cloud account. By default, pipelines can connect to your Redpanda cluster and to publicly routable endpoints. Outbound connections to other destinations, such as a database with a private address in a peered VPC, are blocked. -To allow pipelines to reach these destinations, add the `redpanda_connect.allowed_destination_cidr_ports` attribute to the `redpanda_cluster` resource (requires provider version `>= 2.1.1`). The attribute accepts up to 16 rules, each allowing egress to a destination CIDR block on a port or port range: +To allow pipelines to reach these destinations, add the `redpanda_connect.allowed_destination_cidr_ports` attribute to the `redpanda_cluster` resource (requires provider version `>= 2.1.2`). The attribute accepts up to 16 rules, each allowing egress to a destination CIDR block on a port or port range: [source,hcl] ---- @@ -406,11 +406,9 @@ resource "redpanda_cluster" "example" { } ---- -* `cidr`: The destination IPv4 CIDR block, for example `10.62.0.0/16`. -* `port_start`: The start of the TCP/UDP port range, 1-65535. -* `port_end` (optional): The end of the TCP/UDP port range. To allow a single port, omit this attribute. When set, the value must be greater than or equal to `port_start`. +include::networking:partial$connect-egress-allowlist-rules.adoc[] -You can add or change the `redpanda_connect` block on an existing cluster. Terraform updates the cluster in place without recreating it. +You can add or change the `redpanda_connect` block on an existing cluster. Terraform updates the cluster in place without recreating it. To set the same allowlist with the Cloud API, and for troubleshooting guidance, see xref:networking:byoc/connect-egress-allowlist.adoc[Configure Egress for Redpanda Connect Pipelines]. The allowlist permits outbound traffic from pipelines, but it does not create a network path to the destination. For private destinations, you must also establish routing, for example by peering the Redpanda data plane VPC with the VPC that hosts your database and adding routes in both directions. diff --git a/modules/networking/pages/byoc/connect-egress-allowlist.adoc b/modules/networking/pages/byoc/connect-egress-allowlist.adoc new file mode 100644 index 000000000..0eb9099bc --- /dev/null +++ b/modules/networking/pages/byoc/connect-egress-allowlist.adoc @@ -0,0 +1,158 @@ += Configure Egress for Redpanda Connect Pipelines +:description: Allow Redpanda Connect pipelines on a BYOC cluster to reach destinations that the data plane firewall blocks by default, such as a VPC endpoint inside the Redpanda VPC. +:page-topic-type: how-to +:personas: platform_admin +:learning-objective-1: Identify which pipeline destinations the data plane firewall blocks by default +:learning-objective-2: Add egress allowlist rules to a cluster using the Cloud API or Terraform +:learning-objective-3: Troubleshoot a pipeline connection that the firewall blocks + +// DRAFT for DOC-2069. Open items flagged with TODO before review. + +Redpanda Connect pipelines on BYOC clusters run behind a data plane firewall that blocks outbound connections to addresses inside the Redpanda VPC. Add an egress allowlist to your cluster to let pipelines reach these destinations, such as a VPC endpoint that fronts a third-party service. + +After reading this page, you will be able to: + +* [ ] {learning-objective-1} +* [ ] {learning-objective-2} +* [ ] {learning-objective-3} + +== Default pipeline egress + +On BYOC clusters, pipelines run as pods inside the Redpanda data plane VPC or VNet in your cloud account. Each pipeline pod enforces a firewall that classifies every outbound connection by destination address and port: + +|=== +| Destination | Default behavior + +| Your Redpanda cluster +| Allowed on the Kafka API, HTTP Proxy, and Schema Registry ports. + +| Public endpoints on the internet +| Allowed. + +| Private addresses outside the data plane VPC or VNet, such as a database in a peered network +| Allowed. + +| Addresses inside the data plane VPC or VNet CIDR, and inside the Kubernetes pod and service CIDRs +| Blocked, except on the ports Redpanda uses to operate the cluster. Add an egress allowlist to open these destinations. + +| Link-local addresses (`169.254.0.0/16`), including the cloud provider instance metadata service +| Blocked. You cannot allow these destinations. + +| The Kubernetes API server, and the kubelet ports 10250 and 10255 on the data plane CIDR +| Blocked. You cannot allow these destinations. +|=== + +On BYOVPC clusters, the data plane VPC is the VPC you provide, so the fourth row covers every address in your own VPC CIDR. + +An allowlist rule grants the pipeline permission to open the connection. It does not create a network path to the destination. If the destination sits outside the data plane VPC, you must also establish routing, for example with xref:networking:byoc/aws/vpc-peering-aws.adoc[VPC peering] or a xref:networking:byoc/aws/transit-gateway.adoc[transit gateway]. + +== Prerequisites + +* A BYOC cluster with Redpanda Connect enabled. +* The destination address, expressed as an IPv4 CIDR block, and the TCP or UDP port or port range it listens on. +* To use the Cloud API: a Cloud API access token, and the ability to run `PATCH` requests against your cluster. +* To use Terraform: the xref:manage:terraform-provider.adoc[Redpanda Terraform provider] version 2.1.2 or later. + +== Add allowlist rules with the Cloud API + +=== Get a Cloud API access token + +include::networking:partial$private-links-api-access-token.adoc[] + +=== Update the cluster + +. In the Redpanda Cloud Console, go to the cluster overview and copy the cluster ID from the *Details* section. ++ +[,bash] +---- +CLUSTER_ID= +---- + +. Make a link:/api/doc/cloud-controlplane/operation/operation-clusterservice_updatecluster[`PATCH /v1/clusters/{cluster.id}`] request with the destinations you want to allow. This example allows pipelines to reach a VPC endpoint on port 443, and a service listening on ports 9000 to 9100: ++ +[,bash] +---- +CLUSTER_PATCH_BODY=`cat << EOF +{ + "redpanda_connect": { + "allowed_destination_cidr_ports": [ + { + "cidr": "10.0.32.0/20", + "port_start": 443 + }, + { + "cidr": "10.0.48.0/20", + "port_start": 9000, + "port_end": 9100 + } + ] + } +} +EOF` + +curl -vv -X PATCH \ + -H "Content-Type: application/json" \ + -H "Authorization: Bearer $AUTH_TOKEN" \ + -d "$CLUSTER_PATCH_BODY" $PUBLIC_API_ENDPOINT/v1/clusters/$CLUSTER_ID +---- + +. Check the state of the Update Cluster operation by calling link:/api/doc/cloud-controlplane/operation/operation-operationservice_getoperation[`GET /v1/operations/\{id}`] with the operation ID returned by the previous call. When the state is `STATE_READY`, the allowlist is applied. + +The request replaces the full allowlist. To add a rule to an existing allowlist, call link:/api/doc/cloud-controlplane/operation/operation-clusterservice_getcluster[`GET /v1/clusters/\{id}`] first, then send the existing rules along with the new one. + +== Add allowlist rules with Terraform + +Add the `redpanda_connect.allowed_destination_cidr_ports` attribute to your `redpanda_cluster` resource. Terraform updates the cluster in place without recreating it: + +[source,hcl] +---- +resource "redpanda_cluster" "example" { + # ... other cluster arguments ... + + redpanda_connect = { + allowed_destination_cidr_ports = [ + { + cidr = "10.0.32.0/20" + port_start = 443 + }, + { + cidr = "10.0.48.0/20" + port_start = 9000 + port_end = 9100 + } + ] + } +} +---- + +For a pipeline example that writes to a private PostgreSQL database, see xref:manage:terraform-provider.adoc#enable-egress-to-custom-destinations[Enable egress to custom destinations]. + +== Allowlist rule reference + +include::networking:partial$connect-egress-allowlist-rules.adoc[] + +For the full list of destinations you cannot allow, see <>. + +== When allowlist changes take effect + +A pipeline pod applies the firewall rules when it starts. After the cluster update completes, pipelines that start or restart use the new allowlist. Restart a running pipeline to apply an allowlist change to it. + +// TODO: confirm with engineering whether the platform restarts existing pipeline pods as part of the cluster update, or whether the user must restart them. + +== Troubleshoot a blocked connection + +If a pipeline reports connection timeouts to a destination you allowed: + +. Confirm the cluster update finished. Call `GET /v1/clusters/\{id}` and check that `redpanda_connect.allowed_destination_cidr_ports` lists your rules. +. Restart the pipeline so it picks up the current rules. See <>. +. Confirm the destination address falls inside the CIDR block you allowed, and that the port matches. A rule for `port_start: 443` does not allow a connection to port 8443. +. Confirm a network path exists to the destination. An allowlist rule does not create routing. For destinations outside the data plane VPC, check your peering connection, transit gateway, or firewall rules. +. Confirm the destination is not one that Redpanda always blocks, such as a link-local address. See <>. + +If the connection still fails, contact https://support.redpanda.com/hc/en-us/requests/new[Redpanda Support^]. + +== Next steps + +* xref:develop:connect/connect-quickstart.adoc[Build a data pipeline] +* xref:networking:cloud-security-network.adoc[Network Design and Ports] +* xref:manage:terraform-provider.adoc[Manage clusters with Terraform] diff --git a/modules/networking/pages/cloud-security-network.adoc b/modules/networking/pages/cloud-security-network.adoc index 146220883..0c8fdb93b 100644 --- a/modules/networking/pages/cloud-security-network.adoc +++ b/modules/networking/pages/cloud-security-network.adoc @@ -167,6 +167,14 @@ When private service connectivity is enabled (AWS PrivateLink, Azure Private Lin |=== +== Redpanda Connect pipeline egress + +On BYOC clusters, Redpanda Connect pipelines run as pods inside the data plane VPC or VNet, and each pod enforces its own outbound firewall. By default, pipelines reach your Redpanda cluster on the Kafka API, HTTP Proxy, and Schema Registry ports, public endpoints on the internet, and private addresses outside the data plane VPC or VNet, such as a database in a peered network. + +Pipelines cannot reach addresses inside the data plane VPC or VNet CIDR, or inside the Kubernetes pod and service CIDRs, except on the ports Redpanda uses to operate the cluster. To open these destinations, add an egress allowlist to your cluster. See xref:networking:byoc/connect-egress-allowlist.adoc[Configure Egress for Redpanda Connect Pipelines]. + +Redpanda always blocks pipeline traffic to link-local addresses (`169.254.0.0/16`), including the cloud provider instance metadata service, to the Kubernetes API server, and to the kubelet ports 10250 and 10255. You cannot add these destinations to an egress allowlist. + == NAT gateways A NAT (Network Address Translation) gateway allows resources in a private network to access the internet, while blocking inbound connections. diff --git a/modules/networking/partials/connect-egress-allowlist-rules.adoc b/modules/networking/partials/connect-egress-allowlist-rules.adoc new file mode 100644 index 000000000..4d91680f1 --- /dev/null +++ b/modules/networking/partials/connect-egress-allowlist-rules.adoc @@ -0,0 +1,23 @@ +Each rule allows outbound traffic to one CIDR block on one port or port range: + +|=== +| Field | Required | Description + +| `cidr` +| Yes +| Destination IPv4 CIDR block, for example `10.0.32.0/20`. IPv6 is not supported. + +| `port_start` +| Yes +| First TCP or UDP port to allow, from 1 to 65535. + +| `port_end` +| No +| Last TCP or UDP port to allow. Omit this field to allow a single port. When set, the value must be greater than or equal to `port_start`. +|=== + +Limits: + +* A cluster accepts a maximum of 16 rules. +* Each combination of `cidr`, `port_start`, and `port_end` must be unique. +* Rules cannot override the destinations that Redpanda always blocks, such as link-local addresses and the Kubernetes API server. From f4edceacce76ae7937eca28f255fd387c42f3f9e Mon Sep 17 00:00:00 2001 From: micheleRP Date: Mon, 27 Jul 2026 17:07:47 -0600 Subject: [PATCH 2/3] Apply review fixes to Connect egress allowlist docs - Remove the leftover draft marker from the new page - Give in-VPC DNS resolution its own row, and state it consistently in cloud-security-network.adoc - Replace the brittle "fourth row" reference with the behavior it describes - Trim the Terraform lead-in that duplicated the shared partial, and point the partial's always-blocked bullet at Default pipeline egress - Use glossterm:data plane[] on first mention, and settle on "data plane VPC" - Name the required Cloud API service account instead of "ability to run PATCH requests"; document the port_end upper bound - Drop curl -vv so the bearer token is not echoed into logs (CodeRabbit) - Link the Connect quickstart to the new page for the Cloud API path DOC-2069 Co-Authored-By: Claude Opus 5 (1M context) --- .../pages/connect/connect-quickstart.adoc | 2 +- modules/manage/pages/terraform-provider.adoc | 2 +- .../pages/byoc/connect-egress-allowlist.adoc | 21 +++++++++---------- .../pages/cloud-security-network.adoc | 2 +- .../connect-egress-allowlist-rules.adoc | 6 +++--- 5 files changed, 16 insertions(+), 17 deletions(-) diff --git a/modules/develop/pages/connect/connect-quickstart.adoc b/modules/develop/pages/connect/connect-quickstart.adoc index f463f4c92..756b3c674 100644 --- a/modules/develop/pages/connect/connect-quickstart.adoc +++ b/modules/develop/pages/connect/connect-quickstart.adoc @@ -380,7 +380,7 @@ resource "redpanda_cluster" "example" { The allowlist permits outbound traffic from pipelines, but it does not create a network path to the destination. For private destinations, you must also establish routing, for example by peering the Redpanda data plane VPC with the VPC that hosts your database. -For details, including a pipeline example that writes to a private PostgreSQL database, see xref:manage:terraform-provider.adoc#enable-egress-to-custom-destinations[Enable egress to custom destinations]. +To set the allowlist with the Cloud API, and for the full rule reference and troubleshooting guidance, see xref:networking:byoc/connect-egress-allowlist.adoc[Configure Egress for Redpanda Connect Pipelines]. For a pipeline example that writes to a private PostgreSQL database, see xref:manage:terraform-provider.adoc#enable-egress-to-custom-destinations[Enable egress to custom destinations]. == Clean up diff --git a/modules/manage/pages/terraform-provider.adoc b/modules/manage/pages/terraform-provider.adoc index fae8a3e3c..099f2a229 100644 --- a/modules/manage/pages/terraform-provider.adoc +++ b/modules/manage/pages/terraform-provider.adoc @@ -383,7 +383,7 @@ For the full schema and import syntax, see the https://registry.terraform.io/pro On BYOC clusters, Redpanda Connect pipelines run inside the Redpanda data plane VPC in your cloud account. By default, pipelines can connect to your Redpanda cluster and to publicly routable endpoints. Outbound connections to other destinations, such as a database with a private address in a peered VPC, are blocked. -To allow pipelines to reach these destinations, add the `redpanda_connect.allowed_destination_cidr_ports` attribute to the `redpanda_cluster` resource (requires provider version `>= 2.1.2`). The attribute accepts up to 16 rules, each allowing egress to a destination CIDR block on a port or port range: +To allow pipelines to reach these destinations, add the `redpanda_connect.allowed_destination_cidr_ports` attribute to the `redpanda_cluster` resource (requires provider version `>= 2.1.2`): [source,hcl] ---- diff --git a/modules/networking/pages/byoc/connect-egress-allowlist.adoc b/modules/networking/pages/byoc/connect-egress-allowlist.adoc index 0eb9099bc..e1b81778a 100644 --- a/modules/networking/pages/byoc/connect-egress-allowlist.adoc +++ b/modules/networking/pages/byoc/connect-egress-allowlist.adoc @@ -1,14 +1,12 @@ = Configure Egress for Redpanda Connect Pipelines -:description: Allow Redpanda Connect pipelines on a BYOC cluster to reach destinations that the data plane firewall blocks by default, such as a VPC endpoint inside the Redpanda VPC. +:description: Allow Redpanda Connect pipelines on a BYOC cluster to reach destinations that the data plane firewall blocks by default, such as a VPC endpoint inside the data plane VPC. :page-topic-type: how-to :personas: platform_admin :learning-objective-1: Identify which pipeline destinations the data plane firewall blocks by default :learning-objective-2: Add egress allowlist rules to a cluster using the Cloud API or Terraform :learning-objective-3: Troubleshoot a pipeline connection that the firewall blocks -// DRAFT for DOC-2069. Open items flagged with TODO before review. - -Redpanda Connect pipelines on BYOC clusters run behind a data plane firewall that blocks outbound connections to addresses inside the Redpanda VPC. Add an egress allowlist to your cluster to let pipelines reach these destinations, such as a VPC endpoint that fronts a third-party service. +Redpanda Connect pipelines on BYOC clusters run behind a glossterm:data plane[] firewall that blocks outbound connections to addresses inside the data plane VPC. Add an egress allowlist to your cluster to let pipelines reach these destinations, such as a VPC endpoint that fronts a third-party service. After reading this page, you will be able to: @@ -18,7 +16,7 @@ After reading this page, you will be able to: == Default pipeline egress -On BYOC clusters, pipelines run as pods inside the Redpanda data plane VPC or VNet in your cloud account. Each pipeline pod enforces a firewall that classifies every outbound connection by destination address and port: +On BYOC clusters, pipelines run as pods inside the data plane VPC or VNet in your cloud account. Each pipeline pod enforces a firewall that classifies every outbound connection by destination address and port: |=== | Destination | Default behavior @@ -26,6 +24,9 @@ On BYOC clusters, pipelines run as pods inside the Redpanda data plane VPC or VN | Your Redpanda cluster | Allowed on the Kafka API, HTTP Proxy, and Schema Registry ports. +| DNS resolvers inside the data plane VPC or VNet +| Allowed on port 53. + | Public endpoints on the internet | Allowed. @@ -42,7 +43,7 @@ On BYOC clusters, pipelines run as pods inside the Redpanda data plane VPC or VN | Blocked. You cannot allow these destinations. |=== -On BYOVPC clusters, the data plane VPC is the VPC you provide, so the fourth row covers every address in your own VPC CIDR. +On BYOVPC clusters, the data plane VPC is the VPC you provide, so addresses anywhere in your own VPC CIDR are blocked except on the ports Redpanda uses to operate the cluster. An allowlist rule grants the pipeline permission to open the connection. It does not create a network path to the destination. If the destination sits outside the data plane VPC, you must also establish routing, for example with xref:networking:byoc/aws/vpc-peering-aws.adoc[VPC peering] or a xref:networking:byoc/aws/transit-gateway.adoc[transit gateway]. @@ -50,7 +51,7 @@ An allowlist rule grants the pipeline permission to open the connection. It does * A BYOC cluster with Redpanda Connect enabled. * The destination address, expressed as an IPv4 CIDR block, and the TCP or UDP port or port range it listens on. -* To use the Cloud API: a Cloud API access token, and the ability to run `PATCH` requests against your cluster. +* To use the Cloud API: a Cloud API service account with permission to update the cluster, and an access token. See <>. * To use Terraform: the xref:manage:terraform-provider.adoc[Redpanda Terraform provider] version 2.1.2 or later. == Add allowlist rules with the Cloud API @@ -65,7 +66,7 @@ include::networking:partial$private-links-api-access-token.adoc[] + [,bash] ---- -CLUSTER_ID= +CLUSTER_ID= ---- . Make a link:/api/doc/cloud-controlplane/operation/operation-clusterservice_updatecluster[`PATCH /v1/clusters/{cluster.id}`] request with the destinations you want to allow. This example allows pipelines to reach a VPC endpoint on port 443, and a service listening on ports 9000 to 9100: @@ -90,7 +91,7 @@ CLUSTER_PATCH_BODY=`cat << EOF } EOF` -curl -vv -X PATCH \ +curl -sS -X PATCH \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $AUTH_TOKEN" \ -d "$CLUSTER_PATCH_BODY" $PUBLIC_API_ENDPOINT/v1/clusters/$CLUSTER_ID @@ -131,8 +132,6 @@ For a pipeline example that writes to a private PostgreSQL database, see xref:ma include::networking:partial$connect-egress-allowlist-rules.adoc[] -For the full list of destinations you cannot allow, see <>. - == When allowlist changes take effect A pipeline pod applies the firewall rules when it starts. After the cluster update completes, pipelines that start or restart use the new allowlist. Restart a running pipeline to apply an allowlist change to it. diff --git a/modules/networking/pages/cloud-security-network.adoc b/modules/networking/pages/cloud-security-network.adoc index 0c8fdb93b..967e67ad2 100644 --- a/modules/networking/pages/cloud-security-network.adoc +++ b/modules/networking/pages/cloud-security-network.adoc @@ -169,7 +169,7 @@ When private service connectivity is enabled (AWS PrivateLink, Azure Private Lin == Redpanda Connect pipeline egress -On BYOC clusters, Redpanda Connect pipelines run as pods inside the data plane VPC or VNet, and each pod enforces its own outbound firewall. By default, pipelines reach your Redpanda cluster on the Kafka API, HTTP Proxy, and Schema Registry ports, public endpoints on the internet, and private addresses outside the data plane VPC or VNet, such as a database in a peered network. +On BYOC clusters, Redpanda Connect pipelines run as pods inside the data plane VPC or VNet, and each pod enforces its own outbound firewall. By default, pipelines resolve DNS inside the data plane VPC or VNet, reach your Redpanda cluster on the Kafka API, HTTP Proxy, and Schema Registry ports, connect to public endpoints on the internet, and reach private addresses outside the data plane VPC or VNet, such as a database in a peered network. Pipelines cannot reach addresses inside the data plane VPC or VNet CIDR, or inside the Kubernetes pod and service CIDRs, except on the ports Redpanda uses to operate the cluster. To open these destinations, add an egress allowlist to your cluster. See xref:networking:byoc/connect-egress-allowlist.adoc[Configure Egress for Redpanda Connect Pipelines]. diff --git a/modules/networking/partials/connect-egress-allowlist-rules.adoc b/modules/networking/partials/connect-egress-allowlist-rules.adoc index 4d91680f1..e8bb381ff 100644 --- a/modules/networking/partials/connect-egress-allowlist-rules.adoc +++ b/modules/networking/partials/connect-egress-allowlist-rules.adoc @@ -1,7 +1,7 @@ Each rule allows outbound traffic to one CIDR block on one port or port range: |=== -| Field | Required | Description +| Field or attribute | Required | Description | `cidr` | Yes @@ -13,11 +13,11 @@ Each rule allows outbound traffic to one CIDR block on one port or port range: | `port_end` | No -| Last TCP or UDP port to allow. Omit this field to allow a single port. When set, the value must be greater than or equal to `port_start`. +| Last TCP or UDP port to allow, up to 65535. Omit this field to allow a single port. When set, the value must be greater than or equal to `port_start`. |=== Limits: * A cluster accepts a maximum of 16 rules. * Each combination of `cidr`, `port_start`, and `port_end` must be unique. -* Rules cannot override the destinations that Redpanda always blocks, such as link-local addresses and the Kubernetes API server. +* Rules cannot override the destinations that Redpanda always blocks, such as link-local addresses and the Kubernetes API server. See xref:networking:byoc/connect-egress-allowlist.adoc#default-pipeline-egress[Default pipeline egress]. From 067b19c3543d294cae3fd9a66a5335ca33df36da Mon Sep 17 00:00:00 2001 From: micheleRP Date: Tue, 28 Jul 2026 16:07:41 -0600 Subject: [PATCH 3/3] Correct default egress behavior per engineering confirmation Paul Zhang confirmed on #650 that peered destinations in 10.0.0.0/8 and 172.16.0.0/12 are permitted only on the ports Redpanda uses to operate the cluster, and that the platform restarts existing pipeline pods when the allowlist changes. An earlier draft inferred the opposite from the sidecar rule order. - State that private destinations are blocked on most ports, whether they sit inside the data plane VPC or in a peered network, and that the permitted ranges can change - State that Redpanda restarts pipelines to apply an allowlist change, and remove the TODO that asked - Raise the provider floor in the Connect quickstart to >= 2.1.2, which the Terraform page already required DOC-2069 Co-Authored-By: Claude Opus 5 (1M context) --- .../pages/connect/connect-quickstart.adoc | 2 +- .../pages/byoc/connect-egress-allowlist.adoc | 19 +++++++------------ .../pages/cloud-security-network.adoc | 4 ++-- 3 files changed, 10 insertions(+), 15 deletions(-) diff --git a/modules/develop/pages/connect/connect-quickstart.adoc b/modules/develop/pages/connect/connect-quickstart.adoc index 756b3c674..774008e32 100644 --- a/modules/develop/pages/connect/connect-quickstart.adoc +++ b/modules/develop/pages/connect/connect-quickstart.adoc @@ -360,7 +360,7 @@ The consumer pipeline in this quickstart drops messages instead of writing them On BYOC clusters, pipelines run inside the Redpanda data plane VPC in your cloud account. By default, pipelines can connect to your Redpanda cluster and to publicly routable endpoints. Outbound connections to other destinations, such as a database with a private address in a peered VPC, are blocked. -To allow pipelines to reach these destinations, add an egress allowlist to your cluster using the xref:manage:terraform-provider.adoc[Redpanda Terraform provider] (version `>= 2.1.1`). The `redpanda_connect.allowed_destination_cidr_ports` attribute on the `redpanda_cluster` resource accepts up to 16 rules, each allowing egress to a destination CIDR block on a port or port range: +To allow pipelines to reach these destinations, add an egress allowlist to your cluster using the xref:manage:terraform-provider.adoc[Redpanda Terraform provider] (version `>= 2.1.2`). The `redpanda_connect.allowed_destination_cidr_ports` attribute on the `redpanda_cluster` resource accepts up to 16 rules, each allowing egress to a destination CIDR block on a port or port range: [source,hcl] ---- diff --git a/modules/networking/pages/byoc/connect-egress-allowlist.adoc b/modules/networking/pages/byoc/connect-egress-allowlist.adoc index e1b81778a..1e67f6408 100644 --- a/modules/networking/pages/byoc/connect-egress-allowlist.adoc +++ b/modules/networking/pages/byoc/connect-egress-allowlist.adoc @@ -1,12 +1,12 @@ = Configure Egress for Redpanda Connect Pipelines -:description: Allow Redpanda Connect pipelines on a BYOC cluster to reach destinations that the data plane firewall blocks by default, such as a VPC endpoint inside the data plane VPC. +:description: Allow Redpanda Connect pipelines on a BYOC cluster to reach private destinations that the data plane firewall blocks by default, such as a database in a peered network or a VPC endpoint inside the data plane VPC. :page-topic-type: how-to :personas: platform_admin :learning-objective-1: Identify which pipeline destinations the data plane firewall blocks by default :learning-objective-2: Add egress allowlist rules to a cluster using the Cloud API or Terraform :learning-objective-3: Troubleshoot a pipeline connection that the firewall blocks -Redpanda Connect pipelines on BYOC clusters run behind a glossterm:data plane[] firewall that blocks outbound connections to addresses inside the data plane VPC. Add an egress allowlist to your cluster to let pipelines reach these destinations, such as a VPC endpoint that fronts a third-party service. +Redpanda Connect pipelines on BYOC clusters run behind a glossterm:data plane[] firewall that blocks outbound connections to most private addresses. Add an egress allowlist to your cluster to let pipelines reach a private destination, such as a database in a peered network or a VPC endpoint that fronts a third-party service. After reading this page, you will be able to: @@ -30,11 +30,8 @@ On BYOC clusters, pipelines run as pods inside the data plane VPC or VNet in you | Public endpoints on the internet | Allowed. -| Private addresses outside the data plane VPC or VNet, such as a database in a peered network -| Allowed. - -| Addresses inside the data plane VPC or VNet CIDR, and inside the Kubernetes pod and service CIDRs -| Blocked, except on the ports Redpanda uses to operate the cluster. Add an egress allowlist to open these destinations. +| Private addresses, whether inside the data plane VPC or VNet or in a peered network +| Blocked on most ports. Redpanda permits only the ports it uses to operate the cluster, and the ranges it permits can change. Add an egress allowlist entry for any private destination your pipelines need. | Link-local addresses (`169.254.0.0/16`), including the cloud provider instance metadata service | Blocked. You cannot allow these destinations. @@ -43,7 +40,7 @@ On BYOC clusters, pipelines run as pods inside the data plane VPC or VNet in you | Blocked. You cannot allow these destinations. |=== -On BYOVPC clusters, the data plane VPC is the VPC you provide, so addresses anywhere in your own VPC CIDR are blocked except on the ports Redpanda uses to operate the cluster. +This applies to your own networks. A database in a VPC you peer to the cluster, or anywhere in your own VPC CIDR on a BYOVPC cluster, needs an allowlist entry before pipelines can reach it. An allowlist rule grants the pipeline permission to open the connection. It does not create a network path to the destination. If the destination sits outside the data plane VPC, you must also establish routing, for example with xref:networking:byoc/aws/vpc-peering-aws.adoc[VPC peering] or a xref:networking:byoc/aws/transit-gateway.adoc[transit gateway]. @@ -134,16 +131,14 @@ include::networking:partial$connect-egress-allowlist-rules.adoc[] == When allowlist changes take effect -A pipeline pod applies the firewall rules when it starts. After the cluster update completes, pipelines that start or restart use the new allowlist. Restart a running pipeline to apply an allowlist change to it. - -// TODO: confirm with engineering whether the platform restarts existing pipeline pods as part of the cluster update, or whether the user must restart them. +A pipeline pod applies the firewall rules when it starts. When you change the allowlist, Redpanda restarts your existing pipelines so they pick up the new rules, so each pipeline stops processing briefly while it restarts. You do not need to restart pipelines yourself. == Troubleshoot a blocked connection If a pipeline reports connection timeouts to a destination you allowed: . Confirm the cluster update finished. Call `GET /v1/clusters/\{id}` and check that `redpanda_connect.allowed_destination_cidr_ports` lists your rules. -. Restart the pipeline so it picks up the current rules. See <>. +. Confirm the pipeline restarted after the cluster update, because a pipeline picks up new rules only when it starts. See <>. . Confirm the destination address falls inside the CIDR block you allowed, and that the port matches. A rule for `port_start: 443` does not allow a connection to port 8443. . Confirm a network path exists to the destination. An allowlist rule does not create routing. For destinations outside the data plane VPC, check your peering connection, transit gateway, or firewall rules. . Confirm the destination is not one that Redpanda always blocks, such as a link-local address. See <>. diff --git a/modules/networking/pages/cloud-security-network.adoc b/modules/networking/pages/cloud-security-network.adoc index 967e67ad2..5b94492b7 100644 --- a/modules/networking/pages/cloud-security-network.adoc +++ b/modules/networking/pages/cloud-security-network.adoc @@ -169,9 +169,9 @@ When private service connectivity is enabled (AWS PrivateLink, Azure Private Lin == Redpanda Connect pipeline egress -On BYOC clusters, Redpanda Connect pipelines run as pods inside the data plane VPC or VNet, and each pod enforces its own outbound firewall. By default, pipelines resolve DNS inside the data plane VPC or VNet, reach your Redpanda cluster on the Kafka API, HTTP Proxy, and Schema Registry ports, connect to public endpoints on the internet, and reach private addresses outside the data plane VPC or VNet, such as a database in a peered network. +On BYOC clusters, Redpanda Connect pipelines run as pods inside the data plane VPC or VNet, and each pod enforces its own outbound firewall. By default, pipelines resolve DNS inside the data plane VPC or VNet, reach your Redpanda cluster on the Kafka API, HTTP Proxy, and Schema Registry ports, and connect to public endpoints on the internet. -Pipelines cannot reach addresses inside the data plane VPC or VNet CIDR, or inside the Kubernetes pod and service CIDRs, except on the ports Redpanda uses to operate the cluster. To open these destinations, add an egress allowlist to your cluster. See xref:networking:byoc/connect-egress-allowlist.adoc[Configure Egress for Redpanda Connect Pipelines]. +Connections to private addresses are blocked on most ports, whether the destination sits inside the data plane VPC or VNet or in a network you peer to the cluster. To open a private destination, add an egress allowlist to your cluster. See xref:networking:byoc/connect-egress-allowlist.adoc[Configure Egress for Redpanda Connect Pipelines]. Redpanda always blocks pipeline traffic to link-local addresses (`169.254.0.0/16`), including the cloud provider instance metadata service, to the Kubernetes API server, and to the kubelet ports 10250 and 10255. You cannot add these destinations to an egress allowlist.