Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions modules/ROOT/nav.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -477,6 +477,7 @@
**** xref:networking:serverless/aws/privatelink-ui.adoc[Configure PrivateLink in the Cloud Console]
**** xref:networking:serverless/aws/privatelink-api.adoc[Configure PrivateLink with the Cloud API]
** xref:networking:byoc/index.adoc[BYOC]
*** xref:networking:byoc/connect-egress-allowlist.adoc[Configure Redpanda Connect Pipeline Egress]
*** xref:networking:byoc/aws/index.adoc[AWS]
**** xref:networking:byoc/aws/vpc-peering-aws.adoc[Add a Peering Connection]
**** xref:networking:configure-privatelink-in-cloud-ui.adoc[Configure PrivateLink in the Cloud Console]
Expand Down
4 changes: 2 additions & 2 deletions modules/develop/pages/connect/connect-quickstart.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -360,7 +360,7 @@ The consumer pipeline in this quickstart drops messages instead of writing them

On BYOC clusters, pipelines run inside the Redpanda data plane VPC in your cloud account. By default, pipelines can connect to your Redpanda cluster and to publicly routable endpoints. Outbound connections to other destinations, such as a database with a private address in a peered VPC, are blocked.

To allow pipelines to reach these destinations, add an egress allowlist to your cluster using the xref:manage:terraform-provider.adoc[Redpanda Terraform provider] (version `>= 2.1.1`). The `redpanda_connect.allowed_destination_cidr_ports` attribute on the `redpanda_cluster` resource accepts up to 16 rules, each allowing egress to a destination CIDR block on a port or port range:
To allow pipelines to reach these destinations, add an egress allowlist to your cluster using the xref:manage:terraform-provider.adoc[Redpanda Terraform provider] (version `>= 2.1.2`). The `redpanda_connect.allowed_destination_cidr_ports` attribute on the `redpanda_cluster` resource accepts up to 16 rules, each allowing egress to a destination CIDR block on a port or port range:

[source,hcl]
----
Expand All @@ -380,7 +380,7 @@ resource "redpanda_cluster" "example" {

The allowlist permits outbound traffic from pipelines, but it does not create a network path to the destination. For private destinations, you must also establish routing, for example by peering the Redpanda data plane VPC with the VPC that hosts your database.

For details, including a pipeline example that writes to a private PostgreSQL database, see xref:manage:terraform-provider.adoc#enable-egress-to-custom-destinations[Enable egress to custom destinations].
To set the allowlist with the Cloud API, and for the full rule reference and troubleshooting guidance, see xref:networking:byoc/connect-egress-allowlist.adoc[Configure Egress for Redpanda Connect Pipelines]. For a pipeline example that writes to a private PostgreSQL database, see xref:manage:terraform-provider.adoc#enable-egress-to-custom-destinations[Enable egress to custom destinations].

== Clean up

Expand Down
4 changes: 2 additions & 2 deletions modules/get-started/pages/whats-new-cloud.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,9 @@ This page lists new features added to Redpanda Cloud.

== July 2026

=== Terraform provider: Egress allowlist for Redpanda Connect pipelines
=== Egress allowlist for Redpanda Connect pipelines

The Redpanda Terraform provider (v2.1.1+) now supports allowing Redpanda Connect pipelines on BYOC clusters to open outbound connections to custom destinations, such as a database with a private address in a peered VPC. Use the `redpanda_connect.allowed_destination_cidr_ports` attribute on a `redpanda_cluster` resource to allow egress to up to 16 CIDR and port range combinations. See xref:manage:terraform-provider.adoc#enable-egress-to-custom-destinations[Enable egress to custom destinations].
You can now allow Redpanda Connect pipelines on BYOC clusters to open outbound connections to destinations that the data plane firewall blocks by default. Add up to 16 CIDR and port range combinations to a cluster with the `redpanda_connect.allowed_destination_cidr_ports` field in the Cloud API, or with the same attribute on a `redpanda_cluster` resource in the Redpanda Terraform provider (v2.1.2+). See xref:networking:byoc/connect-egress-allowlist.adoc[Configure Egress for Redpanda Connect Pipelines].

=== New AWS regions for BYOC

Expand Down
8 changes: 3 additions & 5 deletions modules/manage/pages/terraform-provider.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -383,7 +383,7 @@ For the full schema and import syntax, see the https://registry.terraform.io/pro

On BYOC clusters, Redpanda Connect pipelines run inside the Redpanda data plane VPC in your cloud account. By default, pipelines can connect to your Redpanda cluster and to publicly routable endpoints. Outbound connections to other destinations, such as a database with a private address in a peered VPC, are blocked.

To allow pipelines to reach these destinations, add the `redpanda_connect.allowed_destination_cidr_ports` attribute to the `redpanda_cluster` resource (requires provider version `>= 2.1.1`). The attribute accepts up to 16 rules, each allowing egress to a destination CIDR block on a port or port range:
To allow pipelines to reach these destinations, add the `redpanda_connect.allowed_destination_cidr_ports` attribute to the `redpanda_cluster` resource (requires provider version `>= 2.1.2`):

[source,hcl]
----
Expand All @@ -406,11 +406,9 @@ resource "redpanda_cluster" "example" {
}
----

* `cidr`: The destination IPv4 CIDR block, for example `10.62.0.0/16`.
* `port_start`: The start of the TCP/UDP port range, 1-65535.
* `port_end` (optional): The end of the TCP/UDP port range. To allow a single port, omit this attribute. When set, the value must be greater than or equal to `port_start`.
include::networking:partial$connect-egress-allowlist-rules.adoc[]

You can add or change the `redpanda_connect` block on an existing cluster. Terraform updates the cluster in place without recreating it.
You can add or change the `redpanda_connect` block on an existing cluster. Terraform updates the cluster in place without recreating it. To set the same allowlist with the Cloud API, and for troubleshooting guidance, see xref:networking:byoc/connect-egress-allowlist.adoc[Configure Egress for Redpanda Connect Pipelines].

The allowlist permits outbound traffic from pipelines, but it does not create a network path to the destination. For private destinations, you must also establish routing, for example by peering the Redpanda data plane VPC with the VPC that hosts your database and adding routes in both directions.

Expand Down
152 changes: 152 additions & 0 deletions modules/networking/pages/byoc/connect-egress-allowlist.adoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,152 @@
= Configure Egress for Redpanda Connect Pipelines
:description: Allow Redpanda Connect pipelines on a BYOC cluster to reach private destinations that the data plane firewall blocks by default, such as a database in a peered network or a VPC endpoint inside the data plane VPC.
:page-topic-type: how-to
:personas: platform_admin
:learning-objective-1: Identify which pipeline destinations the data plane firewall blocks by default
:learning-objective-2: Add egress allowlist rules to a cluster using the Cloud API or Terraform
:learning-objective-3: Troubleshoot a pipeline connection that the firewall blocks

Redpanda Connect pipelines on BYOC clusters run behind a glossterm:data plane[] firewall that blocks outbound connections to most private addresses. Add an egress allowlist to your cluster to let pipelines reach a private destination, such as a database in a peered network or a VPC endpoint that fronts a third-party service.

After reading this page, you will be able to:

* [ ] {learning-objective-1}
* [ ] {learning-objective-2}
* [ ] {learning-objective-3}

== Default pipeline egress

On BYOC clusters, pipelines run as pods inside the data plane VPC or VNet in your cloud account. Each pipeline pod enforces a firewall that classifies every outbound connection by destination address and port:

|===
| Destination | Default behavior

| Your Redpanda cluster
| Allowed on the Kafka API, HTTP Proxy, and Schema Registry ports.

| DNS resolvers inside the data plane VPC or VNet
| Allowed on port 53.

| Public endpoints on the internet
| Allowed.

| Private addresses, whether inside the data plane VPC or VNet or in a peered network
| Blocked on most ports. Redpanda permits only the ports it uses to operate the cluster, and the ranges it permits can change. Add an egress allowlist entry for any private destination your pipelines need.

| Link-local addresses (`169.254.0.0/16`), including the cloud provider instance metadata service
| Blocked. You cannot allow these destinations.

| The Kubernetes API server, and the kubelet ports 10250 and 10255 on the data plane CIDR
| Blocked. You cannot allow these destinations.
|===

This applies to your own networks. A database in a VPC you peer to the cluster, or anywhere in your own VPC CIDR on a BYOVPC cluster, needs an allowlist entry before pipelines can reach it.

An allowlist rule grants the pipeline permission to open the connection. It does not create a network path to the destination. If the destination sits outside the data plane VPC, you must also establish routing, for example with xref:networking:byoc/aws/vpc-peering-aws.adoc[VPC peering] or a xref:networking:byoc/aws/transit-gateway.adoc[transit gateway].

== Prerequisites

* A BYOC cluster with Redpanda Connect enabled.
* The destination address, expressed as an IPv4 CIDR block, and the TCP or UDP port or port range it listens on.
* To use the Cloud API: a Cloud API service account with permission to update the cluster, and an access token. See <<get-a-cloud-api-access-token>>.
* To use Terraform: the xref:manage:terraform-provider.adoc[Redpanda Terraform provider] version 2.1.2 or later.

== Add allowlist rules with the Cloud API

=== Get a Cloud API access token

include::networking:partial$private-links-api-access-token.adoc[]

=== Update the cluster

. In the Redpanda Cloud Console, go to the cluster overview and copy the cluster ID from the *Details* section.
+
[,bash]
----
CLUSTER_ID=<cluster_id>
----

. Make a link:/api/doc/cloud-controlplane/operation/operation-clusterservice_updatecluster[`PATCH /v1/clusters/{cluster.id}`] request with the destinations you want to allow. This example allows pipelines to reach a VPC endpoint on port 443, and a service listening on ports 9000 to 9100:
+
[,bash]
----
CLUSTER_PATCH_BODY=`cat << EOF
{
"redpanda_connect": {
"allowed_destination_cidr_ports": [
{
"cidr": "10.0.32.0/20",
"port_start": 443
},
{
"cidr": "10.0.48.0/20",
"port_start": 9000,
"port_end": 9100
}
]
}
}
EOF`

curl -sS -X PATCH \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $AUTH_TOKEN" \
-d "$CLUSTER_PATCH_BODY" $PUBLIC_API_ENDPOINT/v1/clusters/$CLUSTER_ID
----

. Check the state of the Update Cluster operation by calling link:/api/doc/cloud-controlplane/operation/operation-operationservice_getoperation[`GET /v1/operations/\{id}`] with the operation ID returned by the previous call. When the state is `STATE_READY`, the allowlist is applied.

The request replaces the full allowlist. To add a rule to an existing allowlist, call link:/api/doc/cloud-controlplane/operation/operation-clusterservice_getcluster[`GET /v1/clusters/\{id}`] first, then send the existing rules along with the new one.

== Add allowlist rules with Terraform

Add the `redpanda_connect.allowed_destination_cidr_ports` attribute to your `redpanda_cluster` resource. Terraform updates the cluster in place without recreating it:

[source,hcl]
----
resource "redpanda_cluster" "example" {
# ... other cluster arguments ...

redpanda_connect = {
allowed_destination_cidr_ports = [
{
cidr = "10.0.32.0/20"
port_start = 443
},
{
cidr = "10.0.48.0/20"
port_start = 9000
port_end = 9100
}
]
}
}
----

For a pipeline example that writes to a private PostgreSQL database, see xref:manage:terraform-provider.adoc#enable-egress-to-custom-destinations[Enable egress to custom destinations].

== Allowlist rule reference

include::networking:partial$connect-egress-allowlist-rules.adoc[]

== When allowlist changes take effect

A pipeline pod applies the firewall rules when it starts. When you change the allowlist, Redpanda restarts your existing pipelines so they pick up the new rules, so each pipeline stops processing briefly while it restarts. You do not need to restart pipelines yourself.

== Troubleshoot a blocked connection

If a pipeline reports connection timeouts to a destination you allowed:

. Confirm the cluster update finished. Call `GET /v1/clusters/\{id}` and check that `redpanda_connect.allowed_destination_cidr_ports` lists your rules.
. Confirm the pipeline restarted after the cluster update, because a pipeline picks up new rules only when it starts. See <<when-allowlist-changes-take-effect>>.
. Confirm the destination address falls inside the CIDR block you allowed, and that the port matches. A rule for `port_start: 443` does not allow a connection to port 8443.
. Confirm a network path exists to the destination. An allowlist rule does not create routing. For destinations outside the data plane VPC, check your peering connection, transit gateway, or firewall rules.
. Confirm the destination is not one that Redpanda always blocks, such as a link-local address. See <<default-pipeline-egress>>.

If the connection still fails, contact https://support.redpanda.com/hc/en-us/requests/new[Redpanda Support^].

== Next steps

* xref:develop:connect/connect-quickstart.adoc[Build a data pipeline]
* xref:networking:cloud-security-network.adoc[Network Design and Ports]
* xref:manage:terraform-provider.adoc[Manage clusters with Terraform]
8 changes: 8 additions & 0 deletions modules/networking/pages/cloud-security-network.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -167,6 +167,14 @@ When private service connectivity is enabled (AWS PrivateLink, Azure Private Lin
|===


== Redpanda Connect pipeline egress

On BYOC clusters, Redpanda Connect pipelines run as pods inside the data plane VPC or VNet, and each pod enforces its own outbound firewall. By default, pipelines resolve DNS inside the data plane VPC or VNet, reach your Redpanda cluster on the Kafka API, HTTP Proxy, and Schema Registry ports, and connect to public endpoints on the internet.

Connections to private addresses are blocked on most ports, whether the destination sits inside the data plane VPC or VNet or in a network you peer to the cluster. To open a private destination, add an egress allowlist to your cluster. See xref:networking:byoc/connect-egress-allowlist.adoc[Configure Egress for Redpanda Connect Pipelines].

Redpanda always blocks pipeline traffic to link-local addresses (`169.254.0.0/16`), including the cloud provider instance metadata service, to the Kubernetes API server, and to the kubelet ports 10250 and 10255. You cannot add these destinations to an egress allowlist.

== NAT gateways

A NAT (Network Address Translation) gateway allows resources in a private network to access the internet, while blocking inbound connections.
Expand Down
23 changes: 23 additions & 0 deletions modules/networking/partials/connect-egress-allowlist-rules.adoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
Each rule allows outbound traffic to one CIDR block on one port or port range:

|===
| Field or attribute | Required | Description

| `cidr`
| Yes
| Destination IPv4 CIDR block, for example `10.0.32.0/20`. IPv6 is not supported.

| `port_start`
| Yes
| First TCP or UDP port to allow, from 1 to 65535.

| `port_end`
| No
| Last TCP or UDP port to allow, up to 65535. Omit this field to allow a single port. When set, the value must be greater than or equal to `port_start`.
|===

Limits:

* A cluster accepts a maximum of 16 rules.
* Each combination of `cidr`, `port_start`, and `port_end` must be unique.
* Rules cannot override the destinations that Redpanda always blocks, such as link-local addresses and the Kubernetes API server. See xref:networking:byoc/connect-egress-allowlist.adoc#default-pipeline-egress[Default pipeline egress].
Loading