Skip to content

Commit c5fb549

Browse files
SkalaNetworksSkalaNetworks
authored
Merge pull request #9 from super-phenix/limitation
feat(network): add nat documentation
2 parents 6e8ae63 + 610cb3e commit c5fb549

4 files changed

Lines changed: 280 additions & 1 deletion

File tree

‎docs/assets/diagrams/hairpin-nat.svg‎

Lines changed: 46 additions & 0 deletions
Loading

‎docs/operations/network/configure-nat-gateways.md‎

Lines changed: 204 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,204 @@
1+
# Configure NAT gateways
2+
3+
NAT gateways connect private Superphenix subnets to an external network. Before creating NAT gateways or Elastic IPs (EIPs), configure:
4+
5+
1. An external interface with the same name on every Talos node.
6+
2. An external subnet from which Superphenix can allocate NAT gateway addresses.
7+
3. The EIP allocation subnet and, when required, the BGP speaker settings for each availability zone (AZ).
8+
9+
## Configure the external interface in Talos
10+
11+
Every node must expose the external network through an interface with the same name. This example uses `ext0`:
12+
13+
```yaml
14+
machine:
15+
network:
16+
interfaces:
17+
- interface: ext0
18+
routes:
19+
- network: 0.0.0.0/0
20+
gateway: 10.81.5.1
21+
```
22+
23+
Replace the addressing and gateway with values for your external network. The interface can be a physical NIC, bond, or VLAN, but its Talos name must be identical on every node.
24+
25+
If physical interface names differ between nodes, add a `LinkAliasConfig` to each node's Talos configuration. Keep `name` identical and change the MAC address for each node:
26+
27+
```yaml
28+
---
29+
apiVersion: v1alpha1
30+
kind: LinkAliasConfig
31+
name: ext0
32+
selector:
33+
match: mac(link.permanent_addr) == "00:1a:2b:3c:4d:5e"
34+
```
35+
36+
Do not use kernel-style names such as `eth0`, `ens3`, or `enp0s31f6` as the common alias.
37+
38+
## Configure the external subnet
39+
40+
Define the external subnet and its network attachment in the AZ's system configuration. The network attachment must use the common Talos interface as its `master`:
41+
42+
```yaml
43+
systemConfiguration:
44+
apps:
45+
misc:
46+
values:
47+
objects:
48+
external-subnet:
49+
spec:
50+
protocol: IPv4
51+
cidrBlock: 10.81.5.0/24
52+
gateway: 10.81.5.1
53+
excludeIps:
54+
- 10.81.5.0..10.81.5.31
55+
- 10.81.5.255
56+
external-subnet-nad:
57+
spec:
58+
config: '{
59+
"cniVersion": "0.3.0",
60+
"type": "macvlan",
61+
"master": "ext0",
62+
"mode": "bridge",
63+
"ipam": {
64+
"type": "kube-ovn",
65+
"server_socket": "/run/openvswitch/kube-ovn-daemon.sock",
66+
"provider": "external-subnet.kube-system"
67+
}
68+
}'
69+
```
70+
71+
Exclude the network and broadcast addresses, node addresses, router addresses, and any other addresses that Superphenix must not allocate. In a BGP deployment, this subnet is only used to address the NAT gateways and their BGP speakers. A private RFC 1918 range is recommended because these addresses do not need to be public.
72+
73+
!!! warning "External network connectivity"
74+
The common interface on every node must reach the external subnet. The backbone must then learn how to reach the EIPs through either ARP or BGP.
75+
76+
## Choose how to announce EIPs
77+
78+
Superphenix can announce EIPs to the backbone using either ARP or BGP.
79+
80+
### ARP announcement
81+
82+
ARP is the default and requires no BGP configuration. Superphenix answers ARP requests for an EIP on the external network, allowing the upstream gateway to send traffic directly to the NAT gateway.
83+
84+
With ARP, EIPs must belong to the external Layer 2 subnet so that the upstream gateway can resolve them.
85+
86+
ARP is appropriate when:
87+
88+
- The NAT gateways and upstream gateway share the same Layer 2 network.
89+
- You want the simplest setup with no BGP peering.
90+
- The external broadcast domain is small enough to extend to every relevant node.
91+
92+
Its main limitation is that EIP reachability depends on a shared Layer 2 domain. Extending that domain across racks or network zones can increase the broadcast scope and make the network harder to scale.
93+
94+
### BGP announcement
95+
96+
BGP advertises EIP routes to one or more upstream routers. It requires an ASN, a remote ASN, and at least one reachable neighbor.
97+
98+
Unlike ARP, BGP does not require EIPs to belong to `external-subnet`. The external subnet provides private addresses for the NAT gateways and connectivity to their BGP neighbors, while EIPs can be allocated from a separate public prefix and advertised to the backbone.
99+
100+
BGP is preferable when:
101+
102+
- The backbone is routed and does not extend the external Layer 2 domain.
103+
- EIP routes must be propagated beyond one subnet or network segment.
104+
- You want explicit routing policy and better route visibility on the backbone.
105+
- You need routing features such as multiple peers or graceful restart.
106+
107+
BGP requires coordination with the network team and matching peer configuration on the upstream routers, but it generally scales better across racks, network zones, and AZs.
108+
109+
!!! tip "Recommended BGP addressing"
110+
Use a private subnet, such as `10.81.5.0/24`, for NAT gateway and BGP speaker addresses. Use a separate public prefix for EIPs. The examples use `192.0.2.0/24`, which is reserved by RFC 5737 for documentation; replace it with a public prefix routed to your organization.
111+
112+
???+ example "Create a separate EIP subnet for BGP"
113+
Create the public EIP subnet in the same way as `external-subnet`. Unlike the existing `external-subnet` objects, these are new resources, so include their `apiVersion`, `kind`, and `metadata`.
114+
115+
A network attachment definition (NAD) is still required, even though BGP-routed EIPs are not mounted on a host interface. Its macvlan `master` can therefore use a placeholder such as `fake`:
116+
117+
```yaml
118+
systemConfiguration:
119+
apps:
120+
misc:
121+
values:
122+
objects:
123+
192.0.2.0-24:
124+
apiVersion: kubeovn.io/v1
125+
kind: Subnet
126+
metadata:
127+
name: 192.0.2.0-24
128+
spec:
129+
protocol: IPv4
130+
cidrBlock: 192.0.2.0/24
131+
gateway: 192.0.2.1
132+
excludeIps:
133+
- 192.0.2.0..192.0.2.1
134+
- 192.0.2.255
135+
192.0.2.0-24-nad:
136+
apiVersion: "k8s.cni.cncf.io/v1"
137+
kind: NetworkAttachmentDefinition
138+
metadata:
139+
name: 192.0.2.0-24
140+
namespace: kube-system
141+
spec:
142+
config: '{
143+
"cniVersion": "0.3.0",
144+
"type": "macvlan",
145+
"master": "fake",
146+
"mode": "bridge",
147+
"ipam": {
148+
"type": "kube-ovn",
149+
"server_socket": "/run/openvswitch/kube-ovn-daemon.sock",
150+
"provider": "192.0.2.0-24.kube-system"
151+
}
152+
}'
153+
```
154+
155+
The placeholder interface is never used to carry EIP traffic. The BGP speaker advertises the EIPs, and the backbone routes their traffic to the NAT gateway through `external-subnet`.
156+
157+
## Configure NAT gateway defaults
158+
159+
Pass the EIP and NAT gateway defaults through the AZ's `systemConfiguration`. To use ARP announcements, set `eipDefault.externalSubnet` to `external-subnet` and omit `bgpSpeaker`, or leave it disabled.
160+
161+
For BGP, set `eipDefault.externalSubnet` to the separate IPAM subnet that contains the public EIP pool. This adjusts the controller so that it allocates EIPs from the public prefix instead of the private subnet used by the NAT gateways.
162+
163+
The following example assumes:
164+
165+
- `external-subnet` is `10.81.5.0/24` and provides private addresses to the NAT gateways.
166+
- The upstream BGP router is `10.81.5.1`.
167+
- The EIP subnet created above is named `192.0.2.0-24` and represents the example public prefix `192.0.2.0/24`.
168+
169+
```yaml
170+
systemConfiguration:
171+
apps:
172+
superphenix-controller:
173+
values:
174+
productsConfig:
175+
eipDefault:
176+
externalSubnet: "192.0.2.0-24"
177+
natGatewayDefault:
178+
bgpSpeaker:
179+
enabled: true
180+
asn: 64892
181+
remoteAsn: 64890
182+
neighbors:
183+
- 10.81.5.1
184+
enableGracefulRestart: true
185+
extraArgs:
186+
- "-v5"
187+
- "--graceful-restart"
188+
```
189+
190+
Apply this configuration to each AZ that should advertise EIP routes. Values can differ by AZ to match the local routers and autonomous systems.
191+
192+
The BGP speaker accepts these settings under `productsConfig.natGatewayDefault.bgpSpeaker`:
193+
194+
- `enabled`: enables or disables BGP route advertisement.
195+
- `asn`: the local ASN used by the AZ's NAT gateways.
196+
- `remoteAsn`: the ASN expected from the upstream BGP peers.
197+
- `neighbors`: one or more upstream BGP peer addresses.
198+
- `holdTime`: the BGP hold time, expressed as a duration such as `"90s"`.
199+
- `routerId`: the BGP router ID.
200+
- `password`: the BGP session password. Store sensitive values through your normal secret-management process.
201+
- `enableGracefulRestart`: enables BGP graceful restart.
202+
- `extraArgs`: additional arguments passed to the BGP speaker.
203+
204+
Only `enabled` and the settings required by your network need to be specified. Coordinate the ASNs, neighbor addresses, authentication, timers, and graceful-restart behavior with the upstream router configuration.

‎docs/operations/network/hairpin-nat.md‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
# Hairpin NAT
2+
3+
Hairpin NAT lets a workload reach another workload through its Elastic IP (EIP), even when both workloads are on the private network. This is also known as NAT loopback.
4+
5+
Without hairpin NAT, traffic sent to the EIP may leave the private network or fail to return through the same NAT gateway. Applications then need different internal and external addresses for the same service.
6+
7+
Superphenix supports hairpin NAT. When a private client connects to an EIP, the NAT gateway:
8+
9+
1. Replaces the destination EIP with the private IP of the target workload (**DNAT**).
10+
2. Replaces the client's private source IP with that same EIP (**SNAT**).
11+
12+
The source translation forces the response back through the NAT gateway, which reverses both translations and returns the response to the client.
13+
14+
![Traffic flow through hairpin NAT](../../assets/diagrams/hairpin-nat.svg)
15+
16+
In this example:
17+
18+
- The client at `10.20.0.2` connects to the service EIP `198.51.100.20`.
19+
- The NAT gateway changes the destination to the server's private IP, `10.20.0.3`.
20+
- The NAT gateway also changes the source from `10.20.0.2` to `198.51.100.20`.
21+
- The server therefore sees a connection from `198.51.100.20`, not from the client's private IP.
22+
- The client continues to see `198.51.100.20` as the service address.
23+
24+
!!! note "Preserving the client address"
25+
Because hairpin NAT translates the source address, the target workload cannot use the packet's source IP to identify the original client. If the application needs that identity, pass it at the application layer or use the private service address directly.
26+
27+
Hairpin NAT applies to both one-to-one EIP mappings and ports exposed with DNAT rules. The EIP and its destination must already be configured on the subnet's NAT gateway.

‎zensical.toml‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -61,8 +61,10 @@ nav = [
6161
]},
6262
{ "Operations" = [
6363
{ "Network" = [
64+
{ "Configure NAT gateways" = "operations/network/configure-nat-gateways.md" },
6465
{ "Setting the MTU of subnets" = "operations/network/setting-the-mtu-of-subnets.md" },
65-
{ "IPv6 configuration" = "operations/network/ipv6-configuration.md" }
66+
{ "IPv6 configuration" = "operations/network/ipv6-configuration.md" },
67+
{ "Hairpin NAT" = "operations/network/hairpin-nat.md" }
6668
]},
6769
{ "Storage" = [
6870
{ "Configuring the storage cluster" = "operations/storage/configuring-the-storage-cluster.md" },

0 commit comments

Comments
 (0)