diff --git a/docs/networking/kubeovn-pureunderlay.md b/docs/networking/kubeovn-pureunderlay.md index 4070edabf2..f903b21a43 100644 --- a/docs/networking/kubeovn-pureunderlay.md +++ b/docs/networking/kubeovn-pureunderlay.md @@ -95,11 +95,13 @@ When setting up a pure underlay, the overlay network must use the following conf For descriptions of settings that you must configure, see [Subnet Settings](./kubeovn-vpc.md#subnet-settings). -![](/img/subnetexternal.png) +![](/img/subnetdhcp.png) :::info important When setting up a pure underlay, you must specify the name of the VLAN network that is mapped to the provider network. +If using this as external network for the `VPC NAT Gateway`, then make sure to enable DHCP and set `dns_server=8.8.8.8` and `natOutgoing` as `true`. +This will make sure to install default route and resolve dns on the VMs which are using these subnets. ::: diff --git a/docs/networking/kubeovn-vpcnatgateway.md b/docs/networking/kubeovn-vpcnatgateway.md index 0d3a3bd080..5337a36b4c 100644 --- a/docs/networking/kubeovn-vpcnatgateway.md +++ b/docs/networking/kubeovn-vpcnatgateway.md @@ -33,114 +33,61 @@ In Harvester v1.6.x and v1.7.x, Outbound and Inbound connectivity for VMs works With the introduction of kubeovn as secondary CNI in Harvester v1.8.0, Outbound and Inbound connectivity for VMs works on subnets created on any custom VPCs using VPC NAT Gateway functionality. -### Example Configuration and Working +### Create a Tenant Network -#### Create a network attachment definition (tenant or internal network) +#### Create an Overlay Network -``` -apiVersion: k8s.cni.cncf.io/v1 -kind: NetworkAttachmentDefinition -metadata: - labels: - network.harvesterhci.io/ready: "true" - network.harvesterhci.io/type: OverlayNetwork - name: vswitchinternal - namespace: default -spec: - config: '{"cniVersion":"0.3.1","name":"vswitchinternal","type":"kube-ovn","server_socket": - "/run/openvswitch/kube-ovn-daemon.sock", "provider": "vswitchinternal.default.ovn"}' -``` +Follow the instructions in [Create an Overlay Network](./harvester-network.md#create-an-overlay-network) to create `vswitchinternal`. -#### Create a network attachment definition (external network) +#### Create a Custom VPC -``` -kubectl get net-attach-def vswitchexternal1 -o yaml -apiVersion: k8s.cni.cncf.io/v1 -kind: NetworkAttachmentDefinition -metadata: - labels: - network.harvesterhci.io/ready: "true" - network.harvesterhci.io/type: OverlayNetwork - name: vswitchexternal - namespace: kube-system -spec: - config: '{"cniVersion":"0.3.1","name":"vswitchexternal","master": "eno50","type":"kube-ovn","server_socket": - "/run/openvswitch/kube-ovn-daemon.sock", "provider": "vswitchexternal.kube-system.ovn"}' +Follow the instructions in [VPC Settings](./kubeovn-vpc.md#vpc-settings) to create `commonvpc`. -``` +#### Create a Subnet in VPC -#### Create a custom vpc +Follow the instructions in [Subnet Settings](./kubeovn-vpc.md#subnet-settings) to create `subnetinternal` in `commonvpc` -``` -apiVersion: kubeovn.io/v1 -kind: Vpc -metadata: - name: custom-vpc -spec: {} +![](/img/subnetinternal.png) -``` -#### create a subnet using the internal or tenant network in `custom-vpc` +:::info important -``` -apiVersion: kubeovn.io/v1 -kind: Subnet -metadata: - name: subnetinternal -spec: - cidrBlock: 172.20.10.0/24 - default: false - enableLb: true - excludeIps: - - 172.20.10.1 - gateway: 172.20.10.1 - gatewayNode: "" - natOutgoing: true - private: false - protocol: IPv4 - provider: vswitchinternal.default.ovn - vpc: custom-vpc +When creating a subnet for tenant network, make sure to enable DHCP and set `dns_server=8.8.8.8` and `natOutgoing` as `true`. +This will make sure to install default route and resolve dns on the VMs which are using these subnets. -``` -#### Create a subnet using the external network +::: -``` -apiVersion: kubeovn.io/v1 -kind: Subnet -metadata: - name: subnetexternal -spec: - cidrBlock: 10.115.48.0/21 - default: false - enableLb: true - excludeIps: - - 10.115.55.254 - gateway: 10.115.55.254 - gatewayNode: "" - natOutgoing: true - private: false - protocol: IPv4 - provider: vswitchexternal.kube-system.ovn - vpc: custom-vpc +### Create an External Network -``` +Follow the instructions in [Pure Underlay Networking](./harvester-network.md#kubeovn-pureunderlay#underlay-configuration) to create an external network. + +### VPC NAT Gateway #### Create the vpc nat gateway config -``` -kind: VpcNatGateway -apiVersion: kubeovn.io/v1 -metadata: - annotations: - k8s.v1.cni.cncf.io/networks: default/vswitchinternal - name: gw1 -spec: - vpc: custom-vpc - subnet: subnetinternal - lanIp: 172.20.10.254 - externalSubnets: - - subnetexternal +1. On the Harvester UI, go to **Overlay Networks > NAT & Internet > Gateways**. + + ![](/img/vpc-nat-config.png) + +1. Click **Create**. + +1. Specify a unique name for the vpc nat gw. + +1. On the **Basic** tab, configure the following settings: + + - **Internal Tenant Network**: Name of the internal tenant network(`vswitchinternal`) + + - **VPC** VPC of the internal tenant network subnet (`commonvpc`) + + - **Subnet**: subnet of the internal tenant network (`subnetinternal`) + + - **LAN IP**: Gw Ip of the internal tenant network + +1. On the **External Subnets** tab, configure the following settings: + + - **Subnet**: subnet of the underlay external network(`subnetexternal`) + +1. Click **Create**. -``` #### Verify if a new vpcnatgw statefulset and a pod created ``` @@ -148,15 +95,15 @@ kubectl get statefulset -n kube-system vpc-nat-gw-gw1 -o yaml apiVersion: apps/v1 kind: StatefulSet metadata: - creationTimestamp: "2026-03-06T22:33:47Z" + creationTimestamp: "2026-06-05T22:42:17Z" generation: 1 labels: app: vpc-nat-gw-gw1 ovn.kubernetes.io/vpc-nat-gw: "true" name: vpc-nat-gw-gw1 namespace: kube-system - resourceVersion: "12139577" - uid: cb48e680-95e8-408d-be47-23499ad0ac5c + resourceVersion: "18233827" + uid: ae7c9895-8a7f-46f6-b71d-be9f9b36b79a spec: persistentVolumeClaimRetentionPolicy: whenDeleted: Retain @@ -172,15 +119,12 @@ spec: template: metadata: annotations: - externalvswitch.kube-system.ovn.kubernetes.io/routes: '[{"dst":"0.0.0.0/0","gw":"10.115.55.254"}]' - internalvswitch.default.ovn.kubernetes.io/ip_address: 172.20.10.254 - internalvswitch.default.ovn.kubernetes.io/logical_switch: internalsubnet - internalvswitch.default.ovn.kubernetes.io/routes: '[{"dst":"10.55.0.0/16","gw":"172.20.10.1"},{"dst":"10.115.48.0/21","gw":"172.20.10.1"}]' - internalvswitch.default.ovn.kubernetes.io/vpc_nat_gw: gw1 - k8s.v1.cni.cncf.io/networks: default/internalvswitch, kube-system/externalvswitch - ovn.kubernetes.io/ip_address: 172.20.10.254 - ovn.kubernetes.io/logical_switch: internalsubnet + k8s.v1.cni.cncf.io/networks: default/vswitchinternal, kube-system/vswitchexternal ovn.kubernetes.io/vpc_nat_gw: gw1 + vswitchexternal.kube-system.ovn.kubernetes.io/routes: '[{"dst":"0.0.0.0/0","gw":"10.115.55.254"}]' + vswitchinternal.default.ovn.kubernetes.io/ip_address: 172.20.10.1 + vswitchinternal.default.ovn.kubernetes.io/logical_switch: subnetinternal + vswitchinternal.default.ovn.kubernetes.io/routes: '[{"dst":"10.55.0.0/16","gw":"172.20.10.1"}]' labels: app: vpc-nat-gw-gw1 ovn.kubernetes.io/vpc-nat-gw: "true" @@ -194,7 +138,7 @@ spec: - name: GATEWAY_V4 value: 10.115.55.254 - name: GATEWAY_V6 - image: docker.io/kubeovn/vpc-nat-gateway:v1.15.0 + image: docker.io/kubeovn/vpc-nat-gateway:v1.16.1 imagePullPolicy: IfNotPresent lifecycle: postStart: @@ -221,12 +165,13 @@ status: availableReplicas: 1 collisionCount: 0 currentReplicas: 1 - currentRevision: vpc-nat-gw-gw1-98ff96d76 + currentRevision: vpc-nat-gw-gw1-5659bc556b observedGeneration: 1 readyReplicas: 1 replicas: 1 - updateRevision: vpc-nat-gw-gw1-98ff96d76 + updateRevision: vpc-nat-gw-gw1-5659bc556b updatedReplicas: 1 + ``` ``` kubectl describe pod vpc-nat-gw-gw1-0 -n kube-system @@ -235,96 +180,95 @@ Namespace: kube-system Priority: 0 Service Account: default Node: hp-46/10.115.14.70 -Start Time: Fri, 06 Mar 2026 22:33:47 +0000 +Start Time: Fri, 05 Jun 2026 22:42:18 +0000 Labels: app=vpc-nat-gw-gw1 apps.kubernetes.io/pod-index=0 - controller-revision-hash=vpc-nat-gw-gw1-98ff96d76 + controller-revision-hash=vpc-nat-gw-gw1-5659bc556b ovn.kubernetes.io/vpc-nat-gw=true statefulset.kubernetes.io/pod-name=vpc-nat-gw-gw1-0 -Annotations: cni.projectcalico.org/containerID: 3cf564b6124aae048a097bd9777dd3ac583b293357cb819a85146515e2ba1e52 - cni.projectcalico.org/podIP: 10.52.1.66/32 - cni.projectcalico.org/podIPs: 10.52.1.66/32 - externalvswitch.kube-system.ovn.kubernetes.io/allocated: true - externalvswitch.kube-system.ovn.kubernetes.io/cidr: 10.115.48.0/21 - externalvswitch.kube-system.ovn.kubernetes.io/gateway: 10.115.55.254 - externalvswitch.kube-system.ovn.kubernetes.io/ip_address: 10.115.48.1 - externalvswitch.kube-system.ovn.kubernetes.io/logical_router: commonvpc - externalvswitch.kube-system.ovn.kubernetes.io/logical_switch: externalsubnet - externalvswitch.kube-system.ovn.kubernetes.io/mac_address: fe:01:4a:61:43:6e - externalvswitch.kube-system.ovn.kubernetes.io/pod_nic_type: veth-pair - externalvswitch.kube-system.ovn.kubernetes.io/routed: true - externalvswitch.kube-system.ovn.kubernetes.io/routes: [{"dst":"0.0.0.0/0","gw":"10.115.55.254"}] - internalvswitch.default.ovn.kubernetes.io/allocated: true - internalvswitch.default.ovn.kubernetes.io/cidr: 172.20.10.0/24 - internalvswitch.default.ovn.kubernetes.io/gateway: 172.20.10.1 - internalvswitch.default.ovn.kubernetes.io/ip_address: 172.20.10.254 - internalvswitch.default.ovn.kubernetes.io/logical_router: commonvpc - internalvswitch.default.ovn.kubernetes.io/logical_switch: internalsubnet - internalvswitch.default.ovn.kubernetes.io/mac_address: d6:2d:c1:95:45:15 - internalvswitch.default.ovn.kubernetes.io/pod_nic_type: veth-pair - internalvswitch.default.ovn.kubernetes.io/routed: true - internalvswitch.default.ovn.kubernetes.io/routes: [{"dst":"10.55.0.0/16","gw":"172.20.10.1"},{"dst":"10.115.48.0/21","gw":"172.20.10.1"}] - internalvswitch.default.ovn.kubernetes.io/vpc_cidrs: ["172.20.10.0/24"] - internalvswitch.default.ovn.kubernetes.io/vpc_nat_gw: gw1 +Annotations: cni.projectcalico.org/containerID: 1b133ea1f17aac64c3d70bde83e1244fe07eb5206a78ec7c76f6099a450e093f + cni.projectcalico.org/podIP: 10.52.1.30/32 + cni.projectcalico.org/podIPs: 10.52.1.30/32 k8s.v1.cni.cncf.io/network-status: [{ "name": "k8s-pod-network", "interface": "eth0", "ips": [ - "10.52.1.66" + "10.52.1.30" ], - "mac": "be:86:12:cc:e3:30", + "mac": "4a:26:76:cd:f3:fa", "default": true, "dns": {} },{ - "name": "default/internalvswitch", + "name": "default/vswitchinternal", "interface": "net1", "ips": [ - "172.20.10.254" + "172.20.10.1" ], - "mac": "d6:2d:c1:95:45:15", + "mac": "22:02:d9:04:e7:bb", "dns": {} },{ - "name": "kube-system/externalvswitch", + "name": "kube-system/vswitchexternal", "interface": "net2", "ips": [ - "10.115.48.1" + "10.115.48.6" ], - "mac": "fe:01:4a:61:43:6e", + "mac": "c6:1d:c3:e8:a5:76", "dns": {}, "gateway": [ "10.115.55.254" ] }] - k8s.v1.cni.cncf.io/networks: default/internalvswitch, kube-system/externalvswitch - ovn.kubernetes.io/ip_address: 172.20.10.254 - ovn.kubernetes.io/logical_switch: internalsubnet + k8s.v1.cni.cncf.io/networks: default/vswitchinternal, kube-system/vswitchexternal ovn.kubernetes.io/vpc_nat_gw: gw1 ovn.kubernetes.io/vpc_nat_gw_init: true + vswitch2.default.ovn.kubernetes.io/vpc_cidrs: ["172.30.0.0/24"] + vswitchexternal.kube-system.ovn.kubernetes.io/allocated: true + vswitchexternal.kube-system.ovn.kubernetes.io/cidr: 10.115.48.0/21 + vswitchexternal.kube-system.ovn.kubernetes.io/gateway: 10.115.55.254 + vswitchexternal.kube-system.ovn.kubernetes.io/ip_address: 10.115.48.6 + vswitchexternal.kube-system.ovn.kubernetes.io/logical_switch: subnetexternal + vswitchexternal.kube-system.ovn.kubernetes.io/mac_address: c6:1d:c3:e8:a5:76 + vswitchexternal.kube-system.ovn.kubernetes.io/pod_nic_type: veth-pair + vswitchexternal.kube-system.ovn.kubernetes.io/provider_network: pn1 + vswitchexternal.kube-system.ovn.kubernetes.io/routed: true + vswitchexternal.kube-system.ovn.kubernetes.io/routes: [{"dst":"0.0.0.0/0","gw":"10.115.55.254"}] + vswitchexternal.kube-system.ovn.kubernetes.io/vlan_id: 2017 + vswitchinternal.default.ovn.kubernetes.io/allocated: true + vswitchinternal.default.ovn.kubernetes.io/cidr: 172.20.10.0/24 + vswitchinternal.default.ovn.kubernetes.io/gateway: 172.20.10.1 + vswitchinternal.default.ovn.kubernetes.io/ip_address: 172.20.10.1 + vswitchinternal.default.ovn.kubernetes.io/logical_router: commonvpc + vswitchinternal.default.ovn.kubernetes.io/logical_switch: subnetinternal + vswitchinternal.default.ovn.kubernetes.io/mac_address: 22:02:d9:04:e7:bb + vswitchinternal.default.ovn.kubernetes.io/pod_nic_type: veth-pair + vswitchinternal.default.ovn.kubernetes.io/routed: true + vswitchinternal.default.ovn.kubernetes.io/routes: [{"dst":"10.55.0.0/16","gw":"172.20.10.1"}] + vswitchinternal.default.ovn.kubernetes.io/vpc_cidrs: ["172.20.10.0/24"] Status: Running -IP: 10.52.1.66 +IP: 10.52.1.30 IPs: - IP: 10.52.1.66 + IP: 10.52.1.30 Controlled By: StatefulSet/vpc-nat-gw-gw1 Containers: vpc-nat-gw: - Container ID: containerd://efcab898c60ad94ee5d79a598f96d5b0becf65c2ae917d6f02246b227be40eca - Image: docker.io/kubeovn/vpc-nat-gateway:v1.15.0 - Image ID: docker.io/kubeovn/vpc-nat-gateway@sha256:4c090dffaaf5c44593883bfa15a4524e860ac4d7c32555d98284aa8fa042ba6a + Container ID: containerd://4e0d66777f1e56f6402e1106fba2feb9b4a6f0eaa467101a5f7b159ff9981c04 + Image: docker.io/kubeovn/vpc-nat-gateway:v1.16.1 + Image ID: docker.io/kubeovn/vpc-nat-gateway@sha256:59afff3ee0583378ae73ead77f3459745fef300d111f3a5e60ca555b19ca8e7b Port: Host Port: Command: sleep infinity State: Running - Started: Fri, 06 Mar 2026 22:34:01 +0000 + Started: Fri, 05 Jun 2026 22:42:29 +0000 Ready: True Restart Count: 0 Environment: GATEWAY_V4: 10.115.55.254 GATEWAY_V6: Mounts: - /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-qgb7d (ro) + /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-97xmd (ro) Conditions: Type Status PodReadyToStartContainers True @@ -333,7 +277,7 @@ Conditions: ContainersReady True PodScheduled True Volumes: - kube-api-access-qgb7d: + kube-api-access-97xmd: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt @@ -358,80 +302,37 @@ vpc-nat-gw-gw1-0:/kube-ovn# ip addr show valid_lft forever preferred_lft forever inet6 ::1/128 scope host proto kernel_lo valid_lft forever preferred_lft forever -2: eth0@if4949: mtu 1450 qdisc noqueue state UP group default qlen 1000 - link/ether be:86:12:cc:e3:30 brd ff:ff:ff:ff:ff:ff link-netnsid 0 - inet 10.52.1.66/32 scope global eth0 +2: eth0@if9727: mtu 1450 qdisc noqueue state UP group default qlen 1000 + link/ether 4a:26:76:cd:f3:fa brd ff:ff:ff:ff:ff:ff link-netnsid 0 + inet 10.52.1.30/32 scope global eth0 valid_lft forever preferred_lft forever - inet6 fe80::bc86:12ff:fecc:e330/64 scope link proto kernel_ll + inet6 fe80::4826:76ff:fecd:f3fa/64 scope link proto kernel_ll valid_lft forever preferred_lft forever -4950: net1@if4951: mtu 1450 qdisc noqueue state UP group default - link/ether d6:2d:c1:95:45:15 brd ff:ff:ff:ff:ff:ff link-netnsid 0 - inet 172.20.10.254/24 brd 172.20.10.255 scope global net1 +9728: net1@if9729: mtu 1450 qdisc noqueue state UP group default + link/ether 22:02:d9:04:e7:bb brd ff:ff:ff:ff:ff:ff link-netnsid 0 + inet 172.20.10.1/24 brd 172.20.10.255 scope global net1 valid_lft forever preferred_lft forever - inet6 fe80::d42d:c1ff:fe95:4515/64 scope link proto kernel_ll + inet6 fe80::2002:d9ff:fe04:e7bb/64 scope link proto kernel_ll valid_lft forever preferred_lft forever -4952: net2@if4953: mtu 1450 qdisc noqueue state UP group default - link/ether fe:01:4a:61:43:6e brd ff:ff:ff:ff:ff:ff link-netnsid 0 - inet 10.115.48.1/21 brd 10.115.55.255 scope global net2 +9730: net2@if9731: mtu 1500 qdisc noqueue state UP group default + link/ether c6:1d:c3:e8:a5:76 brd ff:ff:ff:ff:ff:ff link-netnsid 0 + inet 10.115.48.6/21 brd 10.115.55.255 scope global net2 valid_lft forever preferred_lft forever inet 10.115.55.200/21 scope global secondary net2 valid_lft forever preferred_lft forever - inet6 fe80::fc01:4aff:fe61:436e/64 scope link proto kernel_ll + inet6 fe80::c41d:c3ff:fee8:a576/64 scope link proto kernel_ll valid_lft forever preferred_lft forever vpc-nat-gw-gw1-0:/kube-ovn# ip route show -default via 10.115.55.254 dev net2 -10.55.0.0/16 via 172.20.10.1 dev net1 -10.115.48.0/21 dev net2 proto kernel scope link src 10.115.48.1 -169.254.1.1 dev eth0 scope link -172.20.10.0/24 dev net1 proto kernel scope link src 172.20.10.254 +default via 10.115.55.254 dev net2 onlink +10.55.0.0/16 via 172.20.10.1 dev net1 +10.115.48.0/21 dev net2 proto kernel scope link src 10.115.48.6 +169.254.1.1 dev eth0 scope link +172.20.10.0/24 dev net1 proto kernel scope link src 172.20.10.1 +172.30.0.0/24 via 172.20.10.1 dev net1 ``` -#### Create a provider network with vlan id 2017 and physical interface and a vlan network attached to the provider network. - -``` -apiVersion: kubeovn.io/v1 -kind: ProviderNetwork -metadata: - name: pn1 -spec: - defaultInterface: eno50 -``` - -``` -apiVersion: kubeovn.io/v1 -kind: Vlan -metadata: - name: vlan2017 -spec: - id: 2017 - provider: pn1 -``` - -#### Edit subnet subnetexternal to use vlan as vlan2017 (this will attach this subnet to the provider network underlay) - -``` -apiVersion: kubeovn.io/v1 -kind: Subnet -metadata: - name: subnetexternal -spec: - cidrBlock: 10.115.48.0/21 - default: false - enableLb: true - excludeIps: - - 10.115.55.254 - gateway: 10.115.55.254 - gatewayNode: "" - natOutgoing: true - private: false - protocol: IPv4 - provider: vswitchexternal.kube-system.ovn - vlan: vlan2017 - vpc: custom-vpc - -``` -#### Verify provider network bridge (br-pn1) and external subnet attached on the ovs (patch-localnet.externalsubnet-to-br-int) +#### Verify provider network bridge (br-pn1) and external subnet attached on the ovs (patch-localnet.subnetexternal-to-br-int) ``` kubectl exec -it ovs-ovn-q92zk -n kube-system -- /bin/bash @@ -442,10 +343,10 @@ nobody@hp-65:/kube-ovn$ ovs-vsctl show Port br-pn1 Interface br-pn1 type: internal - Port patch-localnet.externalsubnet-to-br-int - Interface patch-localnet.externalsubnet-to-br-int + Port patch-localnet.subnetexternal-to-br-int + Interface patch-localnet.subnetexternal-to-br-int type: patch - options: {peer=patch-br-int-to-localnet.externalsubnet} + options: {peer=patch-br-int-to-localnet.subnetexternal} Port eno50 trunks: [0, 2017] Interface eno50 @@ -463,10 +364,10 @@ nobody@hp-65:/kube-ovn$ ovs-vsctl show Port ovn0 Interface ovn0 type: internal - Port patch-br-int-to-localnet.externalsubnet - Interface patch-br-int-to-localnet.externalsubnet + Port patch-br-int-to-localnet.subnetexternal + Interface patch-br-int-to-localnet.subnetexternal type: patch - options: {peer=patch-localnet.externalsubnet-to-br-int} + options: {peer=patch-localnet.subnetexternal-to-br-int} Port "7e8b0_37a8eec_h" Interface "7e8b0_37a8eec_h" Port "6159403e_net2_h" @@ -474,32 +375,47 @@ nobody@hp-65:/kube-ovn$ ovs-vsctl show ovs_version: "3.5.3" ``` -#### SNAT for external connectivity from Overlay VMs +### SNAT for external connectivity from Overlay VMs -##### Create EIP and SNAT resource +#### Create EIP -``` -apiVersion: kubeovn.io/v1 -kind: IptablesEIP -metadata: - name: my-eip -spec: - natGwDp: gw1 # Name of your VpcNatGateway - externalSubnet: subnetexternal - v4ip: 10.115.55.200 #random public ip from external subnet -``` +1. On the Harvester UI, go to **Overlay Networks > NAT & Internet > External IPs**. -``` -apiVersion: kubeovn.io/v1 -kind: IptablesSnatRule -metadata: - name: my-snat -spec: - eip: my-eip - internalCIDR: 172.20.10.0/24 #internal subnet CIDR -``` + ![](/img/eip.png) + +1. Click **Create**. + +1. Specify a unique name for the eip. + +1. On the **Basic** tab, configure the following settings: + + - **VPC NAT GW**: Name of the vpc nat gw(`gw1`) + + - **External Subnet** subnet of the underlay external network (`subnetexternal`) + + - **v4 IP**: choose an IP address from external subnet CIDR Range to be used as public IP for SNAT/DNAT + +1. Click **Create**. + +#### Create SNAT + +1. On the Harvester UI, go to **Overlay Networks > NAT & Internet > Source Rules**. + + ![](/img/snat.png) + +1. Click **Create**. + +1. Specify a unique name for the snat. + +1. On the **Basic** tab, configure the following settings: -##### Verify EIP status + - **EIP**: Name of the eip(`my-eip`) + + - **Internal CIDR** `cidr` of the internal tenant subnet + +1. Click **Create**. + +#### Verify EIP status ``` kubectl get eip NAME IP MAC NAT NATGWDP READY @@ -507,7 +423,7 @@ my-eip 10.115.55.200 52:1b:4f:1d:14:ce dnat,snat gw1 true ``` -##### Verify SNAT filter iptable rule created inside the VPC NAT gateway pod +#### Verify SNAT filter iptable rule created inside the VPC NAT gateway pod ``` vpc-nat-gw-gw1-0:/kube-ovn# iptables-legacy-save -t nat @@ -532,36 +448,43 @@ COMMIT ``` -##### Create a VM and attach it to the vswitchinternal overlay network and add the following default route in the guest os. - (172.20.10.254 is the ip addresses on net1 interface on vpc nat gw pod) -``` -ip route add default via 172.20.10.254 dev enp1s0 -``` - +### Create a VM + +See [Create a Virtual Machine](../vm/create-vm.md#how-to-create-a-vm) to create a VM and attach it to the `vswitchinternal` overlay network. + Ping from VM (inside guest os) to 8.8.8.8 must be successful The traffic from VM reaches net1 of vpc nat gw pod and with route installed egress out of net2 and hits the iptable rule for SNAT and translates 172.20.10.0/24 subnet ip to 10.115.55.200 for external connectivity. ![](/img/kubeovnSNATFlow.png) -#### DNAT for inbound access (external to overlay VMs) +### DNAT for inbound access (external to overlay VMs) -##### Use the DNAT resource and use same EIP as SNAT +#### Create DNAT -``` -kind: IptablesDnatRule -apiVersion: kubeovn.io/v1 -metadata: - name: dnat01 -spec: - eip: my-eip - externalPort: '8888' - internalIp: 172.20.10.2 #IP of the overlay VM - internalPort: '80' - protocol: tcp +1. On the Harvester UI, go to **Overlay Networks > NAT & Internet > Destination Rules**. -``` -##### Check the DNAT Filter inside the vpc NAT GW pod + ![](/img/dnat.png) + +1. Click **Create**. + +1. Specify a unique name for the dnat. + +1. On the **Basic** tab, configure the following settings: + + - **EIP**: Name of the eip(`my-eip`) same used for `snat` + + - **External Port** The port on which incoming traffic is received from external clients. + + - **Internal IP** The destination IP address to which the incoming traffic is forwarded.(IP address of the overlay VM) + + - **Internal Port** The destination port on the internal host to which the incoming traffic is translated and forwarded. + + - **Protocol** tcp or udp + +1. Click **Create**. + +#### Check the DNAT Filter inside the vpc NAT GW pod ``` vpc-nat-gw-gw1-0:/kube-ovn# iptables-legacy -t nat -L -n -v 2>/dev/null | grep -E "DNAT" @@ -575,7 +498,10 @@ Chain SHARED_DNAT (1 references) ``` -##### Create a VM attached to overlay network on subnetinternal and Install nginx on VM to check inbound access +### Create a VM + +See [Create a Virtual Machine](../vm/create-vm.md#how-to-create-a-vm) to create a VM attached to overlay network on `subnetinternal` and Install nginx on VM to check inbound access + ``` sudo apt update sudo apt install nginx -y @@ -588,5 +514,3 @@ Now curl -k http://10.115.55.200:8888 from external must be successful ``` ![](/img/kubeovnDNATFlow.png) - - diff --git a/docs/rancher/guest-cluster-overlay-network.md b/docs/rancher/guest-cluster-overlay-network.md new file mode 100644 index 0000000000..f1b3197e04 --- /dev/null +++ b/docs/rancher/guest-cluster-overlay-network.md @@ -0,0 +1,79 @@ +--- +sidebar_position: 11 +sidebar_label: Guest Cluster using Overlay Network +title: "Guest Cluster using Overlay Network" +keywords: + - Harvester + - Rancher + - KubeOVN +--- + + + + + +## Provisioning Downstream Kubernetes Clusters on Harvester Using Rancher and Kube-OVN Networking + +### Introduction + +Rancher provides centralized management for Kubernetes clusters across multiple environments. When integrated with Harvester, Rancher can provision guest Kubernetes clusters directly on virtual machine infrastructure managed by Harvester. + +Using the Harvester Node Driver, Rancher automatically creates virtual machines that serve as control plane and worker nodes. This simplifies cluster deployment and lifecycle management by providing a single interface for infrastructure and Kubernetes operations. + +While guest clusters can use traditional virtual machine networking, they can also leverage Kube-OVN overlay or underlay networking to provide a flexible and Kubernetes-native networking model for workloads. + +## Guest Cluster Deployment on Underlay Network + +Deploying guest clusters on pure underlay networks is functionally similar to using traditional virtual machine VLAN networks, but with the added advantages of Kube-OVN features such as network policies and tenant-isolated VPCs. + +For information about setting up the overlay infrastructure, see [Create an underlay network](../networking/kubeovn-pureunderlay.md#underlay-configuration). + +## Guest Cluster Deployment on Overlay Network + +Deploying guest clusters on overlay networks requires configuring a VPC NAT gateway to provide inbound and outbound connectivity to the virtual machines through DNAT and SNAT. + +The key architectural features are as follows: + +- Harvester provides the virtual machine infrastructure for the guest cluster's control plane and worker nodes. +- Rancher provisions and manages the lifecycle of the guest cluster. +- Kube-OVN provides networking within the guest cluster using overlay tunnels between the guest nodes. +- Pod communication is encapsulated and transported across the overlay network connecting the guest nodes. + +Kube-OVN overlay networking enables guest clusters to manage pod networking independently of the underlying Harvester infrastructure network. This provides the following operational benefits: + +- Simplified cluster deployment +- Reduced dependence on physical network configuration +- Dynamic cluster scaling without additional VLAN planning +- Consistent networking model across environments + +For information about setting up the overlay infrastructure, see [Create a VPC NAT Gateway](../networking/kubeovn-vpcnatgateway.md#kubeovn-as-secondary-cni). + +## Guest Cluster Provisioning + +### Importing Harvester Clusters into Rancher + +Harvester clusters are imported through Rancher's **Virtualization Management** interface. + +The following occur after a Harvester cluster is imported: + +- Rancher recognizes the cluster as a Harvester provider. +- Virtual machine resources become available through Rancher. +- Harvester can be selected as an infrastructure provider when creating downstream clusters. + +For detailed instructions, see [Importing Harvester Cluster](./virtualization-management.md#importing-harvester-cluster). + +### Provisioning a Guest Cluster + +Rancher uses the Harvester Node Driver to automatically provision the required virtual machines and bootstrap Kubernetes on them. + +For detailed instructions, see [Harvester Node Driver](./node/rke2-cluster.md). + +:::info important + +In the **Networks** section of the cluster creation screen, you must specify the correct underlay or overlay network. + +::: + +![](/img/v1.9/rancher/gc-overlayvmnetwork.png) + +![](/img/v1.9/rancher/gc-overlayvm.png) diff --git a/static/img/dnat.png b/static/img/dnat.png new file mode 100644 index 0000000000..ab5982938a Binary files /dev/null and b/static/img/dnat.png differ diff --git a/static/img/eip.png b/static/img/eip.png new file mode 100644 index 0000000000..f4d2e2b17b Binary files /dev/null and b/static/img/eip.png differ diff --git a/static/img/snat.png b/static/img/snat.png new file mode 100644 index 0000000000..1a31c859ed Binary files /dev/null and b/static/img/snat.png differ diff --git a/static/img/subnetdhcp.png b/static/img/subnetdhcp.png new file mode 100644 index 0000000000..51a5cc7ec7 Binary files /dev/null and b/static/img/subnetdhcp.png differ diff --git a/static/img/subnetinternal.png b/static/img/subnetinternal.png new file mode 100644 index 0000000000..f83e9b2c2d Binary files /dev/null and b/static/img/subnetinternal.png differ diff --git a/static/img/v1.9/rancher/gc-overlayvm.png b/static/img/v1.9/rancher/gc-overlayvm.png new file mode 100644 index 0000000000..a854e3a1cd Binary files /dev/null and b/static/img/v1.9/rancher/gc-overlayvm.png differ diff --git a/static/img/v1.9/rancher/gc-overlayvmnetwork.png b/static/img/v1.9/rancher/gc-overlayvmnetwork.png new file mode 100644 index 0000000000..47f8c70988 Binary files /dev/null and b/static/img/v1.9/rancher/gc-overlayvmnetwork.png differ diff --git a/static/img/vpc-nat-config.png b/static/img/vpc-nat-config.png new file mode 100644 index 0000000000..b59c70490b Binary files /dev/null and b/static/img/vpc-nat-config.png differ diff --git a/static/img/vpc-nat-config2.png b/static/img/vpc-nat-config2.png new file mode 100644 index 0000000000..ca49d8dcc4 Binary files /dev/null and b/static/img/vpc-nat-config2.png differ