Kubernetes Networking on Hetzner with Kubespray

Over the past years, I have set up and operated a good number of Kubernetes clusters on Hetzner infrastructure for various customers — on Hetzner Cloud servers, on bare metal servers, and on mixtures of both. Along the way, I have gathered a lot of experience with what works well on Hetzner and where the pitfalls are: MTU values that differ between cloud networks and vSwitches, bare metal nodes that don't report their private IPs, or a cloud controller manager that cannot handle vSwitch routes yet.

With every new cluster, the same networking questions come up again: which IP ranges to use, how to get bare metal servers into the cloud network, and which of the many kubespray variables actually matter. This article is my personal reference for exactly that — a network layout and a kubespray configuration for a Kubernetes cluster running on Hetzner Cloud servers, on Hetzner bare metal servers, or on both.

Two goals and one assumption guided the decisions below:

• The configuration should work for cloud servers as well as for bare metal servers.
• It should be prepared for native routing. Hetzner currently does not support routing pod networks to bare metal servers, so mixed clusters still need an overlay network. But as soon as that support arrives, switching over should be a matter of changing only a few variables.
• Even if Hetzner bare metal servers are used, we assume that a Hetzner Load Balancer is used for ingress.

Network Configuration

Everything starts in the Hetzner Cloud Console:

• Create a Hetzner Cloud project.
• Add a network with an IP range, e.g., 10.0.0.0/16.
• Add a subnet for the load balancer(s), e.g., 10.0.0.0/24.
• Add a subnet for the cloud servers, e.g., 10.0.1.0/24.
• If bare metal servers are involved:
  • Create a vSwitch and add the bare metal servers to it.
  • Add a subnet for the bare metal servers, e.g., 10.0.2.0/24, and connect it to the vSwitch you just created.

Kubespray Configuration

IP Ranges

The pod and service networks are placed inside the IP range of the Hetzner network. That is not required for an overlay network, but it is a precondition for native routing later on, since Hetzner can only route IP ranges that belong to the network itself.

## --- k8s_cluster/k8s-cluster.yml
kube_service_addresses: 10.0.64.0/18
kube_pods_subnet: 10.0.128.0/18

Cloud Controller Manager

To integrate the cluster with the Hetzner API, a cloud controller manager is needed. I prefer to install it myself using the upstream Helm chart rather than relying on the version bundled with kubespray, so kubespray is told to prepare the cluster for an external cloud controller manager without deploying one:

## --- all/all.yml
cloud_provider: external
external_cloud_provider: manual

Container Network Interface

Cilium is my default choice. It is a safe long-term bet: Cilium graduated within the CNCF in 2023, at that time the second most active CNCF project in terms of commits. And it implements the whole data path in eBPF instead of iptables, which is what makes it possible to drop kube-proxy entirely further down (Kubernetes Without kube-proxy):

## --- k8s_cluster/k8s-cluster.yml
kube_network_plugin: cilium
kube_owner: root

Cilium's init containers run as UID 0, but with all capabilities dropped, including CAP_DAC_OVERRIDE — which means that even root is subject to the regular file permission checks. Therefore, kube_owner must be set to root for Cilium.

Network MTU

Cilium needs to know the MTU of the underlying network, and Hetzner uses two different values here:

• 1450 bytes inside a Hetzner cloud network
• 1400 bytes for Hetzner vSwitches

Two things are worth knowing about cilium_mtu. First, it refers to the MTU of the underlying network, not to the one the pods will end up with — Cilium subtracts the encapsulation overhead (50 bytes for VXLAN) on its own. Second, it is a cluster-wide setting, which means the smallest MTU on any path wins:

• cloud servers only: 1450
• bare metal servers involved: 1400, for every node, including the cloud ones

Routing

As long as Hetzner does not route pod networks to bare metal servers, pod traffic has to be encapsulated, so Cilium runs in VXLAN mode. Hetzner states this requirement themselves: "Using the routing feature of private networks is not supported, so this requires a CNI plugin with encapsulation methods, such as Cilium with routing mode tunnel" (Attach Load Balancers to Robot Private IPs). The native routing CIDR is prepared already, but only takes effect once the tunnel is switched off:

## --- k8s_cluster/k8s-net-cilium.yml
cilium_mtu: 1400
cilium_tunnel_mode: vxlan
cilium_native_routing_cidr: "{% if cilium_tunnel_mode == 'disabled' %}{{ kube_pods_subnet }}{% endif %}"

Once Hetzner supports routing pod networks to bare metal servers, setting cilium_tunnel_mode: disabled and re-enabling the route controller of the cloud controller manager (see below) is all it takes — the routes inside the Hetzner network are then maintained for you. cilium_mtu stays at 1400, as the vSwitch remains the limiting factor; dropping the VXLAN header simply gives the pods the full 1400 bytes instead of 1350.

kube-proxy

Cilium can take over the work of kube-proxy and implement Kubernetes services in eBPF instead of iptables or IPVS, so there is no reason to keep kube-proxy around as well:

## --- k8s_cluster/k8s-net-cilium.yml
cilium_kube_proxy_replacement: true

## --- k8s_cluster/k8s-cluster.yml
kube_proxy_remove: true

Cluster Applications

Two Hetzner components complete the setup and are installed into the cluster afterwards:

• hcloud-cloud-controller-manager — the cloud controller manager kubespray deliberately left out above
• csi-driver — for provisioning Hetzner Cloud volumes as persistent volumes

For a cluster containing bare metal servers, the cloud controller manager needs two adjustments. Give it the network (HCLOUD_NETWORK), so that it can add bare metal nodes to a Load Balancer by their private vSwitch IP, but switch off its route controller (HCLOUD_NETWORK_ROUTES_ENABLED=false, plus networking.enabled: false in the Helm chart), as it cannot deal with vSwitches yet. This is also the switch to flip once Hetzner supports routing pod networks to bare metal servers.

Note that the bare metal nodes need an InternalIP for this to work: Hetzner does not report vSwitch IPs automatically, so the address has to be passed to the kubelet via --node-ip (in kubespray, that is the ip variable of the according inventory host).