Blog | Dr. Matthias Lohr

Kubernetes Networking on Hetzner with Kubespray

2026-09-25T00:00:00Z

Over the past years, I have set up and operated a good number of Kubernetes clusters on Hetzner infrastructure for various customers — on Hetzner Cloud servers, on bare metal servers, and on mixtures of both. Along the way, I have gathered a lot of experience with what works well on Hetzner and where the pitfalls are: MTU values that differ between cloud networks and vSwitches, bare metal nodes that don't report their private IPs, or a cloud controller manager that cannot handle vSwitch routes yet.

With every new cluster, the same networking questions come up again: which IP ranges to use, how to get bare metal servers into the cloud network, and which of the many kubespray variables actually matter. This article is my personal reference for exactly that — a network layout and a kubespray configuration for a Kubernetes cluster running on Hetzner Cloud servers, on Hetzner bare metal servers, or on both.

Two goals and one assumption guided the decisions below:

The configuration should work for cloud servers as well as for bare metal servers.
It should be prepared for native routing. Hetzner currently does not support routing pod networks to bare metal servers, so mixed clusters still need an overlay network. But as soon as that support arrives, switching over should be a matter of changing only a few variables.
Even if Hetzner bare metal servers are used, we assume that a Hetzner Load Balancer is used for ingress.

Network Configuration

Everything starts in the Hetzner Cloud Console:

Create a Hetzner Cloud project.
Add a network with an IP range, e.g., 10.0.0.0/16.
Add a subnet for the load balancer(s), e.g., 10.0.0.0/24.
Add a subnet for the cloud servers, e.g., 10.0.1.0/24.
If bare metal servers are involved:
- Create a vSwitch and add the bare metal servers to it.
- Add a subnet for the bare metal servers, e.g., 10.0.2.0/24, and connect it to the vSwitch you just created.

Kubespray Configuration

IP Ranges

The pod and service networks are placed inside the IP range of the Hetzner network. That is not required for an overlay network, but it is a precondition for native routing later on, since Hetzner can only route IP ranges that belong to the network itself.

## --- k8s_cluster/k8s-cluster.yml
kube_service_addresses: 10.0.64.0/18
kube_pods_subnet: 10.0.128.0/18

Cloud Controller Manager

To integrate the cluster with the Hetzner API, a cloud controller manager is needed. I prefer to install it myself using the upstream Helm chart rather than relying on the version bundled with kubespray, so kubespray is told to prepare the cluster for an external cloud controller manager without deploying one:

## --- all/all.yml
cloud_provider: external
external_cloud_provider: manual

Container Network Interface

Cilium is my default choice. It is a safe long-term bet: Cilium graduated within the CNCF in 2023, at that time the second most active CNCF project in terms of commits. And it implements the whole data path in eBPF instead of iptables, which is what makes it possible to drop kube-proxy entirely further down (Kubernetes Without kube-proxy):

## --- k8s_cluster/k8s-cluster.yml
kube_network_plugin: cilium
kube_owner: root

Cilium's init containers run as UID 0, but with all capabilities dropped, including CAP_DAC_OVERRIDE — which means that even root is subject to the regular file permission checks. Therefore, kube_owner must be set to root for Cilium.

Network MTU

Cilium needs to know the MTU of the underlying network, and Hetzner uses two different values here:

1450 bytes inside a Hetzner cloud network
1400 bytes for Hetzner vSwitches

Two things are worth knowing about cilium_mtu. First, it refers to the MTU of the underlying network, not to the one the pods will end up with — Cilium subtracts the encapsulation overhead (50 bytes for VXLAN) on its own. Second, it is a cluster-wide setting, which means the smallest MTU on any path wins:

cloud servers only: 1450
bare metal servers involved: 1400, for every node, including the cloud ones

Routing

As long as Hetzner does not route pod networks to bare metal servers, pod traffic has to be encapsulated, so Cilium runs in VXLAN mode. Hetzner states this requirement themselves: "Using the routing feature of private networks is not supported, so this requires a CNI plugin with encapsulation methods, such as Cilium with routing mode tunnel" (Attach Load Balancers to Robot Private IPs). The native routing CIDR is prepared already, but only takes effect once the tunnel is switched off:

## --- k8s_cluster/k8s-net-cilium.yml
cilium_mtu: 1400
cilium_tunnel_mode: vxlan
cilium_native_routing_cidr: "{% if cilium_tunnel_mode == 'disabled' %}{{ kube_pods_subnet }}{% endif %}"

Once Hetzner supports routing pod networks to bare metal servers, setting cilium_tunnel_mode: disabled and re-enabling the route controller of the cloud controller manager (see below) is all it takes — the routes inside the Hetzner network are then maintained for you. cilium_mtu stays at 1400, as the vSwitch remains the limiting factor; dropping the VXLAN header simply gives the pods the full 1400 bytes instead of 1350.

kube-proxy

Cilium can take over the work of kube-proxy and implement Kubernetes services in eBPF instead of iptables or IPVS, so there is no reason to keep kube-proxy around as well:

## --- k8s_cluster/k8s-net-cilium.yml
cilium_kube_proxy_replacement: true

## --- k8s_cluster/k8s-cluster.yml
kube_proxy_remove: true

Cluster Applications

Two Hetzner components complete the setup and are installed into the cluster afterwards:

hcloud-cloud-controller-manager — the cloud controller manager kubespray deliberately left out above
csi-driver — for provisioning Hetzner Cloud volumes as persistent volumes

For a cluster containing bare metal servers, the cloud controller manager needs two adjustments. Give it the network (HCLOUD_NETWORK), so that it can add bare metal nodes to a Load Balancer by their private vSwitch IP, but switch off its route controller (HCLOUD_NETWORK_ROUTES_ENABLED=false, plus networking.enabled: false in the Helm chart), as it cannot deal with vSwitches yet. This is also the switch to flip once Hetzner supports routing pod networks to bare metal servers.

Note that the bare metal nodes need an InternalIP for this to work: Hetzner does not report vSwitch IPs automatically, so the address has to be passed to the kubelet via --node-ip (in kubespray, that is the ip variable of the according inventory host).

Solar Energy Management with Home Assistant: Pool Heating

2026-06-15T00:00:00Z

We have a solar power system with a battery, and a pool heated by a heat pump. The obvious goal: use excess solar energy to heat the pool for free. The naive solution — turn the heat pump on when there is surplus, turn it off when there isn't — sounds right but creates a problem in practice. Heat pumps are not fans of short cycles. Starting and stopping repeatedly throughout the day causes excessive wear and reduces efficiency significantly.

What I really wanted was for the heat pump to run once a day, for as long as conditions allow, in the window where solar production is highest. This article describes how I built exactly that in Home Assistant, using template sensors and a small automation.

The Scenario

Solar panels feed into a SolarEdge inverter with a battery
A heat pump heats the pool and draws up to 2500 W at full load
Home Assistant has access to live power sensors and a solar forecast via Solcast
The pool filter pump runs on a schedule (schedule.pool_filter_pump_schedule) — the heat pump should only operate within this window

The key insight: the battery should act as an energy buffer, not a competitor. If PV is producing 4 kW and the house only needs 1 kW, those 3 kW are currently going into the battery. The heat pump can take 2.5 kW of that instead — and as long as the remaining solar forecast for the day is sufficient, the battery will still be fully charged by evening.

The Logic

The heat pump should start when two conditions are simultaneously true:

Current surplus ≥ 2500 W — enough PV power above house consumption to run the heat pump right now, without drawing from the battery or grid.
End-of-day energy balance is positive — the remaining PV forecast for today is enough to cover:
- the projected house load for the rest of the schedule window,
- the heat pump running for the entire remaining schedule window,
- and whatever energy the battery still needs to reach 100% charge.

The heat pump should stop when:

The filter pump schedule ends, or
The projected energy balance turns negative — meaning the remaining PV forecast is no longer sufficient to cover house load, continued heat pump operation, and a full battery charge by end of day.

Once stopped, the heat pump does not restart until the next day, regardless of whether conditions improve. This enforces exactly one heating cycle per day.

Surplus Calculation

There is no direct "house consumption" sensor available. Instead, surplus is derived from the sensors that are available:

surplus = battery_charging − battery_discharging + grid_export − grid_import

This gives the net power currently flowing out of the house system — into the battery or onto the grid. That is exactly the power that could be redirected to the heat pump instead.

When the heat pump is already running, this value stays positive as long as PV covers everything. If it goes negative the heat pump is drawing from storage or the grid, which will also be reflected in the energy balance.

Energy Balance Calculation

All values are in Wh:

energy_balance =
    remaining_pv_forecast × 1000
  − battery_deficit
  − house_power × remaining_daylight_hours
  − heat_pump_power × remaining_schedule_hours

Where:

remaining_pv_forecast comes from sensor.solcast_pv_forecast_prognose_verbleibende_leistung_heute (kWh, multiplied by 1000 to get Wh)
battery_deficit = battery capacity × (1 − state of charge) in Wh
house_power = current house consumption in W, estimated from the live power sensors
remaining_daylight_hours = hours until sunset (sun.sun next_setting) — house consumption continues after the schedule ends, competing for the same remaining PV energy
heat_pump_power = 2500 W if the heat pump is currently off, or 0 W if it is already running (its load is already included in house_power and would be double-counted otherwise)
remaining_schedule_hours = hours until the filter pump schedule turns off

A positive energy balance means: even if the heat pump runs for the full remaining schedule window, there will still be enough solar energy to cover house consumption until sunset and fill the battery completely by end of day.

Implementation

Everything lives in a single Home Assistant package file, which keeps the helper, sensors, and automations together and makes the setup self-contained.

Entities Used

Entity	Purpose
`sensor.power_solar_generation`	Live PV production (W)
`sensor.power_grid_import` / `sensor.power_grid_export`	Grid power flow (W)
`sensor.power_battery_charging` / `sensor.power_battery_discharging`	Battery power flow (W)
`sensor.solaredge_battery1_size_max`	Battery capacity (Wh)
`sensor.solaredge_battery1_state_of_charge`	Battery SoC (%)
`sensor.solcast_pv_forecast_prognose_verbleibende_leistung_heute`	Remaining PV forecast today (kWh)
`sun.sun`	Sunset time (for remaining daylight)
`schedule.pool_filter_pump_schedule`	Allowed operating window
`climate.pool_heat_pump`	Heat pump control (mode `heat` / `off`)

Template Sensors

Five template sensors expose the intermediate values, making it easy to see what the automation is "thinking" in the Home Assistant UI:

sensor.pool_heating_surplus_power — current available surplus in watts:

state: >
  {% set bat_charge  = states('sensor.power_battery_charging')    | float %}
  {% set bat_disch   = states('sensor.power_battery_discharging') | float %}
  {% set grid_export = states('sensor.power_grid_export')         | float %}
  {% set grid_import = states('sensor.power_grid_import')         | float %}
  {{ (bat_charge - bat_disch + grid_export - grid_import) | round(0) | int }}

sensor.pool_heating_energy_balance — net Wh available if the heat pump runs for the rest of the schedule window:

state: >
  {% set remaining_pv      = states('sensor.solcast_pv_forecast_prognose_verbleibende_leistung_heute') | float * 1000 %}
  {% set battery_deficit   = states('sensor.pool_heating_battery_deficit')          | float %}
  {% set remaining_sched_h = states('sensor.pool_heating_remaining_schedule_hours') | float %}
  {% set house_w           = states('sensor.pool_heating_house_power')              | float %}
  {% set heat_pump_w       = 0 if not is_state('climate.pool_heat_pump', 'off') else 2500 %}
  {% if is_state('sun.sun', 'below_horizon') %}
    {% set remaining_daylight_h = 0 %}
  {% else %}
    {% set next_setting = state_attr('sun.sun', 'next_setting') | as_datetime %}
    {% set remaining_daylight_h = [(next_setting - now()).total_seconds() / 3600, 0] | max %}
  {% endif %}
  {{ (remaining_pv - battery_deficit - house_w * remaining_daylight_h - heat_pump_w * remaining_sched_h) | round(0) | int }}

The other three sensors (pool_heating_house_power, pool_heating_battery_deficit, pool_heating_remaining_schedule_hours) feed into the energy balance calculation and are also useful for debugging.

Automations

Start — triggered every 5 minutes and immediately when the schedule activates:

condition:
  - condition: state
    entity_id: schedule.pool_filter_pump_schedule
    state: "on"
  - condition: state
    entity_id: input_boolean.pool_heat_pump_ran_today
    state: "off"
  - condition: state
    entity_id: climate.pool_heat_pump
    state: "off"
  - condition: numeric_state
    entity_id: sensor.pool_heating_surplus_power
    above: 2500
  - condition: numeric_state
    entity_id: sensor.pool_heating_energy_balance
    above: 0
action:
  - action: climate.set_hvac_mode
    target:
      entity_id: climate.pool_heat_pump
    data:
      hvac_mode: heat
  - action: input_boolean.turn_on
    target:
      entity_id: input_boolean.pool_heat_pump_ran_today

Stop — triggered on schedule end and every 5 minutes (to catch the energy balance turning negative):

condition:
  - condition: not
    conditions:
      - condition: state
        entity_id: climate.pool_heat_pump
        state: "off"
  - condition: or
    conditions:
      - condition: state
        entity_id: schedule.pool_filter_pump_schedule
        state: "off"
      - condition: numeric_state
        entity_id: sensor.pool_heating_energy_balance
        below: 0

Using the energy balance as the stop condition (rather than instantaneous surplus) means brief cloud shadows do not trigger a stop: a passing cloud does not change the Solcast day forecast, so the energy balance stays positive and the heat pump keeps running. A stop only fires when the overall day forecast genuinely deteriorates.

Daily reset — clears the input_boolean.pool_heat_pump_ran_today flag at midnight.

Setup

Add the package to configuration.yaml:

homeassistant:
  packages:
    pool_heating: !include packages/pool_heating.yaml

That's it — no manual helper creation required, as the input_boolean is declared inside the package file itself.

TeddyCloud on a Public Server IP with LetsEncrypt

2025-01-10T00:00:00Z

TonieBoxes are a great kind of toy for ~~fathers of~~ young children. It allows them to play music or audio books on their own, just by placing funny little figures on it. It's cute, cuddly, extremely intuitive, and not that technically complex. However, there is a major disadvantage: the device is tied to the manufacturer, who has of course invested a lot of time and money in the development of the Toniebox, and it is legitimate for the manufacturer to make money from it. However, there are enough examples of closed, cloud-based systems whose manufacturers discontinue the platform at some point because it is no longer sufficiently profitable to operate. Luckily, some people successfully reverse engineered the box and were able to modify it in a way, which allows to operate your own content server for this system.

When I first came into contact with their software, TeddyCloud, the first question that arose was whether I should install TeddyCloud locally or on a public server. The decision to go with a public server was an easy one: I wanted my kids to be able to use their TonieBox with all its content (both original and custom) while on vacation, for example. After that, however, the question arose as to how I could secure access. Typically, this includes a properly set up SSL and user authentication. For self-hosted services, LetsEncrypt is typically the first choice for SSL certificates. However, TeddyCloud also sets up its own CA to secure communication with the TonieBox, which uses Client Certificate Authentication.

In this context, the TeddyCloud documentation contains the following statement:

Please beware that port 443 cannot be remapped and you cannot use a reverse proxy like nginx or traffik without passing through the TLS (complex, not recommended). The client certificate authentication needs to be done by teddyCloud. Also, there is no SNI.

Challenge Accepted!

Actually, writing this article, I realized that I never checked if custom SSL certificates (e.g., issued by LetsEncrypt) are supported for the TeddyCloud web interface. Anyway, I also couldn't find any documentation considering authentication for TeddyCloud. So I really wanted to have a reverse proxy, dealing with both, SSL encryption for the web interface with a valid (LetsEncrypt) certificate, and authentication (I decided to just use plain old Basic Auth).

Usually, a reverse proxy either just forwards TCP, or it actually does SSL Offloading by decrypting the encrypted HTTPS traffic and only forwarding the HTTP part to the upstream application. However, since TonieBoxes do client certificate authentication and the TeddyCloud wants to do it by itself to identify the TonieBox, SSL Offloading by the reverse proxy was no option. On the other hand, I somehow had to be able to set a switch if a connection (from a TonieBox) should be forwarded to the TeddyCloud port 443, or if it (from a browser) should end up with the web interface, encrypted with a LetsEncrypt certificate.

Luckily, nginx offers a nice option for this: With a stream block, I can basically do TCP forwarding, but with the ssl_preread directive I'm able to somehow peek into the connection and to check if, and which server name is indicated for the SSL connection.

Also, there is no SNI.

Perfect! This saved my day! So, if it's a request with SNI, I do the SSL Offloading stuff with my LetsEncrypt certificate. If there is no SNI, I just forward it to TeddyCloud as is. Works for me :)

Setup

In case you want to use it by yourself, here is the relevant config!

To get things running, do the following steps:

Set up a server with an OS of your choice, install Docker engine and configure your desired domain name to the IP(s) of that server. I also would suggest setting up a firewall and only allowing ports 22 (SSH), 80 (HTTP) and 443 (HTTPS).
Copy the configs to your server (docker-compose.yaml and nginx.conf should be in the same directory) and adjust all occurences of the domain name teddycloud.example.com to your domain.
Comment out the HTTPS server section in nginx.conf, as we don't have the LetsEncrypt certificates yet.
Start the Docker containers, e.g., using docker compose up -d.
Request a LetsEncrypt certificate by running docker compose exec -it certbot certbot certonly --webroot. When you're asked for the webroot directory, please provide /var/www/certbot.
Uncomment the HTTPS server configuration in nginx.conf.
Restart nginx, e.g., using docker compose restart nginx.

Sync Zotero Library to Nextcloud

2022-07-18T00:00:00Z

For my research, I'm using Zotero for bibliography management. It's free, it's great, and it fits perfectly for my needs. In this blog post, I show how to configure Zotero to synchronize your Zotero Library to Nextcloud.

For some of the research papers I've read during my research activities, there are several (mostly, but not always, similar) versions, for some other papers it is very difficult to find the document. Therefore, I decided to always keep a digital copy of the document I've just read, just to ensure to be able to access the exact same version I have accessed before. Zotero allows for attaching files to an entry, and furthermore allows for synchronizing the library as all as the attachments. While synchronizing the bibliography entries meta data (authors, title, ...) seems to be free and unlimited, only 300MB of document storage are for free per account. Zotero offers paid plans to increase the storage limit, or to use own, WebDAV based, storage.

Since I have a running Nextcloud instance with WebDAV support, I decided to use my Nextcloud for the synchronization. Actually, it is quite easy to configure it accordingly, however, I spent some time on finding that out and there are also some open posts in the Zotero forums, therefore I'm going to document my solution here.

Configuring the Synchronization of your Zotero Library to Nextcloud

First, we need to create a folder in Nextcloud. Please note that Zotero requires the path to end with zotero. Also consider if you want to use your global Nextcloud credentials (which I don't recommend to do) or to create a dedicated shared folder for this, which will provide you with extra credentials just for this purpose. Since the name of the folder configured to be shared does not show up in the URL, within the shared folder there has to be the zotero folder containing the actual synchronized attachments.

In my Nextcloud instance, I created a folder PhD/Zotero/zotero and configured and configured the directory PhD/Zotero to be accessible and editable using a link. The link then should look like this:

https://nextcloud.example.com/s/1337R4nd0mSh4r3S3cret

Now, in Zotero client, configure Sync (Edit -> Preferences -> Sync) as follows: Set File Syncing mode to WebDAV, as URL put nextcloud.example.com/public.php/webdav, and as username as well as password use the sharing secret (the last part of the URL). That should be it.

Update

The URL nextcloud.example.com/public.php/webdav is correct when using a sharing secret for the credentials. When using the actual account username and password, the URL is nextcloud.example.com/remote.php/webdav.

Controlling a TOLO Steam Bath with Home Assistant

2021-12-15T00:00:00Z

This one started, as many of my side projects do, with a piece of hardware sitting in front of me and the nagging question: why can't I control this from Home Assistant?

A while ago I built a steam bath together with a member of my family. The steam generator we used came from TOLO (a brand of Steamtec), and it shipped with a so-called App Control Box — a little network module that lets you operate the steam bath from a smartphone app instead of just the wall panel. Temperature, humidity, the lamp, the fan, aroma therapy, the salt bath nebulizer: all of it controllable from your phone. Nice. But of course I didn't want yet another app on my phone — I wanted it in Home Assistant, next to everything else in the house. And later, when I built my own house and put a steam bath into it for my own family, having proper home automation control was no longer a nice-to-have, it was the whole point.

There was just one problem: there is no public API, no documentation, and no official integration. So I had to build one.

Reverse Engineering the Protocol

The App Control Box talks to the smartphone app over the local network, so the obvious first step was to find out how. I put the box and my phone on the same network, captured the traffic while clicking through the app, and started staring at packets.

A few things became clear quickly:

The communication is based on UDP datagrams, not TCP. The app sends a small request packet, the device answers with a response packet.
Everything happens on a fixed port (51500).
The protocol is refreshingly simple: each message is a short, fixed-structure byte sequence — a command byte, a couple of arguments, and (for status/settings responses) a payload that encodes the current state of the device.
There is a tiny keep-alive mechanism (a single byte) to let the device know someone is still listening.

From there it was the usual reverse-engineering grind: toggle one setting in the app, watch which byte in the packet changes, write it down, repeat. Bit by bit the meaning of every field fell into place — target temperature (35–60 °C), target humidity (60–99 %), the various timers (power, fan, salt bath), lamp mode, aroma therapy slot, and a whole set of read-only status values like the current water tank temperature, water level and valve state.

tololib

Once I understood the protocol, I wrapped it in a proper Python library so I'd never have to think about raw bytes again: tololib.

tololib gives you a clean client object that hides all the datagram juggling. Reading the current state of the device is as simple as:

from tololib import ToloClient

client = ToloClient("192.168.1.100")

status = client.get_status()
print(status.power_on, status.current_temperature, status.current_humidity)

settings = client.get_settings()
print(settings.target_temperature, settings.target_humidity)

# turn the steam bath on
client.set_power_on(True)

The library also ships with a command line interface, so you can poke at a device straight from the terminal without writing a single line of Python — handy for debugging and for figuring out whether a problem is in the hardware, the network, or the software on top.

One detail I'm particularly happy with: tololib includes a device simulator. Since the protocol is just UDP request/response, I could implement a fake TOLO device that speaks the same protocol. That meant I could write and run the entire test suite — and later develop the Home Assistant integration — without needing the actual steam bath powered up and reachable. Continuous integration doesn't have a steam bath in the rack, after all.

The library is published on PyPI under the MIT license, and the API documentation lives here.

Legal note: this is a community project. Neither tololib nor I have any professional affiliation with the companies behind TOLO. I just wanted to control my own steam bath.

The Home Assistant Integration

With a stable library doing the heavy lifting, building the TOLO integration for Home Assistant became the fun part. tololib is the core; the integration is a relatively thin layer mapping the device's capabilities onto Home Assistant entities:

a climate entity for the steam bath itself, with the heat and dry operating modes, target temperature and target humidity;
a light entity for the RGB lamp, including its automatic colour-fading mode;
a fan entity for the ventilation, with its timer;
switches for aroma therapy and the salt bath nebulizer;
and a set of sensors and binary sensors exposing the diagnostics — tank temperature, water level, timers and valve state.

Because the integration communicates with the device entirely over the local network, it runs as a local polling integration: no cloud, no account, no external dependency. Discovery makes setup painless — in most cases Home Assistant finds the App Control Box on the network and you just confirm.

I contributed the integration to Home Assistant core, and after the usual (very thorough, and very helpful) review process, it shipped with the Home Assistant 2021.12 release. Seeing your own little reverse-engineering hobby project become a one-click integration that anyone with the same hardware can use is a genuinely good feeling.

Wrapping Up

If you happen to own a TOLO / Steamtec steam bath with an App Control Box, you can add it straight from the Home Assistant integrations UI — no extra setup required. And if you just want to talk to one of these devices from your own Python code or from the command line, grab tololib from PyPI.

Found a bug, or missing a feature? Open an issue — contributions are very welcome.

GPG Agent for SSH Authentication (Update)

2021-06-14T00:00:00Z

In my last post regarding the usage of GPG Agent for SSH Authentication (read here) I presented my first solution to replace the default OpenSSH Agent with GPG's SSH Agent support. With the update to Ubuntu 21.04 I had to reconfigure this, since the current way stopped working.

Prequisites

The prequisites are almost the same as in my last article. For this article, I assume that a GPG key pair is available and working (e.g. a YubiKey or a file based key pair) for signing and encryption. Furthermore, since I'm still using Ubuntu (now in version 21.04), this tutorial is most probably specific to systems running Ubuntu 21.04. However, I guess the most parts of this tutorial can be transported to other Linux operating systems as well.

Setup GPG Agent for SSH Authentication

Actually, after I upgraded to Ubuntu 21.04, I found that there is a much simpler way to get GPG Agent for SSH Authentication running as I have desribed in my last article related to this topic. This time, just three simple steps are enough:

First, we need to enable the GnuPG agent: echo "use-agent" >> ~/.gnupg/gpg.conf
Second, we need to tell the GnuPG agent also to enable support for SSH: echo "enable-ssh-support" >> ~/.gnupg/gpg-agent.conf
Third, we need to inform SSH to use the GnuPG provided SSH agent by manually configuring the IdentityAgent (OpenSSH documentation). In my case (with Ubuntu 21.04), the SSH auth socket created by GnuPG agent was located at /run/user/1000/gnupg/S.gpg-agent.ssh: echo "IdentityAgent /run/user/1000/gnupg/S.gpg-agent.ssh" >> ~/.ssh/config

Optional Steps

Currently, by default, also the Gnome Keyring based SSH agent is starting (at least on my system). In order to disable, I had to uncheck SSH Key Agent in the gnome-session-properties applet.

Helm Charts for Hetzner Cloud

2020-08-23T00:00:00Z

Update: The Helm charts presented in this article are superseded by the official Helm charts provided by Hetzner, available from the repository https://charts.hetzner.cloud (hcloud-cloud-controller-manager, hcloud-csi). Please use those for new clusters. For an up-to-date network setup of Kubernetes clusters on Hetzner, see Kubernetes Networking on Hetzner with Kubespray.

Recently, Hetzner announced the launch of load balancers for the Hetzner Cloud. Thanks to this, it is now possible to realize highly available Kubernetes clusters without MetalLB and custom scripting for Floating IP assignment. For automatic cloud management, Hetzner provides the hcloud-cloud-controller-manager, which is also able to create and manage these load balancers, and the hcloud-csi-driver, which enables cloud volume integration into your Hetzner Cloud Kubernetes cluster.

Unfortunately, Hetzner only provides manifest files for installing these into your cluster. Since, in my opinion, Helm is the most convenient way of installing and managing applications in Kubernetes, I created Helm charts for both of them.

hcloud-cloud-controller-manager

The cloud controller manager integrates the Kubernetes cluster with the Hetzner Cloud API: it initializes nodes with their Hetzner Cloud metadata, removes nodes whose servers have been deleted, manages routes in Hetzner Cloud networks, and creates Hetzner Cloud Load Balancers for services of type LoadBalancer.

The Helm chart for hcloud-cloud-controller-manager is available at https://gitlab.com/MatthiasLohr/hcloud-cloud-controller-manager-helm-chart.

hcloud-csi-driver

The CSI driver allows Kubernetes to provision Hetzner Cloud volumes as persistent volumes, and to attach them to the cloud server the pod is scheduled on.

The Helm chart for hcloud-csi-driver is available at https://gitlab.com/MatthiasLohr/hcloud-csi-driver-helm-chart.

Legal Notice: I'm not affiliated with Hetzner in any way. In this blog, I just present my experiences and learnings working with different technologies and providers, such as Kubernetes clusters in the Hetzner environment.

Kubernetes Cluster on Hetzner Bare Metal Servers

2020-06-29T00:00:00Z

If you want to run your own Kubernetes Cluster, you have plenty of possibilities: You can set up a single node cluster using minikube locally or on a remote machine. You can also set up a multi node cluster on VPS or using managed cloud providers such as AWS or GCE. Alternatively, you can use hardware, e.g. Raspberry Pis or bare metal servers. However, without the functionality provided by a managed cloud provider, it is difficult to take full advantage of the complete high availability capabilities of Kubernetes. We have tried - and present here the instructions for a highly available Kubernetes cluster on Hetzner bare metal servers.

Why Bare Metal?

GCE and AWS are very expensive, especially when your cluster is growing. Raspberry Pis are cheap, but also quite limited regarding resources. Usual VPS providers lack support of High Availability, and a single node cluster is per definition a whole Single-Point-of-Failure - nice for testing, but not for production.

It is also possible to set up a Kubernetes cluster on bare metal. In this case, implementation of High Availability depends on the features the colocation provider is offering. Somehow, you have to create a machine-independend load balancer, which redirects traffic to working nodes and ignores broken nodes.

Dorian Cantzen (extrument.com) and I have taken up that challenge. In this article, we will describe the steps to set up a production ready Kubernetes cluster on Hetzner bare metal servers using the Hetzner vSwitch feature.

Important notice: Neither the author of this article (Matthias Lohr) nor Dorian Cantzen are affiliated with Hetzner nor paid for writing this article. This article is a technical report, initiated by Matthias Lohr, summarizing the findings when trying to find a feasible solution for a highly available Kubernetes cluster on bare metal machines. We found Hetzner as part of one possible solution, most probably there are more eligible providers out there for a similar solution.

The High Availability Challenge

The basic goal of a server cluster is reliability in terms of high availability and fault-tolerance. If one component of a cluster fails, cluster logic will automatically use another component for the task.

Generally, you can devide cluster components in three categories: computational resources, storage resources and networking. The core component of Kubernetes is a scheduler for computational loads (Pods), which provide services such as web portals etc. Ceph clusters or Kubernetes-based solutions like Rook provide redundant storage. High Availability for the remaining part, the networking, requires special support by the colocation provider. Usually, when using mainstream hosters like Hetzner, a server has a single NIC with a single IP (ok, one IPv4 and one IPv6) address. Typically, a DNS record points to one or multiple IP addresses. However, when pointing to multiple IPs, and the server behind this IP is not available, the user will get connection errors. So, DNS can't help to provide a solution here. What we need is an IP address which can be shared between multiple servers.

We found that using Hetzner vSwitches, it is possible to route IPs or IP subnets into a VLAN (IEEE 802.1Q) where each server can be connected. It’s then up to the servers to decide which one should reply to incoming traffic for this/these IP(s). IP migrations can be done completely within the cluster servers, without notifying an external API.

Setting up a Kubernetes Cluster on Hetzner Bare Metal Servers

Create VLAN (Hetzner vSwitch)

First, we have to create the VLAN, which connects the servers. You can do that in the Hetzner vSwitch configuration area

After the vSwitch is created, you have to assign the servers you want to add to your Kubernetes cluster to the vSwitch:

On the IPs tab, you can order additional IPs or IP subnets, which are routed to the vSwitch and therefore not assigned to a single server. We will use MetalLB to manage these IP address(es).

Now, you should set up the servers with your favorite OS capable of running Kubernetes. After the standard setup has finished, we need to configure the VLAN and the additional IPs. According to the official Hetzner documentation, you have to create a virtual network interface with VLAN taggings. But since you want to use the IPs within the Kubernetes cluster, you have to add some additional ip rules.

Below you will find an example for a working netplan configuration. This configuration uses 10.233.255.0/24 as internal network range for cluster internal communication and 321.321.321.32/28 as subnet assigned to the vSwitch. 10.233.0.0/18 is the default service IP range used by kubespray, 10.233.64.0/18 the according default Pod IP range.

network:
  version: 2
  vlans:
    # Configure vSwitch public
    enp4s0.4000:
      id: 4000
      link: enp4s0
      mtu: 1400
      addresses:
        - 10.233.255.1/24
      routes:
        - to: 0.0.0.0/0
          via: 321.321.321.33
          table: 1
          on-link: true
      routing-policy:
        - from: 321.321.321.32/28
          to: 10.233.0.0/18
          table: 254
          priority: 0
        - from: 321.321.321.32/28
          to: 10.233.64.0/18
          table: 254
          priority: 0
        - from: 321.321.321.32/28
          table: 1
          priority: 10
        - to: 321.321.321.32/28
          table: 1
          priority: 10

Alternatively, you can use our Hetzner vSwitch ansible role which we developed during our experiments.

Test if the VLAN works properly by try to ping all nodes using their private IP addresses (10.233.255.1, 10.233.255.2, ...).

Setup Kubernetes

Now, since the networking stuff is up and running, you are ready to install Kubernetes. We did that using Ansible/kubespray, which offers a quite convenient and production ready solution for managing bare metal Kubernetes clusters. Use the server's internal IP addresses in your inventory.

# example inventory
[all]
node1 ip=10.233.255.1 etcd_member_name=etcd1
node2 ip=10.233.255.2 etcd_member_name=etcd2
node3 ip=10.233.255.3 etcd_member_name=etcd3

Before installing the network plugin, ensure that you set the right MTU for your Kubernetes networking plugin. Hetzner vSwitch interfaces have a MTU of 1400. When using e.g. Calico, which has a 20 bytes overhead, you need to set the MTU for Calico to 1380.

Install and configure MetalLB to use the IP/subnet assigned to the Hetzner vSwitch. Please ensure, that you do not configure the whole subnet, but exclude the two first and the last IP address.

Example

subnet assigned: 321.321.321.32/28

IPs in subnet: 321.321.321.32 - 321.321.321.47

subnet address (not usable): 321.321.321.32

subnet gateway address (used by Hetzner): 321.321.321.33

subnet broadcast address (not usable): 321.321.321.47

remaining usable IPs/MetalLB range: 321.321.321.34 - 321.321.321.46

That's it! Now you can start using Kubernetes Services with type LoadBalancer and one of the usable IP addresses to get traffic into your cluster. MetalLB will care about assigning these IP addresses to working nodes. If one node goes down, MetalLB will reassign the IP address to a working node.

The only bottleneck we didn't figure out yet: Does Hetzner have a redundant (highly available) setup for the vSwitches...?

Maintenance of Long-Living Smart Contracts

2020-03-02T00:00:00Z

Last week at the 7th Collaborative Workshop on Evolution and Maintenance of Long-Living Systems, together with Sven Peldszus, I presented a paper regarding the Maintenance of Long-Living Smart Contracts.

Lohr, M., & Peldszus, S. (2020). Maintenance of long-living smart contracts. CEUR Workshop Proceedings, 2581.

Take-Aways

After the presentation, we had a very interesting discussion about Software Security (as everybody desires to have) and Software Immutability (as the Ethereum blockchain promotes). The essence of the discussion was that there is a need of further investigation about this conflict and how it can be solved.

As outcome of the presentation and the discussion, my best idea would be some kind of smart contract update support provided by Ethereum. For example, this could be done by introducing a successor field when a smart contract selfdestruct method is called. The successor field should then contain the address of the updated, succeeding smart contract.

Gitlab on a Diskstation

2019-12-29T00:00:00Z

Sometimes, regardless of the possibilities offered by "the cloud", you want to host important services yourself. For me as a software and DevOp engineer, this applies to my source code. For this reason, I host my GitLab instance myself. Since the GitLab package for DSM provided by Synology is outdated, I will explain here how to install the latest version of GitLab on a DiskStation using Docker.

Preparations

Before we can install GitLab on a DiskStation, we need to make some preparations:

Install the Docker package from the Synology Package Center.
Create a Shared Folder for GitLab. For this article, I assume that the Shared Folder is named gitlab and is created on the first volume, so its path on the filesystem will be /volume1/gitlab.
Activate DSM SSH access. You can also activate Public Key Authentication for DiskStation SSH access following this tutorial.

Setup GitLab on a DiskStation

Now we are going to install GitLab. To do so, we need to download the latest docker image of GitLab first. Open the Docker App in DSM, select Registry in the left menu and download the latest gitlab/gitlab-ce image.

You can watch the download progress in the menu on the left in Image. As soon as the download is complete, a new container can be created with Launch in the upper left corner.

Before we click on Next, we have to do some configuration in the Advanced Settings dialog.

Volumes

We need to persist three directories of GitLab:

/etc/gitlab - configuration directory
/var/opt/gitlab - user-generated content (repositories, database, ...)
/var/log/gitlab - logfiles

We do this by defining directory volumes. This is done by mapping subdirectories of the previously created shared directory into the container by defining a mount path.

Port Settings

To allow access to the GitLab instance from outside, we need to define port bindings. These port bindings will forward DiskStation host ports to the GitLab Docker container.

After applying all changes and starting the container, it will take a couple of minutes for GitLab to bootstrap the instance. After a while, you should be able to access your new GitLab instance at http://:8080.

On this screen, you can set the password for the root user of GitLab.

Final Configuration

Before your GitLab instance is ready for usage, we have to finalize the configuration. Use the DSM Docker application for shutting down the container.

When the container is off, connect to your DiskStation via SSH. Open the GitLab configuration file /volume1/gitlab/config/gitlab.rb and adjust (at least) the following configuration:

external_url - Set to the URL you want to use for accessing GitLab.
gitlab_rails['gitlab_shell_ssh_port'] = 7999 - Use port 7999 for Git via SSH, since port 22 is already taken by your DiskStation.

Further Considerations

External Access

You have to decide how to configure external access to your GitLab instance. The simplest option might be to configure a port forwarding from your router to the GitLab ports on your DiskStation. Another possibility is to use DSM's reverse proxy.

HTTPS Encryption

Both, DSM and GitLab have built-in support for requesting LetsEncrypt certificates for HTTPS. If you're not using DSM's reverse proxy, you should configure LetsEncrypt in your gitlab.rb configuration.

If you're using the DSM reverse proxy, you can still use GitLab's capabilities for getting LetsEncrypt certificates or configure HTTPS offloading (the reverse proxy terminates the HTTPS connection and forwards requests internally using HTTP) and use DSM's capabilities for getting LetsEncrypt certificates. When configuring the hosts, please be sure to enable Websocket Support.

The End

After you finished configuration, use the DSM Docker GUI to start your GitLab container again.

Congratulation! Your GitLab on a DiskStation is now up and running!

For questions, don't hesitate to leave a comment below. For personal support, you can also contact me directly.

2nd IEE International Conference on Blockchain - Paper on Data Genuineness

2019-07-18T00:00:00Z

Yesterday I attended the 2nd IEEE International Conference on Blockchain conference in Atlanta, Georgia. Besides many interesting and exciting lectures, I also presented my first paper there:

https://mlohr.com/publications/

I would also like to say thanks for the interesting conversations I had at the conference.

Check for Docker image updates

2019-02-08T00:00:00Z

For certain docker-driven services, I would like to check regularly for new versions. One way to do this is to query Docker Hub via Cronjob. Here I provide a little bash script to check for Docker image updates on Docker Hub.

#!/bin/bash
# Example usage:
# ./docker-image-update-check.sh gitlab/gitlab-ce update-gitlab.sh

IMAGE="$1"
COMMAND="$2"

echo "Fetching Docker Hub token..."
token=$(curl --silent "https://auth.docker.io/token?scope=repository:$IMAGE:pull&service=registry.docker.io" | jq -r '.token')

echo -n "Fetching remote digest... "
digest=$(curl --silent -H "Accept: application/vnd.docker.distribution.manifest.v2+json" \
    -H "Authorization: Bearer $token" \
    "https://registry.hub.docker.com/v2/$IMAGE/manifests/latest" | jq -r '.config.digest')
echo "$digest"

echo -n "Fetching local digest...  "
local_digest=$(docker images -q --no-trunc $IMAGE:latest)
echo "$local_digest"

if [ "$digest" != "$local_digest" ] ; then
    echo "Update available. Executing update command..."
    ($COMMAND)
else
    echo "Already up to date. Nothing to do."
fi

The script is also available as a GitHub Gist.

Here is a repository with this and some more scripts: https://gitlab.com/MatthiasLohr/omnibus-gitlab-management-scripts

If you have questions or problems, please create a ticket here.

FritzBox LAN 2 LAN VPN with pfSense

2019-02-04T00:00:00Z

In this article I described how to set up a FritzBox LAN 2 LAN VPN with StrongSwan. Meanwhile I replaced Ubuntu on the server with pfSense. Of course I have set up my FritzBOX VPN connections again. So here's a tutorial on how to set up a FritzBox LAN 2 LAN VPN with pfSense.

The prerequisites remain the same in comparison to the StrongSwan instructions:

Register your FritzBox with a DynDNS service (e.g. https://myfritz.net) and find your FritzBox domain name (e.g. myfb.myfritz.net)
Find your FritzBox’ private subnet, typically 192.168.178.0/24
Find (or define) the subnet on the remote site, e.g. 192.168.42.0/24
Find the hostname of the remote site, e.g remote.example.com
Define a secret secret, e.g S3cret123!(no, please do not use that, that’s my secret secret!)

Configure your FritzBox

Last time I presented a large configuration file that had to be imported into the FritzBox to set up the VPN connection. In the meantime I have found which encryption and hashing algorithms the FritzBox uses by default, so that we can simply use the default settings of the FritzBox and therefore the web interface built into FritzOS 7.x:

The pfSense configuration is similarly simple:

Conclusion

In my opinion, it's pretty easy to set up a FritzBox LAN 2 LAN VPN with pfSense. The only hard thing is to figure out the preferred encryption and hashing algorithms supported by the FritzBox.

I have this running now with pfSense 2.4.4 with both a FritzBox 7490 and a FritzBox 7590.

An additional note: Sometimes does a Dual Stack connection not seem to be completely stable. In this case it helps to set Internet Protocol to IPv4 in phase 1.

Ansible Role for tinc VPN

2019-01-28T00:00:00Z

When setting up Kubernetes clusters, it makes sense for the individual nodes of Kubernetes to live in the same private network. If Kubernetes is set up on bare metal machines from suppliers such as Hetzner, it may not necessarily be possible to set up a common network of this kind natively. This is where tinc comes in: it makes it very easy to set up a virtual network across all participating nodes. To keep the configuration of tinc parallel to that of Kubernetes (I use Kubespray for my Kubernetes setup), I developed an Ansible Role for tinc VPN and made it available on GitHub.

Features

Installing and setting up tinc VPN service
In-place private key generation (private keys are never copied)
Support for additional nodes where host machines are not covered by the playbook
Support for custom routes for the VPN interface
Support for joining existing bridge interfaces on the host machine
Custom scripting for up/down hook scripts

For setup instructions or a tutorial how to use my Ansible Role for tinc VPN please check the README. It always contains the up-to-date instructions for using this role and will be updated, if new features come up.

FritzBox LAN 2 LAN VPN with StrongSwan

2019-01-19T00:00:00Z

There are a lot of instructions available on how to connect your FritzBox to a server via VPN. But since it took me a long time to find a working tutorial myself, here again a post describing how to set up a FritzBox LAN 2 LAN VPN with StrongSwan (based on the site https://seffner-schlesier.de/news/ipsec-zwischen-avm-fritzbox-und-strongswan/).

Prerequisites

Register your FritzBox with a DynDNS service (e.g. https://myfritz.net) and find your FritzBox domain name (e.g. myfb.myfritz.net)
Find your FritzBox’ private subnet, typically 192.168.178.0/24
Find (or define) the subnet on the remote site, e.g. 192.168.42.0/24
Find the hostname of the remote site, e.g remote.example.com
Define a secret secret, e.g S3cret123!(no, please do not use that, that’s my secret secret!)

Configure your FritzBox

You can configure FritzBox VPN connections via the web interface, but some parameters seem to be set there, which are not easily accepted on the remote side. Therefore you have to create the following configuration file locally and import it into your FritzBox (replace the example values):

vpncfg {
  connections {
    enabled = yes;
    editable = no;
    conn_type = conntype_lan;
    name = "remote.example.com";
    boxuser_id = 0;
    always_renew = yes;
    reject_not_encrypted = no;
    dont_filter_netbios = yes;
    localip = 0.0.0.0;
    local_virtualip = 0.0.0.0;
    remoteip = 0.0.0.0;
    remote_virtualip = 0.0.0.0;
    remotehostname = "remote.example.com";
    keepalive_ip = 0.0.0.0;
    localid {
      fqdn = "myfb.myfritz.net";
    }
    remoteid {
      fqdn = "remote.example.com";
    }
    mode = phase1_mode_idp;
    phase1ss = "all/all/all";
    keytype = connkeytype_pre_shared;
    key = "S3cret123!";
    cert_do_server_auth = no;
    use_nat_t = yes;
    use_xauth = no;
    use_cfgmode = no;
    phase2localid {
      ipnet {
        ipaddr = 192.168.178.0;
        mask = 255.255.255.0;
      }
    }
    phase2remoteid {
      ipnet {
        ipaddr = 192.168.42.0;
        mask = 255.255.255.0;
      }
    }
    phase2ss = "esp-all-all/ah-none/comp-all/pfs";
    accesslist = "permit ip any 192.168.42.0 255.255.255.0";
  }
  ike_forward_rules = "udp 0.0.0.0:500 0.0.0.0:500",
  "udp 0.0.0.0:4500 0.0.0.0:4500";
}

Configure StrongSwan

/etc/ipsec.conf:

config setup
conn %default
 left=remote.example.com
 leftsubnet=192.168.42.0/24
 authby=secret
 auto=start

conn fb
 ike=aes256-sha-modp1024
 esp=aes256-sha1-modp1024
 right=myfb.myfritz.net
 rightid=@myfb.myfritz.net
 rightsubnet=192.168.178.0/24
 ikelifetime=3600s
 keylife=3600s

/etc/ipsec.secrets:

@remote.example.com @myfb.myfritz.net : PSK "S3cret123!"

Software/Hardware versions

I have successfully connected a FritzBox 7430 as well as a FritzBox 7590 with FritzOS 7.01.

On the server side Ubuntu 18.04 is running with StrongSwan 5.6.2.

I hope you will also successfully set up your FritzBox LAN 2 LAN VPN with StrongSwan! Good luck!

Websockets for Synology DSM

2019-01-15T00:00:00Z

It's happened to me several times now that an application I run on my DS 1817+ has problems with websockets. This is because I use the reverse proxy built into DSM, which does not support websockets by default. For this reason, here's a little tutorial on how to enable Websockets for Synology DSM reverse proxy.

Enable Websockets in DSM Reverse Proxy

Actually, it is extremely easy to enable Websockets for Synology DSM reverse proxy:

Open Control Panel > Application Portal
Change to the Reverse Proxy tab
Select the proxy rule for which you want to enable Websockets and click on Edit
Change to the Custom Headers tab
Add two entries in the list:
- Name: "Upgrade", Value: "$http_upgrade"
- Name: "Connection", Value: "$connection_upgrade"

Repeat these steps for every rule where you want to enable Websockets.

In my local setup, I need this for GitLab Mattermost (running within a docker container) and DSM Virtual Machine Manager Console.

Signing PGP Keys

2018-11-11T00:00:00Z

An essential part of PGP is the mutual validation of key pairs. This confirms that the information about the owner stored in the key corresponds to reality (e.g. ownership of this key). This post describes the process of signing PGP keys with GnuPG.

With GnuPG it is very easy to sign foreign public keys. GnuPG offers a selection of options to configure the creation of the signature. I will introduce the most important ones here.

Copy&Paste

Here the Copy&Paste command (because people usually just take the first code they see):

gpg --ask-cert-level --ask-cert-expire --sign-key <fingerprint of key to be signed>

Please read below to understand what you are doing!

Simple Signing

The fastest way to generate a signature for a key is the following command (please do not use that!):

This command creates a signature that says nothing about whether and how you verified the identity of the owner of the key. In practical terms, the signature is therefore worthless. In addition, the expiration date of the signature is set to the same date as the key. This is not wrong for now, but there may be cases where you want to explicitly define a different date.

Define your key pairs to be used

First of all, if you have several keys, it is important to define exactly which key you want to use for the signature. You can do this in GnuPG with the parameter -u . To avoid mistakes, you should use the fingerprint of the matching key pair.

gpg -u <your key fingerprint> --sign-key <fingerprint of key to be signed>

Define certification level

The certification level indicates how it was verified that the key actually belongs to the registered owner (name and e-mail address). The following levels are available:

(0) I will not answer. (default) (1) I have not checked at all. (2) I have done casual checking. (3) I have done very careful checking.

Level 3 describes a thorough check with personal meeting and ID, Level 2 e.g. a check by telephone (if you know the voice). To specify the level of the check, you need to add the --ask-cert-level parameter:

gpg -u <your key fingerprint> --ask-cert-level --sign-key <fingerprint of key to be signed>

Define certification expiry time

A PGP key pair can (should) have an expiration date. This ensures that old keys that are no longer in use become unusable. Furthermore, it is possible to change the expiration time of the key pair if necessary, i.e. to extend it (please read OpenGPG Best Practices for reasons why you should do that). If you sign a key, the expiration date of the signature is set to the current expiration date of the key. If you now change the expiration date of the key, the date of the signature is not adjusted (a new signature would have to be created for this). In order to save the need for a new signature (and the corresponding effort for validation), the signature can be given a different expiration time than the key when it is created. GnuPG provides the parameter --ask-cert-expire for this:

gpg -u <your key fingerprint> --ask-cert-level --ask-cert-expire --sign-key <fingerprint of key to be signed>

However, even with signatures, you should consider carefully whether the expiration date is "never" really justified.

Get started!

Now you can start signing! Meet your friends, acquaintances and colleagues and sign each other your keys.

By the way, you can find my Publiy Key here.

Docker MTU issues and solutions

2018-10-16T00:00:00Z

If you want to use Docker on servers or virtual machines, technical limitations can sometimes lead to a situation in which - even without intentional limitation - it is not possible to access the outer world from a docker container.

Docker MTU configuration

A common problem when operating dockers within a virtualization infrastructure is that the network cards provided to virtual machines do not have the default MTU of 1500. This is often the case, for example, when working in a cloud infrastructure (e.g. OpenStack). The Docker Daemon does not check the MTU of the outgoing connection at startup. Therefore, the value of the Docker MTU is set to 1500.

Detecting the problem

With the command ip link you can display the locally configured network cards and their MTU:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
  link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1454 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
  link/ether aa:bb:cc:dd:ee:ff brd ff:ff:ff:ff:ff:ff
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default
  link/ether uu:vv:ww:xx:yy:zz brd ff:ff:ff:ff:ff:ff

If the outgoing interface (in this case ens3) has an MTU smaller than 1500, some action is required. If it is greater than or equal to 1500, this problem does not apply to you.

Solving the problem (docker daemon)

To solve the problem, you need to configure the Docker daemon in such a way that the virtual network card of newly created containers gets an MTU that is smaller than or equal to that of the outgoing network card. For this purpose create the file /etc/docker/daemon.json with the following content:

{
 "mtu": 1454
}

In this example, I chose 1454 as the value, as this corresponds to the value of the outgoing network card (ens3). After restarting the Docker daemon, the MTU of new containers should be adapted accordingly. However, docker-compose create a new (bridge) network for every docker-compose environment by default.

Solving the problem (docker-compose)

If you work with docker-compose, you will notice that in containers created by docker-compose, the MTU of the daemon is not inherited. This happens because the mtu entry in /etc/docker/daemon.json file only affects the default bridge. Therefore you have to specify the MTU explicitly in the docker-compose.yml for the newly created network:

...
networks:
 default:
  driver: bridge
  driver_opts:
   com.docker.network.driver.mtu: 1454

After rebuilding the docker-compose environment (docker-compose down; docker-compose up), the containers should use the modified MTU.

I personally don't like this solution, because the docker-compose files have to be specially adapted to their environment and therefore lose their portability. Unfortunately, I am not aware of any other solution to this problem at the moment.

Raspberry Pi Kubernetes Cluster

2018-09-25T00:00:00Z

In order to gain experience with a Kubernetes cluster or to be able to experiment with it, a functioning cluster is required. Since most conceptual challenges do not require a high performance test cluster, it is also sufficient to build a smaller and therefore more cost-effective one. For this reason I decided to set up a Raspberry Pi Kubernetes Cluster for testing purposes.

Shopping List

4x Raspberry Pi 3 Model B+
4x microSD Card (I'm using SanDisk Ultra 64GB)
Power supply unit for the Raspberry Pi devices. You can also use any other 5V power source which provides enough current.
4 Micro USB cables for connecting the power suppy unit
4 Layer Acrylic Cluster Case

If the Raspberry Pis are not to be connected via WLAN but cable, the corresponding network components are also required:

5 Port Switch
4 short ethernet cables

The website of Hypriot has a very good tutorial how to set up a Kubernetes cluster with Raspberry Pi boards: https://blog.hypriot.com/post/setup-kubernetes-raspberry-pi-cluster/. If you need some configuration examples (executable on a Raspberry Pi Kubernetes Cluster) please check out my GitHub repository with configuration examples: https://github.com/MatthiasLohr/kubernetes-rpi-examples.

GPG Agent for SSH in Gnome

2018-06-13T00:00:00Z

In How to set up your YubiKey NEO I already mentioned that you can also use your YubiKey as SSH key. In GPG Agent Forwarding I show how to forward your GPG agent to remote machines for decryption/signing. What's missing is a tutorial on how to make it all work together, how to use your GPG Agent for SSH in Gnome.

Prerequisites

This manual refers to combining a YubiKey (as GPG smart card) with GPG agent with SSH support as ssh-agent replacement in Ubuntu 18.04 with Gnome. I assume you have already read the article How to set up your YubiKey NEO and set up your YubiKey (or any other smart card) and generated the SSH keys. I cannot exclude that the manual will also work for other distributions/versions, but I have tested it exclusively for Ubuntu 18.04 with Gnome. Experience reports or problems can be left in the comment field. If anyone has more useful information, I'd be happy to update this article.

Setup

Actually the configuration is quite simple, but it took me a long time to figure out how it works. Here are the steps to take to use the GPG Agent for SSH in Gnome:

First we have to disable the Gnome keyring. The problem is that the Gnome keyring itself starts an SSH agent and sets the variable $SSH_AUTH_SOCK - overwriting the value of already running, other agents. I found the solution to this step here: https://wiki.archlinux.org/index.php/GNOME/Keyring#Disable_keyring_daemon_components. However, deactivating the autostart was enough for me. I didn't have to set GSM_SKIP_SSH_AGENT_WORKAROUND. mkdir -p ~/.config/autostart cp /etc/xdg/autostart/gnome-keyring-ssh.desktop ~/.config/autostart echo "Hidden=true" >> ~/.config/autostart/gnome-keyring-ssh.desktop
The next step is to activate the GPG agent: echo "use-agent" >> ~/.gnupg/gpg.conf
Next, enable SSH support for the GPG agent: echo "enable-ssh-support" >> ~/.gnupg/gpg-agent.conf

Then log off (or restart) for security to stop all running services (and agents). Next time you login, $SSH_AUTH_SOCK should point to the GPG agent socket:

echo $SSH_AUTH_SOCK
/run/user/1000/gnupg/S.gpg-agent.ssh

Now you can test if it's working. Enjoy! :)