OpenShift 4 with Jumbo frames in the installation stage

why Jumbo Frames

Kubernetes (And OpenShift as a result) uses SDN (Software define network) to manage the Pod and the services IP addresses.
During my line of work I notice that the east / west network connectivity when Pods are running on 2 different nodes is reaching only about 25% of the actual bandwidth between the nodes.

For Example if the nodes are on 10G interfaces the Pod 2 Pod communication in the SDN will reach between 1.8 to 2.4 Gigabit which is far less than expected. After several tests I notice that when I’ve increased the MTU on (ALL) the nodes’ interfaces the east / west communication has increased to 75% — 80% so in a 10G network I reached between 7.5 to 8 Gigabit per second.

(If you run tests yourself you can use the iperfcon tool I built here).

About Jumbo Frames

jumbo frames are Ethernet frames with more than 1500 bytes of payload, the limit set by the IEEE 802.3 standard. Commonly, jumbo frames can carry up to 9000 bytes of payload, but smaller and larger variations exist and some care must be taken using the term. Many Gigabit Ethernet switches and Gigabit Ethernet network interface controllers and some Fast Ethernet switches and Fast Ethernet network interface cards can support jumbo frames.

A very important note about Jumbo frames is that all the network appliances in the Broadcast domain must be configured with Jumbo frames as well. That will include the switches , routers , Load Balancer and the Bastion server.

In OpenShift

In OpenShift we can not change the MTU post installation so we need to take care of it during the installation.
During a UPI installation there are 2 places we need to update. Both are available to us after we create the manifests.

Manifests

There are 2 files we need to update in the “openshift” and the “manifests” directories.

  1. Create a new machineConfig in the openshift directory for the physical interfaces.
  2. Modify the MTU on the OpenShiftSDN network with advanced configuration.

Start the installation

Let’s start the installation assuming you are in your installation directory (I will not provide an example for the install-config.yaml file in this tutorial)

$ mkdir ~/install/
$ openshift-install create manifests --dir=./

For the physical interfaces we need to add a small script for the dispatcher.d directory under NetworkManager so it will run just as the network starts.

First let’s create the script in advanced :

$ mkdir /tmp/mtu
$ cd /tmp/mtu

Now for the script :

$ echo '#!/bin/shMTU=9000ifaces=$(ip -br addr show | grep -i up | grep -v veth | awk '{print $1}')for iface in ${ifaces}; do 
ip link set ${iface} mtu ${MTU}
done' > 30-mtu

Now that the file is ready let’s generate a base64 file from it :

$ cat 30-mtu | base64 -w0
IyEvYm.........................fQpkb25lCg==

For the next step we will create a MachineConfig that will run the script trough NetworkManager.

Go to the openshift directory under your installation directory :

$ cd ~/install/openshift

And now we will create a machineConfig file with the base64 results

$ cat > 10-workers-jumbo-frames.yaml << EOF
kind: MachineConfig
apiVersion: machineconfiguration.openshift.io/v1
metadata:
name: 10-worker-mtu
creationTimestamp:
labels:
machineconfiguration.openshift.io/role: worker
spec:
osImageURL: ''
config:
ignition:
version: 3.2.0
storage:
files:
- filesystem: root
path: "/etc/NetworkManager/dispatcher.d/30-mtu"
contents:
source: data:text/plain;charset=utf-8;base64,$(cat /tmp/mtu/30-mtu | base64 -w0)
verification: {}
mode: 0755
systemd:
units:
- contents: |
[Unit]
Requires=systemd-udevd.target
After=systemd-udevd.target
Before=NetworkManager.service
DefaultDependencies=no
[Service]
Type=oneshot
ExecStart=/usr/sbin/restorecon /etc/NetworkManager/dispatcher.d/30-mtu
[Install]
WantedBy=multi-user.target
name: one-shot-mtu.service
enabled: true
EOF

And another one for the master nodes :

$ cat > 10-masters-jumbo-frames.yaml << EOF
kind: MachineConfig
apiVersion: machineconfiguration.openshift.io/v1
metadata:
name: 10-master-mtu
creationTimestamp:
labels:
machineconfiguration.openshift.io/role: master
spec:
osImageURL: ''
config:
ignition:
version: 3.2.0
storage:
files:
- filesystem: root
path: "/etc/NetworkManager/dispatcher.d/30-mtu"
contents:
source: data:text/plain;charset=utf-8;base64,$(cat /tmp/mtu/30-mtu | base64 -w0)
verification: {}
mode: 0755
systemd:
units:
- contents: |
[Unit]
Requires=systemd-udevd.target
After=systemd-udevd.target
Before=NetworkManager.service
DefaultDependencies=no
[Service]
Type=oneshot
ExecStart=/usr/sbin/restorecon /etc/NetworkManager/dispatcher.d/30-mtu
[Install]
WantedBy=multi-user.target
name: one-shot-mtu.service
enabled: true
EOF

Review the files and make sure the base64 are equal to your previous command output.

For the SDN configuration we need to add a few lines to the network definition file.

First we need to deduct 192 bit from the 9000 MTU we defined on the physical network interface to allow 2 encapsulation. so we will ran simple math solution :

$ echo $((9000-192))
8808

Let’s go to the “manifests” directory

$ cd ../manifests

Now let’s edit the file called “cluster-network-02-config.yml”

$ vi cluster-network-02-config.yml

Before editing it the file should look like this (IP will match your install-config.yaml configuration) :

apiVersion: config.openshift.io/v1
kind: Network
metadata:
creationTimestamp: null
name: cluster
spec:
clusterNetwork:
- cidr: 10.128.0.0/13
hostPrefix: 23
externalIP:
policy: {}
networkType: OpenShiftSDN
serviceNetwork:
- 172.30.0.0/16
status: {}

Next we will add advanced configuration for the default network with the following section :

defaultNetwork:
type: OpenShiftSDN
openshiftSDNConfig:
mode: NetworkPolicy
mtu: 8808
vxlanPort: 4789

After our addition the file should look like this :

apiVersion: config.openshift.io/v1
kind: Network
metadata:
creationTimestamp: null
name: cluster
spec:
clusterNetwork:
- cidr: 10.128.0.0/13
hostPrefix: 23
externalIP:
policy: {}
networkType: OpenShiftSDN
serviceNetwork:
- 172.30.0.0/16
defaultNetwork:
type: OpenShiftSDN
openshiftSDNConfig:
mode: NetworkPolicy
mtu: 8808
vxlanPort: 4789

Save the file.

Once we configured both parts we can go head and continue with the installation. By the time the installation is complete your OpenShift will be working with Jumbo frames.

If you have any question feel free to responed/ leave a comment.
You can find on linkedin at : https://www.linkedin.com/in/orenoichman
Or twitter at : https://twitter.com/ooichman

--

--

--

Open Source contributer for the past 15 years

Love podcasts or audiobooks? Learn on the go with our new app.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Oren Oichman

Oren Oichman

Open Source contributer for the past 15 years

More from Medium

Azure HA Kubernetes Monitoring using Prometheus and Thanos

Build Golden Images For Your Openshift Virtualization Cloud Using RHEL Image Builder

Introduce HashiCorp Vault secrets into Kubernetes via CSI secret store driver

How to Set up a Highly Available Kubernetes cluster with Kubeadm