May 30, 2022

4 min read

OpenShift 4 with Jumbo frames in the installation stage

why Jumbo Frames

Kubernetes (And OpenShift as a result) uses SDN (Software define network) to manage the Pod and the services IP addresses.
During my line of work I notice that the east / west network connectivity when Pods are running on 2 different nodes is reaching only about 25% of the actual bandwidth between the nodes.

For Example if the nodes are on 10G interfaces the Pod 2 Pod communication in the SDN will reach between 1.8 to 2.4 Gigabit which is far less than expected. After several tests I notice that when I’ve increased the MTU on (ALL) the nodes’ interfaces the east / west communication has increased to 75% — 80% so in a 10G network I reached between 7.5 to 8 Gigabit per second.

(If you run tests yourself you can use the iperfcon tool I built here).

About Jumbo Frames

jumbo frames are Ethernet frames with more than 1500 bytes of payload, the limit set by the IEEE 802.3 standard. Commonly, jumbo frames can carry up to 9000 bytes of payload, but smaller and larger variations exist and some care must be taken using the term. Many Gigabit Ethernet switches and Gigabit Ethernet network interface controllers and some Fast Ethernet switches and Fast Ethernet network interface cards can support jumbo frames.

Restriction

A very important note about Jumbo frames is that all the network appliances in the Broadcast domain must be configured with Jumbo frames as well. That will include the switches , routers , Load Balancer and the Bastion server.

In OpenShift

In OpenShift we can not change the MTU post installation so we need to take care of it during the installation.
During a UPI installation there are 2 places we need to update. Both are available to us after we create the manifests.

Manifests

There are 2 files we need to update in the “openshift” and the “manifests” directories.

Create a new machineConfig in the openshift directory for the physical interfaces.
Modify the MTU on the OpenShiftSDN network with advanced configuration.

Start the installation

Let’s start the installation assuming you are in your installation directory (I will not provide an example for the install-config.yaml file in this tutorial)

$ mkdir ~/install/
$ openshift-install create manifests --dir=./

Physical Interfaces

For the physical interfaces we need to add a small script for the dispatcher.d directory under NetworkManager so it will run just as the network starts.

First let’s create the script in advanced :

$ mkdir /tmp/mtu
$ cd /tmp/mtu

Now for the script :

$ echo '#!/bin/shMTU=9000ifaces=$(ip -br addr show | grep -i up | grep -v veth | awk '{print $1}')for iface in ${ifaces}; do 
   ip link set ${iface} mtu ${MTU}
done' > 30-mtu

Now that the file is ready let’s generate a base64 file from it :

$ cat 30-mtu | base64 -w0
IyEvYm.........................fQpkb25lCg==

For the next step we will create a MachineConfig that will run the script trough NetworkManager.

Go to the openshift directory under your installation directory :

$ cd ~/install/openshift

And now we will create a machineConfig file with the base64 results

$ cat > 10-workers-jumbo-frames.yaml << EOF
kind: MachineConfig
apiVersion: machineconfiguration.openshift.io/v1
metadata:
  name: 10-worker-mtu
  creationTimestamp: 
  labels:
    machineconfiguration.openshift.io/role: worker
spec:
  osImageURL: ''
  config:
    ignition:
      version: 3.2.0
    storage:
      files:
      - filesystem: root
        path: "/etc/NetworkManager/dispatcher.d/30-mtu"
        contents:
          source: data:text/plain;charset=utf-8;base64,$(cat /tmp/mtu/30-mtu | base64 -w0)
          verification: {}
        mode: 0755
    systemd:
      units:
        - contents: |
            [Unit]
            Requires=systemd-udevd.target
            After=systemd-udevd.target
            Before=NetworkManager.service
            DefaultDependencies=no
            [Service]
            Type=oneshot
            ExecStart=/usr/sbin/restorecon /etc/NetworkManager/dispatcher.d/30-mtu
            [Install]
            WantedBy=multi-user.target
          name: one-shot-mtu.service
          enabled: true
EOF

And another one for the master nodes :

$ cat > 10-masters-jumbo-frames.yaml << EOF
kind: MachineConfig
apiVersion: machineconfiguration.openshift.io/v1
metadata:
  name: 10-master-mtu
  creationTimestamp: 
  labels:
    machineconfiguration.openshift.io/role: master
spec:
  osImageURL: ''
  config:
    ignition:
      version: 3.2.0
    storage:
      files:
      - filesystem: root
        path: "/etc/NetworkManager/dispatcher.d/30-mtu"
        contents:
          source: data:text/plain;charset=utf-8;base64,$(cat /tmp/mtu/30-mtu | base64 -w0)
          verification: {}
        mode: 0755
    systemd:
      units:
        - contents: |
            [Unit]
            Requires=systemd-udevd.target
            After=systemd-udevd.target
            Before=NetworkManager.service
            DefaultDependencies=no
            [Service]
            Type=oneshot
            ExecStart=/usr/sbin/restorecon /etc/NetworkManager/dispatcher.d/30-mtu
            [Install]
            WantedBy=multi-user.target
          name: one-shot-mtu.service
          enabled: true
EOF

Review the files and make sure the base64 are equal to your previous command output.

SDN MTU

For the SDN configuration we need to add a few lines to the network definition file.

First we need to deduct 192 bit from the 9000 MTU we defined on the physical network interface to allow 2 encapsulation. so we will ran simple math solution :

$ echo $((9000-192))
8808

Let’s go to the “manifests” directory

$ cd ../manifests

Now let’s edit the file called “cluster-network-02-config.yml”

$ vi cluster-network-02-config.yml

Before editing it the file should look like this (IP will match your install-config.yaml configuration) :

apiVersion: config.openshift.io/v1
kind: Network
metadata:
  creationTimestamp: null
  name: cluster
spec:
  clusterNetwork:
  - cidr: 10.128.0.0/13
    hostPrefix: 23
  externalIP:
    policy: {}
  networkType: OpenShiftSDN
  serviceNetwork:
  - 172.30.0.0/16
status: {}

Next we will add advanced configuration for the default network with the following section :

defaultNetwork:
  type: OpenShiftSDN
  openshiftSDNConfig:
    mode: NetworkPolicy
    mtu: 8808
    vxlanPort: 4789

After our addition the file should look like this :

apiVersion: config.openshift.io/v1
kind: Network
metadata:
  creationTimestamp: null
  name: cluster
spec:
  clusterNetwork:
  - cidr: 10.128.0.0/13
    hostPrefix: 23
  externalIP:
    policy: {}
  networkType: OpenShiftSDN
  serviceNetwork:
  - 172.30.0.0/16
  defaultNetwork:
    type: OpenShiftSDN
    openshiftSDNConfig:
      mode: NetworkPolicy
      mtu: 8808
      vxlanPort: 4789

Save the file.

Once we configured both parts we can go head and continue with the installation. By the time the installation is complete your OpenShift will be working with Jumbo frames.

If you have any question feel free to responed/ leave a comment.
You can find on linkedin at : https://www.linkedin.com/in/orenoichman
Or twitter at : https://twitter.com/ooichman