OpenShift 4 with Jumbo frames in the installation stage
why Jumbo Frames
Kubernetes (And OpenShift as a result) uses SDN (Software define network) to manage the Pod and the services IP addresses.
During my line of work I notice that the east / west network connectivity when Pods are running on 2 different nodes is reaching only about 25% of the actual bandwidth between the nodes.
For Example if the nodes are on 10G interfaces the Pod 2 Pod communication in the SDN will reach between 1.8 to 2.4 Gigabit which is far less than expected. After several tests I notice that when I’ve increased the MTU on (ALL) the nodes’ interfaces the east / west communication has increased to 75% — 80% so in a 10G network I reached between 7.5 to 8 Gigabit per second.
(If you run tests yourself you can use the iperfcon tool I built here).
About Jumbo Frames
jumbo frames are Ethernet frames with more than 1500 bytes of payload, the limit set by the IEEE 802.3 standard. Commonly, jumbo frames can carry up to 9000 bytes of payload, but smaller and larger variations exist and some care must be taken using the term. Many Gigabit Ethernet switches and Gigabit Ethernet network interface controllers and some Fast Ethernet switches and Fast Ethernet network interface cards can support jumbo frames.
Restriction
A very important note about Jumbo frames is that all the network appliances in the Broadcast domain must be configured with Jumbo frames as well. That will include the switches , routers , Load Balancer and the Bastion server.
In OpenShift
In OpenShift we can not change the MTU post installation so we need to take care of it during the installation.
During a UPI installation there are 2 places we need to update. Both are available to us after we create the manifests.
Manifests
There are 2 files we need to update in the “openshift” and the “manifests” directories.
- Create a new machineConfig in the openshift directory for the physical interfaces.
- Modify the MTU on the OpenShiftSDN network with advanced configuration.
Start the installation
Let’s start the installation assuming you are in your installation directory (I will not provide an example for the install-config.yaml file in this tutorial)
$ mkdir ~/install/
$ openshift-install create manifests --dir=./
Physical Interfaces
For the physical interfaces we need to add a small script for the dispatcher.d directory under NetworkManager so it will run just as the network starts.
First let’s create the script in advanced :
$ mkdir /tmp/mtu
$ cd /tmp/mtu
Now for the script :
$ echo '#!/bin/shMTU=9000ifaces=$(ip -br addr show | grep -i up | grep -v veth | awk '{print $1}')for iface in ${ifaces}; do
ip link set ${iface} mtu ${MTU}
done' > 30-mtu
Now that the file is ready let’s generate a base64 file from it :
$ cat 30-mtu | base64 -w0
IyEvYm.........................fQpkb25lCg==
For the next step we will create a MachineConfig that will run the script trough NetworkManager.
Go to the openshift directory under your installation directory :
$ cd ~/install/openshift
And now we will create a machineConfig file with the base64 results
$ cat > 10-workers-jumbo-frames.yaml << EOF
kind: MachineConfig
apiVersion: machineconfiguration.openshift.io/v1
metadata:
name: 10-worker-mtu
creationTimestamp:
labels:
machineconfiguration.openshift.io/role: worker
spec:
osImageURL: ''
config:
ignition:
version: 3.2.0
storage:
files:
- filesystem: root
path: "/etc/NetworkManager/dispatcher.d/30-mtu"
contents:
source: data:text/plain;charset=utf-8;base64,$(cat /tmp/mtu/30-mtu | base64 -w0)
verification: {}
mode: 0755
systemd:
units:
- contents: |
[Unit]
Requires=systemd-udevd.target
After=systemd-udevd.target
Before=NetworkManager.service
DefaultDependencies=no
[Service]
Type=oneshot
ExecStart=/usr/sbin/restorecon /etc/NetworkManager/dispatcher.d/30-mtu
[Install]
WantedBy=multi-user.target
name: one-shot-mtu.service
enabled: true
EOF
And another one for the master nodes :
$ cat > 10-masters-jumbo-frames.yaml << EOF
kind: MachineConfig
apiVersion: machineconfiguration.openshift.io/v1
metadata:
name: 10-master-mtu
creationTimestamp:
labels:
machineconfiguration.openshift.io/role: master
spec:
osImageURL: ''
config:
ignition:
version: 3.2.0
storage:
files:
- filesystem: root
path: "/etc/NetworkManager/dispatcher.d/30-mtu"
contents:
source: data:text/plain;charset=utf-8;base64,$(cat /tmp/mtu/30-mtu | base64 -w0)
verification: {}
mode: 0755
systemd:
units:
- contents: |
[Unit]
Requires=systemd-udevd.target
After=systemd-udevd.target
Before=NetworkManager.service
DefaultDependencies=no
[Service]
Type=oneshot
ExecStart=/usr/sbin/restorecon /etc/NetworkManager/dispatcher.d/30-mtu
[Install]
WantedBy=multi-user.target
name: one-shot-mtu.service
enabled: true
EOF
Review the files and make sure the base64 are equal to your previous command output.
SDN MTU
For the SDN configuration we need to add a few lines to the network definition file.
First we need to deduct 192 bit from the 9000 MTU we defined on the physical network interface to allow 2 encapsulation. so we will ran simple math solution :
$ echo $((9000-192))
8808
Let’s go to the “manifests” directory
$ cd ../manifests
Now let’s edit the file called “cluster-network-02-config.yml”
$ vi cluster-network-02-config.yml
Before editing it the file should look like this (IP will match your install-config.yaml configuration) :
apiVersion: config.openshift.io/v1
kind: Network
metadata:
creationTimestamp: null
name: cluster
spec:
clusterNetwork:
- cidr: 10.128.0.0/13
hostPrefix: 23
externalIP:
policy: {}
networkType: OpenShiftSDN
serviceNetwork:
- 172.30.0.0/16
status: {}
Next we will add advanced configuration for the default network with the following section :
defaultNetwork:
type: OpenShiftSDN
openshiftSDNConfig:
mode: NetworkPolicy
mtu: 8808
vxlanPort: 4789
After our addition the file should look like this :
apiVersion: config.openshift.io/v1
kind: Network
metadata:
creationTimestamp: null
name: cluster
spec:
clusterNetwork:
- cidr: 10.128.0.0/13
hostPrefix: 23
externalIP:
policy: {}
networkType: OpenShiftSDN
serviceNetwork:
- 172.30.0.0/16
defaultNetwork:
type: OpenShiftSDN
openshiftSDNConfig:
mode: NetworkPolicy
mtu: 8808
vxlanPort: 4789
Save the file.
Once we configured both parts we can go head and continue with the installation. By the time the installation is complete your OpenShift will be working with Jumbo frames.
If you have any question feel free to responed/ leave a comment.
You can find on linkedin at : https://www.linkedin.com/in/orenoichman
Or twitter at : https://twitter.com/ooichman