Deploying Openshift Worker on Physical blade Server with Trunk networking (bond/single interface)

Why this article

recently I had the pleasure of deploying OpenShift 4 in a disconnected environment over a set of a mix between physical servers and virtual servers. The bootstrap server and the masters are running on VMWARE and the workers are running on HP Blade servers.

the Installation in been done over PXE install.

What went wrong ?

for some reason the physical server are getting the IP for the PXE boot but due to VLAN TAG issues the installation in unable to continue past this point

Solution

for a solution we decided to take the Fedora Coreos LiveCD and boot the server from it .

once the livecd completed the boot we need to define the IP and VLAN TAG on the interface.

so first lets create the VLAN:

# ip link add link eth0 name eth0.100 type vlan id 100

then we added the IP :

# ip addr add 192.168.0.5/24 brd 192.168.100.255 dev eth0.100

and adding a default route :

# ip route add default via 192.168.0.1

once the IP was configured we test the IP by pinging the Default Gateway :

# ping 192.168.0.1

if Everything is working as expected we can first pull the ignition file :

# curl -LO http://bastion/pub/ign/worker.ign

I would recommend downloading the Image file as well so the installation process will be quickly

# curl -LO http://bastion/pub/pxe/rhocs-4.3.8.raw.xz

once both download are completed we can run the coreos-installer command (make sure you are a root user OR use sudo )

# coreos-installer -i worker.ign -f rhcos-4.3.8.raw.xz --firstboot-args "rd.neednet=1 vlan=vlan100:eth0 ip=vlan100:dhcp --insecure /dev/sda

Flags :

1. the flags “-i” and “-f” points to the ignition file and to the image file respectively
2. firstboot-args → set an addition first boot arguments that Openshift needs when it boots for the first time
3. insecure → we are using it because we do not provide a signature file for our rhcos image.

the process should be over very quickly and once it is done we can reboot the server :

# reboot

once the Server completes the boot for the first time , reboot it again and then it will come up with it’s new VLAN Tagged interface and are good to go.

working with Bond and VLAN TAG

In some cases the PXE boot is working with bond configuration but then Red Hat CoreOS does not boot (after the first boot) with bond configuration.
for that purpose we need to generate the files and add them to the ignition file.

PXE configuration

In order to work with PXE boot and bond configuration while providing VLAN tag the following configuration should be set in the MAC configuration of the tftp boot server:

rd.neetnet=1 ip=bond0.100:dhcp vlan=bond0.100:bond0 bond=bond0:eth0,eth1:mode=active-backup,primary=eth0,miimon=100

Once we have added the above parameters to the PXE boot we need to generate the files for the ignition file:

Due to the fact that we want to use static IP on our Servers once they are boot we need to generate the final bond.vlan file for each server but we don’t want to change the ignition file which was provided by the openshift-install command so we will create and append file for each server (No , there is no other way to avoid it ) and add the files to that server.

For example if our physical server in named worker1.exmpale.com with the interfaces eth0 ,eth1 , and needs a VLAN tag of 100 to work then we create the following files for it:

# cd /var/www/html/pub/ign/
# cat >> worker1.ign << EOF
{
  "ignition": {
    "config": {
      "append": [
        {
          "source": "http://bastion/pub/ign/worker.ign",
          "verification": {}
        }
      ]
    },
    "timeouts": {},
    "version": "2.1.0"
  },
  "networkd": {},
  "passwd": {},
  "storage": {},  
  "systemd": {}

}
EOF

Note that we are still using the original worker.ign file but it is now as append to our file (nothing gets lost…).

now we will create the ifcfg-eth0 , ifcfg-eth1 , ifcfg-bond0 and the ifcfg-bond0.vlan file (the last one contains the IP address of the server)

Sense the ignition file works with base64 coding we need to first create the files and then generate the base64 code from them :

For ifcfg-eth0

# mkdir network-scripts (can be in any directory)
# cd network-scripts
# cat > ifcfg-eth0 << EOF
DEVICE=eth0
NAME=eth0
BOOTPROTO=none
SLAVE=yes
MASTER=bond0
EOF

For ifcfg-eth1

# cat > ifcfg-eth1 << EOF
DEVICE=eth1
NAME=eth1
BOOTPROTO=none
SLAVE=yes
MASTER=bond0
EOF

For the bond interface :

# cat > ifcfg-bond0 << EOF
NAME=bond0
DEVICE=bond0
ONBOOT=yes
BOOTPROTO=none
TYPE=Bond
BONDING_MASTER=yes
BONDING_OPTS="mode=1 miimon=100"
EOF

The last interface will be ifcfg-bond0.100 and it will hold the static IP for the Server:

# cat > ifcfg-bond0.100 << EOF
NAME="bond0.100"
IPADDR="192.168.10.2"
NETMASK="255.255.255.0"
GATEWAY="192.168.1.1"
BOOTPROTO="static"
DEVICE="bond0.100"
TYPE="Vlan"
ONBOOT="yes"
VLAN=yes
DNS1="192.168.10.11"
DNS2="192.168.10.12"
DNS3="192.168.10.13"
EOF

Your files should look like this :

network-scripts]$ ls -la
total 24
drwxrwxr-x.  2 paas paas 4096 Jun  3 11:11 .
drwx------. 29 paas paas 4096 Jun  3 11:10 ..
-rw-rw-r--.  1 paas paas  112 Jun  3 11:11 ifcfg-bond0
-rw-rw-r--.  1 paas paas  220 Jun  3 11:11 ifcfg-bond0.100
-rw-rw-r--.  1 paas paas   60 Jun  3 11:10 ifcfg-eth0
-rw-rw-r--.  1 paas paas   60 Jun  3 11:11 ifcfg-eth1

Now that we have all the files in place we will generate a base64 coding from them :

# for file in `ls ifcfg-*`; do
echo $file
cat $file | base64 -w0 ; echo
done

that will print the interface name and it’s base64 content.

Modifying Append

Once we have the base64 in place let’s update the append file we create for the server in the storage section to make sure it distribute the files to the right location .

the storage section should look like :

..
"storage": {
  "files": [
     {
        "contents": {
          "source": "data:text/plain;base64,<ETH0_BASE64>"
        },
        "filesystem": "root",
        "mode": 644,
        "path": "/etc/sysconfig/network-scripts/ifcfg-eth0"
     },
     {            
        "contents": {
          "source": "data:text/plain;base64,<ETH1_BASE64>"
        },
        "filesystem": "root",
        "mode": 644,
        "path": "/etc/sysconfig/network-scripts/ifcfg-eth1"
     },
     {            
        "contents": {
          "source": "data:text/plain;base64,<BOND0_BASE64>"
        },
        "filesystem": "root",
        "mode": 644,
        "path": "/etc/sysconfig/network-scripts/ifcfg-bond0"
     },
     {            
        "contents": {
          "source": "data:text/plain;base64,<BOND0_100_BASE64>"
        },
        "filesystem": "root",
        "mode": 644,
        "path": "/etc/sysconfig/network-scripts/ifcfg-bond0.100"
     },
  ]
},

As you can see I left the base64 empty (where the name of the interface) for you to complete.

The easist way to complete it is with the jq and sed command for each interface :

# export ETH0_BASE64=`cat ifcfg-eth0 | base64 -w0`
# cat worker1.ign | jq .storage.files | sed -i "s/<ETH0_BASE64>/${ETH0_BASE64}/g" worker1.ign

Repeat those steps for each interface and your file is ready.

once you are done all we need to do is to change the original ignition file to our newly append one in the PXE boot file :

Change the coreos.inst.ignition_url to the append file which should look like the following reference :

coreos.inst.ignition_url=http://bastion.exmaple.com/pub/ign/worker1.ign

Next time you will boot the server (from PXE) it will use the append file (for worker1) and will input the bond configuration after the “first boot” sequence.

That is it
we can now continue with our Cluster deployment.