OpenShift 4 in an Air Gap (disconnected) environment (Part 2 — installation)

The Installation

In this part we will focus on the deployment part and what we need to do for the installation process to be successful in our air Gapped (disconnected) environment.
If you need any help with setting up the Infrastructure prior to the installation you can refer to the first part article (should be published by the end of February)

Author’s advice

Copy this article to a PDF and make sure it is available in your Air Gapped environment to avoid typos or any other misconfigurations

Scenario

For a very basic scenario we will need to following servers :

  • 1 installation Server (prefered RHEL7/8)
  • 1 guest VM without OS for the bootstrap (will be RHCOS)
  • 3 guest VM without OS for the Masters (will be RHCOS)
  • 3 guest VM without OS for the Workers (will be RHCOS)

Creating the registry

Before we begin as I always recommend, lets start a screen session in case we are disconnect for any reason:

$ screen -S ocp
OR (tmux new-session -s ocp)
$ mkdir /opt/registry
$ export REGISTRY_BASE="/opt/registry"
$ mkdir -p ${REGISTRY_BASE}/{auth,certs,data,downloads}
$ mkdir -p ${REGISTRY_BASE}/downloads/{images,tools,secrets}
$ vi /etc/hosts
$ 127.0.0.1 registry

packages :

Both the external and internal server will be needing a few package to make our work easier, so let’s go ahead and install them :
(you will need to enable EPEL in order to obtain all the tools)

$ yum install -y jq openssl podman p7zip httpd-tools curl wget screen nmap telnet ftp tftp openldap-clients tcpdump wireshark xorg-x11-xauth tmux net-tools nfs-utils sg3_utils bind-utils rlwrap uucp

NOTE

If you choose to use docker (and the docker service) instead of podman you do need to make sure you are consistent with this decision trough our tutorial.

$ cd ${REGISTRY_BASE}/certs/
$ cat >csr_answer.txt << EOF
[req]
default_bits = 4096
prompt = no
default_md = sha256
distinguished_name = dn
[ dn ]
C=US
ST=New York
L=New York
O=MyOrg
OU=MyOU
emailAddress=me@working.me
CN = registry
EOF
$ openssl req -newkey rsa:4096 -nodes -sha256 -keyout domain.key -x509 -days 365 -out domain.crt -config <( cat csr_answer.txt )
$ ls -al
total 20
drwxr-xr-x. 2 root root 4096 Jan 8 13:49 .
drwxr-xr-x. 7 root root 4096 Jan 8 09:57 ..
-rw-r — r — . 1 root root 175 Jan 8 13:48 csr_answer.txt
-rw-r — r — . 1 root root 1972 Jan 8 13:49 domain.crt
-rw-r — r — . 1 root root 3272 Jan 8 13:49 domain.key
$ cp ${REGISTRY_BASE}/certs/domain.crt /etc/pki/ca-trust/source/anchors/$ update-ca-trust extract
$ htpasswd -bBc ${REGISTRY_BASE}/auth/htpasswd myuser mypassword
$ export FIREWALLD_DEFAULT_ZONE=`firewall-cmd --get-default-zone`
$ echo ${FIREWALLD_DEFAULT_ZONE}
public
$ firewall-cmd --add-port=5000/tcp --zone=${FIREWALLD_DEFAULT_ZONE} --permanent$ firewall-cmd --reload
$ echo 'podman run --name my-registry --rm -d -p 5000:5000 \
-v ${REGISTRY_BASE}/data:/var/lib/registry:z \
-v ${REGISTRY_BASE}/auth:/auth:z -e "REGISTRY_AUTH=htpasswd" \
-e "REGISTRY_AUTH_HTPASSWD_REALM=Registry" \
-e "REGISTRY_HTTP_SECRET=ALongRandomSecretForRegistry" \
-e REGISTRY_AUTH_HTPASSWD_PATH=/auth/htpasswd \
-v ${REGISTRY_BASE}/certs:/certs:z \
-e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt \
-e REGISTRY_HTTP_TLS_KEY=/certs/domain.key \
docker.io/library/registry:2' > ${REGISTRY_BASE}/downloads/tools/start_registry.sh
$ chmod a+x ${REGISTRY_BASE}/downloads/tools/start_registry.sh$ ${REGISTRY_BASE}/downloads/tools/start_registry.sh
$ curl -u myuser:mypassword -k https://registry:5000/v2/_catalog 
{"repositories":[]}

syncing the repository

First lets grep the build version we are going to download.
This is located on the release.txt file in the openshift download directory.
By the way lets put the output in and “env” file to use it in our internal server

$ export OCP_RELEASE=$(curl -s https://mirror.openshift.com/pub/openshift-v4/clients/ocp/latest/release.txt | grep 'Name:' | awk '{print $NF}')
$ echo "export OCP_RELEASE=${OCP_RELEASE}" >> ${REGISTRY_BASE}/downloads/tools/env_ocp
$ wget https://mirror.openshift.com/pub/openshift-v4/clients/ocp/latest/openshift-client-linux-${OCP_RELEASE}.tar.gz -P ${REGISTRY_BASE}/downloads/tools/$ tar -xzf ${REGISTRY_BASE}/downloads/tools/openshift-client-linux-${OCP_RELEASE}.tar.gz -C ${REGISTRY_BASE}/downloads/tools/$ ln -s ${REGISTRY_BASE}/downloads/tools/oc /usr/local/bin/oc
$ export OCP_VERSION=4.6
$ echo "export OCP_VERSION=4.6" >> ${REGISTRY_BASE}/downloads/tools/env_ocp
$ OCP_ISO_VERSION=$(curl -s https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/latest/latest/sha256sum.txt | grep live| awk -F\- '{print $2}' | head -1)
$ echo ${OCP_ISO_VERSION}
$ wget https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/latest/latest/rhcos-${OCP_ISO_VERSION}-x86_64-live-initramfs.x86_64.img -P ${REGISTRY_BASE}/downloads/images/$ wget https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/latest/latest/rhcos-${OCP_ISO_VERSION}-x86_64-live-kernel-x86_64 -P ${REGISTRY_BASE}/downloads/images/$ wget https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/latest/latest/rhcos-${OCP_ISO_VERSION}-x86_64-metal.x86_64.raw.gz -P ${REGISTRY_BASE}/downloads/images/

obtaining the pull secret

In order to obtain the pull secret we will need to go to :

  1. Click on the “create cluster” button
  2. Select “Openshift container platform”
  3. Select our installation type (select Bare Metal even if you are planning to deploy it on VM)
  4. Download the pull secret ( or click on the “copy Pull Secret”) as seen in the following image
$ cd ${REGISTRY_BASE}/downloads/secrets/
$ cat > pull-secret.json << EOF
(CTRL+V)
EOF
$ cat pull-secret.json | jq
$ echo -n ‘myuser:mypassword’ | base64 -w0
$ REG_SECRET=`echo -n 'myuser:mypassword' | base64 -w0`
$ cat pull-secret.json | jq '.auths += {"registry:5000": {"auth": "REG_SECRET","email": "me@working.me"}}' | sed "s/REG_SECRET/$REG_SECRET/" > pull-secret-bundle.json
$ cat pull-secret-bundle.json | jq
$ echo '{ "auths": {}}' | jq '.auths += {"registry:5000": {"auth": "REG_SECRET","email": "me@working.me"}}' | sed "s/REG_SECRET/$REG_SECRET/" | jq -c .> pull-secret-registry.json
$ export LOCAL_REGISTRY='registry:5000'$ export OCP_RELEASE="${OCP_RELEASE}-x86_64" $ export LOCAL_REPOSITORY='ocp/openshift4' $ export PRODUCT_REPO='openshift-release-dev' $ export LOCAL_SECRET_JSON="${REGISTRY_BASE}/downloads/secrets/pull-secret-bundle.json" $ export RELEASE_NAME="ocp-release" 
$ echo "export LOCAL_REGISTRY='registry:5000'" >> ${REGISTRY_BASE}/downloads/tools/env_ocp$ echo '[[ ! ${OCP_RELEASE} =~ 'x86_64' ]] && export OCP_RELEASE="${OCP_RELEASE}-x86_64"' >> ${REGISTRY_BASE}/downloads/tools/env_ocp$ echo "export LOCAL_REPOSITORY='ocp/openshift4'" >> ${REGISTRY_BASE}/downloads/tools/env_ocp$ echo "export PRODUCT_REPO='openshift-release-dev'" >> ${REGISTRY_BASE}/downloads/tools/env_ocp$ echo 'export LOCAL_SECRET_JSON="${REGISTRY_BASE}/downloads/secrets/pull-secret-bundle.json"' >> ${REGISTRY_BASE}/downloads/tools/env_ocp$ echo 'export RELEASE_NAME="ocp-release"' >> ${REGISTRY_BASE}/downloads/tools/env_ocp

Start the Mirroring

Now let’s start the “oc release” mirroring :

$ oc adm -a ${LOCAL_SECRET_JSON} release mirror \
--from=quay.io/${PRODUCT_REPO}/${RELEASE_NAME}:${OCP_RELEASE} \
--to=${LOCAL_REGISTRY}/${LOCAL_REPOSITORY} \
--to-release-image=${LOCAL_REGISTRY}/${LOCAL_REPOSITORY}:${OCP_RELEASE} \
2>&1 | tee ${REGISTRY_BASE}/downloads/secrets/mirror-output.txt

Generating the openshift-install binary

This part is the most important part of the installation so don’t skip it !!!
In order to create an installation program which is based on the content and name of the registry you’ve just mirrored we will run the “oc” command which in result will generate the “openshift-install” binary to our needs.

$ cd ${REGISTRY_BASE}/downloads/tools/$ oc adm -a ${LOCAL_SECRET_JSON} release extract --command=openshift-install "${LOCAL_REGISTRY}/${LOCAL_REPOSITORY}:${OCP_RELEASE}"$ echo $?

Install config

Now we can create our install-config.yaml file which will be needed for our installation process, the reason that we are doing it now is to save us a few typos and to make sure we have everything we need from the internet to our Air Gaped environment

NOTE!!!

The file name must be "install-config.yaml".
This is the file our installation command expects to read from.
This is how the file should look like:

$ cd ${REGISTRY_BASE}/downloads/tools$ cat > install-config.yaml << EOF
apiVersion: v1
baseDomain: example.com
controlPlane:
name: master
hyperthreading: Enabled
replicas: 3
compute:
- name: worker
hyperthreading: Enabled
replicas: 3
metadata:
name: test-cluster
networking:
clusterNetworks:
- cidr: 10.128.0.0/14
hostPrefix: 23
machineNetwork:
- cidr: 172.18.0.0/16
networkType: OpenShiftSDN
serviceNetwork:
- 172.30.0.0/16
platform:
none: {}
fips: false
pullSecret: '{"auths": ...}'
sshKey: 'ssh-ed25519 AAAA...'
additionalTrustBundle: |
-----BEGIN CERTIFICATE-----
<...base-64-encoded, DER - CA certificate>
-----END CERTIFICATE-----
EOF

Saving the Registry

After we completed the export and generated the binary files the only thing that is left is making sure we are working with the same registry on the internal Server as we work with the the external server so far.
In order to achieve that we simple export the registry to a tar file and save it in our REGISTRY_BASE directory but first we will stop the registry:

$ podman stop my-registry

$ podman rm --force my-registry
$ podman save docker.io/library/registry:2 -o ${REGISTRY_BASE}/downloads/images/registry.tar

Generating the TAR files

Since we put everything under one directory, all we need to do is TAR it to a single file, create a checksum of this file and split the file so it will be easier to import it to the air gaped environment :

$ cd ${REGISTRY_BASE}$ tar -zcf ocp43-registry.tar.gz *$ md5sum ocp43-registry.tar.gz
8103e6d50b622c0879b602f187d26327 ocp43-registry.tar.gz
$ split -b 1G ocp43-registry.tar.gz "ocp43-registry.tar.gz.part"

Generating the 7zip files

It is much more easier to split the file with 7zip

$ 7za a -t7z -v1g -m0=lzma -mx=9 -mfb=64 -md=32m -ms=on ocp43-registry.7z ${BASE_REGISTRY}

Deploying the Registry internally

This tutorial is continuing from the point after all the files are in the “Air Gap” environment.
Everything we do from now on will be done on the internal Server!!!
Which will also be referred as the installation (or bastion) server

$ORIGIN exapmle.com.
ntp A RECORD
registry A RECORD
bastion A RECORD
bootstrap A RECORD
master-01 A RECORD
master-02 A RECORD
master-03 A RECORD
worker-01 A RECORD
worker-02 A RECORD
worker-03 A RECORD
$ORIGIN ocp4.exmaple.com.
control-plane-0 A RECORD (master-01)
control-plane-1 A RECORD (master-02)
control-plane-2 A RECORD (master-03)
etcd-0 A RECORD (master-01)
etcd-1 A RECORD (master-02)
etcd-2 A RECORD (master-03)
_etcd-server-ssl._tcp IN SRV 0 10 2380 etcd-0
_etcd-server-ssl._tcp IN SRV 0 10 2380 etcd-1
_etcd-server-ssl._tcp IN SRV 0 10 2380 etcd-2
ocp4-bootstrap A RECORD (bootstrap)
bootstrap-0 A RECORD
api A RECORD (Load Balancer VIP)
api-int A RECORD (Load Balancer VIP)
$ORIGIN apps.ocp4.example.com.
* A RECORD (Load Balancer VIP)
$ cat ocp43-registry.tar.gz.part* > ocp43-registry.tar.gz
#(you can skip this part if you running 7zip)
$ md5sum ocp43-registry.tar.gz
8103e6d50b622c0879b602f187d26327 ocp43-registry.tar.gz
$ yum install -y jq ftp openssl iperf3 weechat p7zip curl wget tftp telnet podman httpd-tools tcpdump nmap net-tools screen tmux bind-utils nfs-utils sg3_utils nmap-ncat rlwrap uucp openldap-clients xorg-x11-xauth wireshark unix2dos unixODBC policycoreutils-python-utils vim-*

NOTE

Make sure you have at list 15G of space available in your destination directory:

$ mkdir /opt/registry$ export REGISTRY_BASE=/opt/registry$ tar -zxvf ocp43-registry.tar.gz -C ${REGISTRY_BASE}#(or '7za x ocp43-registry.7z*’)
$ source ${REGISTRY_BASE}/downloads/tools/env_ocp
$ cd ${REGISTRY_BASE}/certs
$ rm -f domain.*
$ cat > csr_answer.txt << EOF
[req]
default_bits = 4096
prompt = no
default_md = sha256
x509_extensions = req_ext
req_extensions = req_ext
distinguished_name = dn

[ dn ]
C=US
ST=New York
L=New York
O=MyOrg
OU=MyOU
emailAddress=me@working.me
CN = registry

[ req_ext ]
subjectAltName = @alt_names

[ alt_names ]
DNS.1 = registry
DNS.2 = registry.example.com
DNS.3 = registry.example.local
EOF
$ openssl -newkey rsa:4096 -nodes -sha256 -keyout domain.key -out \ domain.csr -config<( cat csr_answer.txt)
$ openssl req -in domain.csr -noout -text | grep DNS
$ cat new-cert.crt ca.crt > ${REGISTRY_BASE}/certs/domain.crt
$ podman load -i ${REGISTRY_BASE}/downloads/images/registry.tar $ podman image list REPOSITORY   TAG      IMAGE ID       CREATED       SIZE
<none> <none> 708bc6af7e5e 6 weeks ago 26.3 MB
$ podman tag 708bc6af7e5e docker.io/library/registry:2$ ${REGISTRY_BASE}/downloads/tools/start_registry.sh
$ firewall-cmd --add-port=5000/tcp --zone=${FIREWALLD_DEFAULT_ZONE} --permanent$ firewall-cmd --reload

Service

Because it is our internal server and OpenShift needs the registry available

NOTE!

If you are getting an errors at this point make sure that docker is not running OR you can run the same command with docker if you wish.
This part can be a bit tricky but is very important in this tutorial.

$ curl -u myuser:mypassword -k https://registry:5000/v2/_catalog 
{"repositories":[ocp/openshift4]}

Installation

For the installation it self I recommend creating a new user rather then using root , mostly because it is bast practice and to be more specific the rest of the installation process does not require root privileges so there is no need of using it.

$ useradd ocp

ACL

The first thing that we need to do is to make sure our user “ocp” has access to the directories (and sub directories) we created in our infrastructure (read part 1 for more information)

$ setfacl -m u:ocp:rwx -R ${REGISTRY_BASE}
$ setfacl -d -m u:ocp:rwx -R ${REGISTRY_BASE}
$ mkdir /var/www/html/pub
$ mkdir /var/www/html/pub/{pxe,ign}
$ setfacl -m u:ocp:rwx -R /var/www/html/pub
$ setfacl -d -m u:ocp:rwx -R /var/www/html/pub

alternatives

In our world development is moving fast forward, so we need a simple tool to keep us (operators) focus on the jobs and not on the versioning , which is where alternative comes into play.
The tools gives us the ability to run multiple versions of the same binary and set a different default depending on our needs. more so it is a great tracking mechanism for what has been installed and used.

$ mkdir /opt/openshift
$ mkdir /opt/openshift/bin
$ source ${REGISTRY_BASE}/downloads/tools/env$ cp ${REGISTRY_BASE}/downloads/tools/oc /opt/openshift/bin/oc-${OCP-RELEASE}$ cp ${REGISTRY_BASE}/downloads/tools/kubectl /opt/openshift/bin/kubectl-${OCP-RELEASE}$ cp ${REGISTRY_BASE}/downloads/tools/openshift-install /opt/openshift/bin/openshift-install-${OCP-RELEASE}
$ alternatives --install /usr/bin/oc oc /opt/openshift/bin/oc-${OCP-RELEASE} 10$ alternatives --install /usr/bin/kubectl kubectl /opt/openshift/bin/kubectl-${OCP-RELEASE} 10$ alternatives --install /usr/bin/openshift-install openshift-install /opt/openshift/bin/openshift-install-${OCP-RELEASE} 10

Bash auto completion

To make our life easier the tools are been deployed with a set os templates to enable use to use those tools with bash auto completion.
To generate the bash auto completion scripts run the following command :

$ yum install -y bash-completion.noarch bash-completion-extras.noarch$ oc completion bash > /etc/bash_completion.d/oc$ openshift-install completion bash > /etc/bash_completion.d/openshift-install
(Log out and Login for usage)

Install config

To get the installation running we need to switch to our new user and start a screen session:

$ su - ocp 
$ screen -S ocp

Generate SSH key

One of the keys we need to add to the installation template is a public ssh key of the user which will be able to login with the “core” users to our cluster servers.
In order to generate the key run the ssh-keygen coomand :

$ ssh-keygen -t rsa -N '' -f ~/.ssh/id_rsa
$ mkdir ocp4
$ cd ocp4
$ INTERNAL_REPO_NAME=`curl -u myuser:mypassword -k https://registry:5000/v2/_catalog | jq .repositories | grep ocp`
#(or export INTERNAL_REPO_NAME="ocp/openshift4" )
$ echo $INTERNAL_REPO_NAME
$ ocp/openshift4

Optional

If you have other repositories and you see a comma at the end of the output , then you can remove it with the following command :

$ INTERNAL_REPO_NAME=`echo ${INTERNAL_REPO_NAME} | sed 's/\,//'`
$ cp ${REGISTRY_BASE}/downloads/tools/install-config.yaml ~/ocp4/
apiVersion: v1
baseDomain: example.com
controlPlane:
name: master
hyperthreading: Enabled
replicas: 3
compute:
- name: worker
hyperthreading: Enabled
replicas: 3
metadata:
name: test-cluster
networking:
clusterNetworks:
- cidr: 10.128.0.0/14
hostPrefix: 23
machineNetwork:
- cidr: 172.18.0.0/16
networkType: OpenShiftSDN
serviceNetwork:
- 172.30.0.0/16
platform:
none: {}
fips: false
pullSecret: '{"auths":{"registry:5000":{"auth":"bXl1c2VyOm15cGFzc3dvcmQ=","email":"me@working.me"}}}'
sshKey: '< Your Public SSH Key>'
additionalTrustBundle: |
-----BEGIN CERTIFICATE-----
<YOUR CA certificate>
-----END CERTIFICATE-----
imageContentSources:
- mirrors:
- registry:5000/ocp/openshift4
source: quay.io/openshift-release/ocp-release
- mirrors:
- registry:5000/ocp/openshift4
source: quay.io/openshift-release/ocp-v4.0-art-dev

NOTE !!!

  1. The Domain that should contian all our A and SRV records is “test-cluster.example.com” but if your DHCP is setting the hostnames with the format of “host.example.com” that is valid as well.
  2. The “imageContentSources” is taken from the mirror-output.txt file and should be exactly as it shown in the file (make sure you fix the indentation part of the file).
  3. When we update the custom CA we need to make sure we are using the right indentation which means 5 spaces from the left.
  4. your pullSecret should now be your pull secret file of your internal registry only.
    if you changed the username/password of the registry then you should regenerate the pull-secret-registry.json as you did on the external server. In case you didn’t you can just take it as a oneline output

Lazzy

If you are lazzy like and you want to create the file quickly just run the following command:

$ export CLUSTER_NAME="test-cluster"$ export CLUSTER_DOMAIN="example.com"$ cat > install-config.yaml << EOF
apiVersion: v1
baseDomain: ${CLUSTER_DOMAIN}
controlPlane:
name: master
hyperthreading: Disabled
replicas: 3
compute:
- name: worker
hyperthreading: Disabled
replicas: 3
metadata:
name: ${CLUSTER_NAME}
networking:
clusterNetworks:
- cidr: 10.128.0.0/14
hostPrefix: 23
machineNetwork:
- cidr: 172.18.0.0/16
networkType: OpenShiftSDN
serviceNetwork:
- 172.30.0.0/16
platform:
none: {}
fips: false
EOF
$ REG_SECRET=`echo -n 'myuser:mypassword' | base64 -w0`$ echo -n "pullSecret: '" >> install-config.yaml && echo '{ "auths": {}}' | jq '.auths += {"registry:5000": {"auth": "REG_SECRET","email": "me@working.me"}}' | sed "s/REG_SECRET/$REG_SECRET/" | jq -c . | sed "s/$/\'/g" >> install-config.yaml
$ echo -n "sshKey: '" >> install-config.yaml && cat ~/.ssh/id_rsa.pub | sed "s/$/\'/g" >> install-config.yaml
$ echo "additionalTrustBundle: |" >> install-config.yaml$ cat ca.crt | sed 's/^/\ \ \ \ \ /g' >> install-config.yaml
$ cat ${REGISTRY_BASE}/downloads/secrets/mirror-output.txt | grep -A7 imageContentSources >> install-config.yaml

backup

There is a very probable chance you will need to run this installation more then once. in order to save time keep a backup of your install-config.yaml file in your home directory:

$ cp install-config.yaml ~/install-config.yaml.bck

Installation begin

Generate the Kubernetes manifests for the cluster:

$ openshift-install create manifests --dir=./
$ openshift-install create ignition-configs --dir=./
openshift-install — ignition
$ cp *.ign /var/www/html/pub/ign/
$ chmod a+r /var/www/html/pub/ign/*.ign

PXE Install

Now create a new directory under the tftpboot directory

$ mkdir /var/lib/tftpboot/rhcos/
$ mkdir /var/lib/tftpboot/pxelinux.cfg/
$ cp ${REGISTRY_BASE}/downloads/images/rhcos-${OCP_ISO_VERSION}-x86_64-live-initramfs.x86_64.img /var/lib/tftpboot/rhcos/rhcos-initramfs.img$ cp ${REGISTRY_BASE}/downloads/images/rhcos-${OCP_ISO_VERSION}-x86_64-live-kernel.x86_64 /var/lib/tftpboot/rhcos/rhcos-kernel
$ cp ${REGISTRY_BASE}/downloads/images/rhcos-${OCP_ISO_VERSION}-x86_64-metal.x86_64.raw.gz /var/www/html/pub/pxe/rhcos-metal.raw.gz
$ semanage fcontext -a -t httpd_sys_content_t "/var/www/html/pub(/.*)?"
$ restorecon -R -v /var/www/html/pub
$ cat > /var/lib/tftpboot/pxelinux.cfg/bootstrap << EOF
DEFAULT pxeboot
TIMEOUT 5
PROMPT 0
LABEL pxeboot
KERNEL rhcos/rhcos-kernel
APPEND ip=dhcp rd.neednet=1 initrd=rhcos/rhcos-initramfs.img console=tty0 console=ttyS0 coreos.inst=yes coreos.inst.install_dev=sda coreos.inst.image_url=http://<HTTP_server>/pub/pxe/rhcos-metal.raw.gz coreos.inst.ignition_url=http://<HTTP_server>/pub/ign/bootstrap.ign
EOF
$ cat > /var/lib/tftpboot/pxelinux.cfg/master << EOF
DEFAULT pxeboot
TIMEOUT 5
PROMPT 0
LABEL pxeboot
KERNEL rhcos/rhcos-kernel
APPEND ip=dhcp rd.neednet=1 initrd=rhcos/rhcos-initramfs.img console=tty0 console=ttyS0 coreos.inst=yes coreos.inst.install_dev=sda coreos.inst.image_url=http://<HTTP_server>/pub/pxe/rhcos-metal.raw.gz coreos.inst.ignition_url=http://<HTTP_server>/pub/ign/master.ign
EOF
$ cat > /var/lib/tftpboot/pxelinux.cfg/worker << EOF
DEFAULT pxeboot
TIMEOUT 5
PROMPT 0
LABEL pxeboot
KERNEL rhcos/rhcos-kernel
APPEND ip=dhcp rd.neednet=1 initrd=rhcos/rhcos-initramfs.img console=tty0 console=ttyS0 coreos.inst=yes coreos.inst.install_dev=sda coreos.inst.image_url=http://<HTTP_server>/pub/pxe/rhcos-metal.raw.gz coreos.inst.ignition_url=http://<HTTP_server>/pub/ign/worker.ign
EOF
mac-pxe-update() {
ln -s $1 $(echo "$2" | sed 's/^/01-/g' | sed 's/:/-/g')
}
mac-pxe-update bootstrap <BOOTSTRAP MAC> #(and so on for all the MACs)

Boot the Server

now that everything is set you can boot all the server by the following order

  1. masters
  2. workers

NOTE!!!

Do not continue to the next server unless you make sure the server booted with no errors.

$ ssh core@bootstrap

timezone

Check that all RHCOS and installer machines timezone is set to Asia/Jerusalem or to any other Timezone, and that time synchronized

$ timedatectl
Local time: Tue 2019–12–24 15:10:05 IST
Universal time: Tue 2019–12–24 13:10:05 UTC
RTC time: Tue 2019–12–24 13:10:04
Time zone: Asia/Jerusalem (IST, +0200)
System clock synchronized: yes
NTP service: active
RTC in local TZ: no
$ sudo timedatectl set-timezone Asia/Jerusalem

registry testing

this is a very important point , make sure you are able to access your registry from the bootstrap server, this test will save you a lot of time (and frustration) later on .

$ curl -u myuser:mypassword https://registry:5000/v2/_catalog

Static IP

In case your environment has an issue with DHCP then this is the point where you need to configure it locally before you continue with the installation.

$ exit

Openshift-install (continue)

now we will first run the bootstrap installation :

$ openshift-install --dir=./ wait-for bootstrap-complete --log-level debug
INFO Waiting up to 30m0s for the Kubernetes API at https://api.ocp4.example.com:6443...
INFO API v1.13.4+b626c2fe1 up
INFO Waiting up to 30m0s for the bootstrap-complete event…
$ ssh core@bootstrap "journalctl -xe"

Logging into the cluster

$ export KUBECONFIG=/home/ocp/ocp/auth/kubeconfig
$ oc whoami
system:admin

Approving the CSRs for your machines

When you add machines to a cluster, two pending certificate signing request (CSRs) are generated for each machine that you added. You must confirm that these CSRs are approved or, if necessary, approve them yourself.
Confirm that the cluster recognizes the machines:

$ oc get node
NAME STATUS ROLES AGE VERSION
master-0 Ready master 63m v1.13.4+b626c2fe1
master-1 Ready master 63m v1.13.4+b626c2fe1
master-2 Ready master 64m v1.13.4+b626c2fe1
worker-0 NotReady worker 76s v1.13.4+b626c2fe1
worker-1 NotReady worker 70s v1.13.4+b626c2fe1

NOTE

if you only see the masters that means you need to approve the CSR for the worker nodes. Once you approve them you will see the Workers in “NotReady” state at the beginning.
This is a normal behavior.

$ oc get csr
NAME AGE REQUESTOR CONDITION
csr-8b2br 15m system:serviceaccount:openshift-machine-config-operator:node-bootstrapper Pending
$ oc adm certificate approve <csr_name>
$ oc get csr -o name | xargs oc adm certificate approve

Initial Operator configuration

$ watch -n5 oc get clusteroperators
$  oc patch configs.imageregistry.operator.openshift.io cluster --type merge --patch '{“spec”:{“storage”:{“emptyDir”:{}}}}'

completing the installation

Now to complete the installation run :

$ openshift-install --dir=./ wait-for install-complete | tee install-complete

Open Source contributer for the past 15 years

Open Source contributer for the past 15 years