OpenShift 4 in an Air Gap (disconnected) environment (Part 3 — customization)

7 min readFeb 20, 2020

Customization

In this part we will go over some more advanced configuration which will help us get our cluster to the requested enterprise grade level.

I will not cover all the scenarios out there but I will touch base on the more commodity requirement such as :

Update cluster with custom RootCA
Generate custom ingress certificate
Configure Infra Server with MachineSet
Configure CVR — Centralized Virtual routing
Cluster internal registry with Object Storage

custom RootCA

In order to use customer’s internal certificates in the cluster we should first update the cluster with our private root CA so internal processes which are connecting through the route will not fail.

There are two simple steps to achieve that goal.

Update the openshift-config namespace with the Root CA certificate.

$ cat > user-ca-bundle.yaml << EOF
apiVersion: v1
data:
  ca-bundle.crt: |
     -----BEGIN CERTIFICATE-----
     <---Root CA Certificate--->
     -----END CERTIFICATE-----
kind: ConfigMap
metadata:
  name: user-ca-bundle
namespace: openshift-config
EOF$ oc create -f user-ca-bundle.yaml

that alone will not be enough so the solution is to update the cluster wide proxy configuration and add the custom CA to it :

$ oc edit proxy/cluster
trustedCA:     
  name: user-ca-bundle

this will force all the pods in the customer to work with the CA we provided in the user-ca-bundle.yaml configMap.

2. Add the Root CA to the trusted-ca-bundle ConfigMap.

$ oc edit cm trusted-ca-bundle -n openshift-config-managed

Copy/Paste the content of the Root CA certificate file at the beginning of the “ca-bundle.crt” data (with consideration for the correct indentation).

Ingress wildcard certificate

We would like to provide the ingress router a wildcard certificate which will verify every request against our CA.

First we want to create the details file for the creation of the csr.

$ cat > csr_details.txt << EOF
[req]
default_bits = 4096
prompt = no
default_md = sha256
x509_extensions = req_ext
req_extensions = req_ext
distinguished_name = dn
[ dn ]
C=US
ST=New York
L=New York
O=MyOrg
OU=MyOU
emailAddress=me@working.me
CN = *.apps.openshift.example.com
[ req_ext ]
subjectAltName = @alt_names
[ alt_names ]
DNS.1 = *.apps.openshift.example.com
DNS.2 = apps.openshift.example.com
EOF

Let’s create the key for the router certificate

$ openssl genrsa -out router.key 4096

Now we will generate the request into CSR file.

$ openssl req -new -sha256 -nodes -key router.key -out router.csr -config <(cat csr_details.txt)

NOTE: Test the output. Verify you can see the alternatives

 $ openssl req -in router.csr -noout -text | grep -i dns

Now you need to generate the certificate file for the router, and create a bundle certificate with the root CA certificate.

the certificate file is been done by our company CA signing our certificate request or we can run a self sign CA and add it as mentioned in section 1 (You can go through the OpenSSL process in my Tutorial Working with OpenSSL and DNS alterNames)

To bundle our certificate and the CA run the “cat” command for both of the files where our certificate is the first file

$ cat router.crt ca.crt > router-bundle.crt

Now all we have left is to attach the certificate bundle to the ingress router, we’ll do it by creating a secret tls that will contain the bundle and the key.

$ oc create secret tls router --cert=./router-bundle.crt --key=./router.key -n openshift-ingress

Finally patch the ingress controller operator to use the router certificate as the new default certificate.

$ oc patch ingresscontrollers.operator default --type=merge -p '{"spec":{"defaultCertificate": {"name": "router"}}}' -n openshift-ingress-operator

Configure Infra Server

In order to label some of our nodes as infra nodes, we need to use machineset.

NOTE: Machineset is an object that requires for an openshift cluster to be aware of it’s cloud infrastructure. In our case, a bare metal installation in a disconnected environment, we are not required to create a machineset for the cluster but we will do it anyway because we want to be able to standardize our work which may be require in future versions.

First, lets discover the infrastructureID

$ oc get -o jsonpath='{.status.infrastructureName}{"\n"}' infrastructure cluster

Save the output in clipboard for the next step.

Now we’ll create the yaml file for the machineset

Paste the output of the last command instead of the <infrastructureID>

Replace <role>with infra with <zone> with default

$ cat > machineset.yaml << EOF
apiVersion: machine.openshift.io/v1beta1
kind: MachineSet
metadata:
  labels:
    machine.openshift.io/cluster-api-cluster: <infrastructureID> 
    machine.openshift.io/cluster-api-machine-role: <role> 
    machine.openshift.io/cluster-api-machine-type: <role> 
  name: <infrastructureID>-<role>-<region> 
  namespace: openshift-machine-api
spec:
  replicas: 1
  selector:
    matchLabels:
      machine.openshift.io/cluster-api-cluster: <infrastructureID> 
      machine.openshift.io/cluster-api-machineset: <infrastructureID>-<role>-<region> 
  template:
    metadata:
      creationTimestamp: null
      labels:
        machine.openshift.io/cluster-api-cluster: <infrastructureID> 
        machine.openshift.io/cluster-api-machine-role: <role> 
        machine.openshift.io/cluster-api-machine-type: <role> 
        machine.openshift.io/cluster-api-machineset: <infrastructureID>-<role>-<region> 
    spec:
      metadata:
        creationTimestamp: null
        labels: node-role.kubernetes.io/<role>: ""
EOF

Now lets create it and tag the nodes we would like to label as “infra”.

$ oc create -f machineset.yaml -n openshift-machine-api

Verify the creation of the machineset

$ oc get machineset -n openshift-machine-api

To apply the labels to the relevant nodes, you need to edit them, and add the following line to the labels section

node-role.kubernetes.io/infra: ""

Verify that the nodes are labeled correctly

$ oc get node <node_name> --show-labels

Configure CVR

Overview

In OCP 4 we switched from CVR (Centralized Virtual Routing) to DVR (Distributed Virtual Routing). In our case because we want to control and maximize our resources we would like to manually enforce CVR.

We can achieve it by configuring router pods to run only on infra-labeled nodes and they will redirect the traffic to the app pods, with no consideration for which nodes they actually run on. You can call it “Manual CVR” if it easier to understand.

CVR Design

DVR Design

Workflow

For that part we will start by moving the ingress-router to run on infrastructure nodes only

$ oc edit ingresscontroller default -n openshift-ingress-operator

add the following lines to the spec

spec:
    nodePlacement:
      nodeSelector:
        matchLabels:
          node-role.kubernetes.io/infra: ""

Verify that the ingress router pods runs on the correct nodes

$ oc get pods -o wide -n openshift-ingress

Note that now your routers will only run on the workers that are tagged with the “infra” label.
In case you want your infrastructure to be identical to Openshift version 3 then you need to remove the label of “workers” from the infra servers.

OpenShift Registry with Object Storage

In a production environment you would not want to keep the internal registry in an “EmptyDir” state… you will want a persistent storage. I order to work with an Object storage we will need to trick the cluster configuration “as if” it is running on AWS when in fact it is in an internal object storage with an s3 implementation

The step are :

Running ceph demo server
Creating a secret for the credentials
Modifying the image registry configuration to work with s3

Ceph demo

first we need to make sure that the Ceph image is available in our local registry (you can use the registry from part 2) and there is nothing running on the server

create the necessary directories for ceph :

$ mkdir /data/
$ mkdir -p /data/etc/ceph/
$ mkdir -p /data/var/lib/ceph/

Now run the following command :

podman run -d — name demo -e MON_IP=1.1.1.1 \
-e CEPH_PUBLIC_NETWORK=1.1.1.1/32 \
-e RGW_NAME=storage.example.com --net=host \
-v /data/var/lib/ceph:/var/lib/ceph:z \
-v /data/etc/ceph:/etc/ceph:z -e CEPH_DEMO_UID=qqq \
-e CEPH_DEMO_ACCESS_KEY=qqq \
-e CEPH_DEMO_SECRET_KEY=qqq \
-e CEPH_DEMO_BUCKET=qqq ceph/daemon demo

as you can notice we are using the server network interface so we need to make sure the following DNS records exists:

$ORIGIN example.com
storage         IN     A 1.1.1.1
*.storage       IN     A 1.1.1.1

Now lets login to the object storage server with the “awscli” tool

$ aws configure
AWS Access Key ID [None]: qqq
AWS Secret Access Key [None]: qqq
Default region name [None]: 
Default output format [None]:

And create a Bucket

$ aws s3 mb s3://ocp4-storage --endpoint-url http://storage.example.com:8080

and test it :

$ aws s3 ls --endpoint-url http://ocp4-storage.storage.example.com:8080'Buckets'

Now that our object storage is set we will continue

AWS secret credentials

In addition to the configs.imageregistry.operator.openshift.io and ConfigMap resources, configuration is provided to the Operator by a separate secret resource located within the openshift-image-registry namespace.

The image-registry-private-configuration-user secret provides credentials needed for storage access and management. It overrides the default credentials used by the Operator, if default credentials were found.

For S3 on AWS storage the secret is expected to contain two keys:

REGISTRY_STORAGE_S3_ACCESSKEY
REGISTRY_STORAGE_S3_SECRETKEY

So let create the secret:

$ oc create secret generic image-registry-private-configuration-user --from-literal=REGISTRY_STORAGE_S3_ACCESSKEY=qqq --from-literal=REGISTRY_STORAGE_S3_SECRETKEY=qqq --namespace openshift-image-registry

Now lets update the image registry configuration so the the “EmptyDir” will be replace with the following fields :

oc edit configs.imageregistry.operator.openshift.io/cluster
  managementState: Managed
  storage:
    s3:
      bucket: ocp4-storage
      encrypt: false
      regionEndpoint: http://storage.example.com:8080

Once everything is set you can see your configuration in the deployment environments :

oc describe deployment image-registry | grep "REGISTRY_STORAGE"

the output should look like :

Now we can switch the registry to manage :

#  oc patch configs.imageregistry.operator.openshift.io cluster --type merge --patch '{"spec":{"managementState":"Managed"}}'

and Expose the registry :

# oc patch configs.imageregistry.operator.openshift.io/cluster --patch '{"spec":{"defaultRoute":true}}' --type=merge