Deploying a Single Operator on OpenShift Disconnected

Why this Tutorial

While working in a Disconnected (Air Gaped) environment and OpenShift 4 we want make the must of our OpenShift. For that we would like to have OLM accessible for us but unfortunately this case is not so easy to achieve and maintain.

For that I would suggest to take one Operator at the time and ask our self what operators do we actually need and pick them up one by one.

In this tutorial

In this tutorial we will take the keepalived operator and make it available in our air gaped environment.

What Do we need ?

for our operation we will need :

1. podman
2. oc (openshift client — command line)
3. running registry (for multiple images operator)
4. skopeo

Operator

For those of you how don’t know in OpenShift we have to ability to extend the services we provide (and much more) using an Operator.

In Openshift we have an Operator Life-cycle Manager (OLM) which is responsible for adding the Operators to our running OpenShift Cluster.

In this part we are going to take the specific Operator, download all what we need for an Air Gaped (Disconnected) environment and then deploy it in to OpenShift.

Operator Life-cycle Manager (OLM) always installs Operators from the latest version of an Operator catalog. As of OpenShift Container Platform 4.3, Red Hat-provided Operators are distributed via Quay App Registry catalogs from quay.io.

redhat-operators

Public catalog for Red Hat products packaged and shipped by Red Hat. Supported by Red Hat.

certified-operators

Public catalog for products from leading independent software vendors (ISVs). Red Hat partners with ISVs to package and ship. Supported by the ISV.

community-operators

Public catalog for software maintained by relevant representatives in the operator-framework/community-operators GitHub repository. No official support.

find the Operator

first think we need is to make a list of all the operators in from all their communities:
We will create a directory for our content :

# export OLM_DIR=/opt/OLM/
# mkdir ${OLM_DIR}
# cd ${OLM_DIR}

To get the list of packages that are available for the default OperatorSources, run the following curl commands from your workstation without network restrictions:

# curl https://quay.io/cnr/api/v1/packages?namespace=redhat-operators > packages.txt
# curl https://quay.io/cnr/api/v1/packages?namespace=community-operators >> packages.txt
# curl https://quay.io/cnr/api/v1/packages?namespace=certified-operators >> packages.txt

Now that we have all of our OLM packages in a single location we can isolate what we need :

First let’s isolate the name with the jq command and the grep commnad :

export the name of the Operator

# export OPERATOR_NAME="keepalived"

grep the name of the operator

# cat packages.txt | jq . | grep $OPERATOR_NAME
"name": "community-operators/keepalived-operator",

Let’s put it in a variable :

# export OPR_NAME=$(cat packages.txt | jq . | grep $OPERATOR_NAME | awk -F \" '{print $4}')
# echo ${OPR_NAME}
# echo "export OPR_NAME=${OPR_NAME}" > env
# export OPR_SORT_NAME=$(cat packages.txt | jq . | grep $OPERATOR_NAME | awk -F \" '{print $4}' | cut -d '/' -f 2)
# echo ${OPR_SORT_NAME}
# echo "export OPR_SORT_NAME=${OPR_SORT_NAME}" >> env

Now that we have the name we can isolated the relevant information with a JSON query :

#  cat packages.txt | jq -r --arg OPR_NAME "${OPR_NAME}" '.[] | select(.name==$OPR_NAME)'
{
  "channels": null,
  "created_at": "2020-03-04T09:00:59",
  "default": "0.2.2",
  "manifests": [
    "helm"
  ],
  "name": "community-operators/keepalived-operator",
  "namespace": "community-operators",
  "releases": [
    "0.2.2",
    "0.2.0",
    "0.1.0"
  ],
  "updated_at": "2020-08-26T14:35:48",
  "visibility": "public"
}

With this output we can see that the default version is “0.2.2” and the catalog community is “community-operators”.

Let’s put the default version in a variable (to make our lives easier)

# OPR_DEFAULT_VER=$(cat packages.txt | jq -r --arg OPR_NAME "${OPR_NAME}" '.[] | select(.name==$OPR_NAME)' | grep default | awk -F \" '{print $4}')
# echo $OPR_DEFAULT_VER
0.2.2
# echo "export OPR_DEFAULT_VER=${OPR_DEFAULT_VER}" >> env

Pull Operator content.

For a given Operator in the package list, you must pull the latest released content:

# curl https://quay.io/cnr/api/v1/packages/${OPR_NAME}/${OPR_DEFAULT_VER}/ | jq .

This will output the package information.

to Extract the relevant digest we can run the following command :

# DIGEST=$(curl https://quay.io/cnr/api/v1/packages/${OPR_NAME}/${OPR_DEFAULT_VER}/ | jq . | grep -B 2 gzip | grep digest | awk -F \" '{print $4}')
# echo $DIGEST
585435fd8b621efb456b7b26882b268f63e9269543db11c1636759987ed1bfe6
# echo "export DIGEST=${DIGEST}" >> env

Now we will use it to pull the gzipped archive:

# mkdir ${OLM_DIR}/TARs
# curl -XGET https://quay.io/cnr/api/v1/packages/${OPR_NAME}/blobs/sha256/${DIGEST} \
    -o ${OLM_DIR}/TARs/${OPR_SORT_NAME}.tar.gz

To pull the information out, you must untar the archive into a manifests/<operator_name>/ directory with all the other Operators that you want. For example, to untar to an existing directory:

# mkdir manifests
# tar -zxvf ${OLM_DIR}/TARs/${OPR_SORT_NAME}.tar.gz -C manifests

Break apart bundle.yaml content, if necessary.

In your new manifests/<operator_name> directory, the goal is to get your bundle in the following directory structure:

manifests/
└── ${OPR_SORT_NAME}
    ├── ${OPR_DEFAULT_VER}
    │   ├── clusterserviceversion.yaml
    │   └── customresourcedefinition.yaml
    └── package.yaml

Now that we made all the necessary changes we will need to address the images.

Identify images required by the Operators you want to use.

Inspect the CSV files of each Operator for image: fields to identify the pull specs for any images required by the Operator, and note them for use in a later step.

For example, in the following deployments spec of an keepalived-operator CSV:

# cd manifests/${OPR_SORT_NAME}*/

Now find the lastest version with the “ls” command and “cd” into it.

# export LATEST_VERSION=$(ls | sort | grep -v package | tail -1)
# cd ${LATEST_VERSION}

Look at the clusterserviceversion.yaml file to see all the needed images.

# cat *clusterserviceversion.yaml | grep image
    mediatype: image/png
                image: quay.io/redhat-cop/keepalived-operator:v0.2.2
                imagePullPolicy: Always

Single Image

In our case the operator needs only 1 image , so lets go ahead and pull it and save is with podman:

# cd ~/OLM
# mkdir images
#  skopeo copy docker://quay.io/redhat-cop/keepalived-operator:v0.2.2 docker-archive:images/keepalived-operator.tar

Multiple Images

auth file

For multiple images we need to set a temporary registry and then user skopeo to copy everything to that registry.

First set the environment variable for the REGISTRY login credentials :

# export REGISTRY_AUTH_FILE=$HOME/.registry/auths.json

And let’s create the directory and file :

# mkdir $HOME/.registry/
# echo '{"auths":{}}' > ~/.registry/auths.json

Create a temporary registry

# export REGISTRY_BASE="${OLM_DIR}/registry"
# mkdir $REGISTRY_BASE
# export REGISTRY_FQDN="registry"

Create the relevant sub directories :

# mkdir -p ${REGISTRY_BASE}/{auth,certs,data,downloads}
# mkdir -p ${REGISTRY_BASE}/downloads/{images,logs,tools,secrets}

A simple but a tricky part , here we will want to call the registry the same name as we would in the internal LAN but we probably do not want to write our internal domain in an external server so we will use the hostname and not FQDN.

We will edit the /etc/hosts file of the external Server and add the “registry” record to it:

# cat >> /etc/hosts << EOF
127.0.0.1      ${REGISTRY_FQDN}
EOF

From now on our registry will be named “registry”.

Next we will create some kind of “answer file” to our self signed certificate:

$ cd ${REGISTRY_BASE}/certs/
$ cat >csr_answer.txt << EOF
[req]
default_bits = 4096
prompt = no
default_md = sha256
x509_extensions = req_ext
req_extensions = req_ext
distinguished_name = dn
[ dn ]
C=US
ST=New York
L=New York
O=MyOrg
OU=MyOrgUnit
emailAddress=me@working.me
CN = ${REGISTRY_FQDN}
[ req_ext ]
subjectAltName = @alt_names
[ alt_names ]
DNS.1 = ${REGISTRY_FQDN}
EOF

change the values under the DN section as you see fit (here it does not really matter)

Now lets generate the self signed certificate:

$ openssl req -newkey rsa:4096 -nodes -sha256 -keyout domain.key -x509 -days 365 -out domain.crt -config <( cat csr_answer.txt )

The output of this command will be 2 new files which we will use for our registry’s SSL certificate:

$ ls -al
total 20
drwxr-xr-x. 2 root root 4096 Jan 8 13:49 .
drwxr-xr-x. 7 root root 4096 Jan 8 09:57 ..
-rw-r — r — . 1 root root 175 Jan 8 13:48 csr_answer.txt
-rw-r — r — . 1 root root 1972 Jan 8 13:49 domain.crt
-rw-r — r — . 1 root root 3272 Jan 8 13:49 domain.key

Also, if needed and you haven’t done so already, make sure you trust the self-signed certificate. This is needed in order for oc to be able to login to your registry during the mirror process.

$ cp ${REGISTRY_BASE}/certs/domain.crt /etc/pki/ca-trust/source/anchors/
$ update-ca-trust extract

Generate a username and password (must use bcrypt formatted passwords), for access to your registry.

$ htpasswd -bBc ${REGISTRY_BASE}/auth/htpasswd myuser mypassword

first get your firewalld zone:

$ export FIREWALLD_DEFAULT_ZONE=`firewall-cmd --get-default-zone`
$ echo ${FIREWALLD_DEFAULT_ZONE}
public

My output is “public” but it can be “dmz” , “internal” or “public” for you.

Make sure to open port 5000 on your host, as this is the default port for the registry.

$ firewall-cmd --add-port=5000/tcp --zone=${FIREWALLD_DEFAULT_ZONE} --permanent
$ firewall-cmd --reload

Now you’re ready to run the container. Here I specify the directories I want to mount inside the container. I also specify I want to run on port 5000 and that I want it in daemon mode.
I would recommend you put this in a shell script under ${REGISTRY_BASE}/downloads/tools so it will be easy to run it again in the internal server:

$ echo 'podman run --name my-registry --rm -d -p 5000:5000 \
-v ${REGISTRY_BASE}/data:/var/lib/registry:z \
-v ${REGISTRY_BASE}/auth:/auth:z -e "REGISTRY_AUTH=htpasswd" \
-e "REGISTRY_AUTH_HTPASSWD_REALM=Registry" \
-e "REGISTRY_HTTP_SECRET=ALongRandomSecretForRegistry" \
-e REGISTRY_AUTH_HTPASSWD_PATH=/auth/htpasswd \
-v ${REGISTRY_BASE}/certs:/certs:z \
-e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt \
-e REGISTRY_HTTP_TLS_KEY=/certs/domain.key \
docker.io/library/registry:2' > ${REGISTRY_BASE}/downloads/tools/start_registry.sh

I am using “echo” here instead of “cut” because I want to preserve our variables with in the command.
The reason for that is to allow us to select a different Directory base for our internal registry.

Now change the file permission and run it :

$ chmod a+x ${REGISTRY_BASE}/downloads/tools/start_registry.sh
$ ${REGISTRY_BASE}/downloads/tools/start_registry.sh

Verify connectivity to your registry with curl. Provide it the username and password you created.

$ curl -u myuser:mypassword -k \
  https://${REGISTRY_FQDN}:5000/v2/_catalog 
{"repositories":[]}

This should return an “empty” repository for now

First login to both registries :

# export REGISTRY_SRC="" # the image external registry
# export REGISTRY_DST="${REGISTRY_FQDN}:5000"
# podman login $REGISTRY_SRC
# podman login $REGISTRY_DST --tls-verify=false

Go back to the Package directory

# cd ${OLM_DIR}/manifests/${OPR_SORT_NAME}*/${LATEST_VERSION}/

Extract the packages names :

the cluster service version contains a list of images the operators needs to work with.

some of the CSV file contain the images in a different so we need first to make sure the output is not empty :

run the “cat *clusterserviceversion.yaml | grep 'image:' | awk '{print $2}'” command. If the output is not the full path of the images then you should change the $2 to $3 as such :
“cat *clusterserviceversion.yaml | grep ‘image:’ | awk ‘{print $3}’

And run the output to a file :

# mkdir ${OLM_DIR}/imageList
# cat *clusterserviceversion.yaml | grep 'image:' | awk '{print $2}' > ${OLM_DIR}/imageList/${OPR_SORT_NAME}-images_list.txt

To Copy all the images with the digest we are going to use scopeo with “-a” flag to make sure we are copying everything we need.

# for image in `cat ${OLM_DIR}/imageList/${OPR_SORT_NAME}-images_list.txt`; do
IMAGE_NAME=$( echo $image | awk -F \/ '{print $3}') 
echo "${image}=${REGISTRY_DST}/${OPR_SORT_NAME}/${IMAGE_NAME}" >>  ${OLM_DIR}/imageList/packageImages-list.txt
skopeo copy -a --authfile $REGISTRY_AUTH_FILE \
--dest-tls-verify=false docker://${image} \
docker://${REGISTRY_DST}/${OPR_SORT_NAME}/${IMAGE_NAME} \
| tee -a ${REGISTRY_BASE}/downloads/logs/skopeo-logs.txt
done

Operator Catalog

First let’s copy the images we need for the build and add them to our temporary registry :

# skopeo copy --all --dest-tls-verify=false --authfile $REGISTRY_AUTH_FILE docker://registry.redhat.io/openshift4/ose-operator-registry:v4.6.0 docker://${REGISTRY_DST}/olm/ose-operator-registry:v4.6.0
# skopeo copy --all --dest-tls-verify=false --authfile $REGISTRY_AUTH_FILE docker://registry.access.redhat.com/ubi8/ubi docker://${REGISTRY_DST}/ubi8/ubi
# skopeo copy --all --dest-tls-verify=false --authfile $REGISTRY_AUTH_FILE docker://registry.access.redhat.com/ubi8/ubi-minimal docker://${REGISTRY_DST}/ubi8/ubi-minimal

Now that we have the image we will need to create the Operator Catalog Image :

In your OLM directory (where we created the manifests directory) save the following to a Dockerfile, for example named custom-registry.Dockerfile

# cd ${OLM_DIR}
# cat > custom-registry.Dockerfile << EOF
FROM registry.redhat.io/openshift4/ose-operator-registry:v4.6.0 AS builder
COPY manifests manifests
RUN /bin/initializer -o ./bundles.db  
FROM registry.access.redhat.com/ubi8/ubi
COPY --from=builder /registry/bundles.db /bundles.db 
COPY --from=builder /usr/bin/registry-server /registry-server 
COPY --from=builder /bin/grpc_health_probe /bin/grpc_health_probe
EXPOSE 50051  
ENTRYPOINT ["/registry-server"]  
CMD ["--database", "bundles.db"]
EOF

Once we created the file we need to generate an image build :

# buildah bud -f custom-registry.Dockerfile -t ${OPR_SORT_NAME}-catalog

Once the build is complete then we will save the image to our directory :

# podman tag localhost/${OPR_SORT_NAME}-catalog ${REGISTRY_DST}/olm/${OPR_SORT_NAME}-catalog
# podman push ${REGISTRY_DST}/olm/${OPR_SORT_NAME}-catalog --tls-verify=false

Now will be a good point to pre configure our YAML file for our source catalog :

# mkdir ${OLM_DIR}/YAML/
# cat > ${OLM_DIR}/YAML/${OPR_SORT_NAME}-catalog.yaml << EOF
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: ${OPR_SORT_NAME}-catalog
  namespace: openshift-marketplace
spec:
  displayName: ${OPR_SORT_NAME} Operator Catalog
  sourceType: grpc
  image: <your local registry>/catalogsource/${OPR_SORT_NAME}-catalog:latest
EOF

To use mirrored images, you must first create an ImageContentSourcePolicy for each image to change the source location of the Operator catalog image. For example:

# cat > ${OLM_DIR}/YAML/${OPR_SORT_NAME}-imageSource.yaml << EOF
apiVersion: operator.openshift.io/v1alpha1
kind: ImageContentSourcePolicy
metadata:
  name: ${OPR_SORT_NAME}
spec:
  repositoryDigestMirrors:
EOF

And we are going to add the images we just copied to our internal registry:

# for LINE in `cat ${OLM_DIR}/imageList/packageImages-list.txt`; do
echo "  - mirrors:" >> ${OLM_DIR}/YAML/${OPR_SORT_NAME}-imageSource.yaml
DST_IMAGE=$(echo $LINE | awk -F \= '{print $2}' | awk -F \@ '{print $1}')
echo "    - $DST_IMAGE" >> ${OLM_DIR}/YAML/${OPR_SORT_NAME}-imageSource.yaml
SRC_IMAGE=$(echo $LINE | awk -F \= '{print $1}' | awk -F \@ '{print $1}')
echo "    source: $SRC_IMAGE" >> ${OLM_DIR}/YAML/${OPR_SORT_NAME}-imageSource.yaml
done

And now stop the registry and save the image as a TAR file :

# podman stop my-registry
# skopeo copy docker://docker.io/library/registry:2 docker-archive:${REGISTRY_BASE}/downloads/images/registry.tar

Now we need to zip everything and take it to our disconnected environment.

# 7za a -t7z -v1g -m0=lzma -mx=9 -mfb=64 -md=32m -ms=on ${OPR_SORT_NAME}.7z ${OLM_DIR}
(you can use TAR and split if you wish)

Once the files are created then we can take them and bring them to our disconnected network.

Disconnected Environment

Once the files are Air Gaped we can unzip them :

# 7za x <file name>.7z*
(if you need more then 1 image
# source OLM/env

Now that we opened the zip files we can get into our directory and start applying our files .

Before starting make sure your existing OLM catalog is disabled :

# oc patch OperatorHub cluster --type json \
    -p '[{"op": "add", \
"path": "/spec/disableAllDefaultSources", "value": true}]'

For the Images we need to import them from the file , tag them and push them to our local registry.

So for example we need to run :

# export LOCAL_REPO=registry.example.local
# podman load -i images/${OPR_SORT_NAME}-catalog.tar
# podman load -i images/keepalived-operator.tar
# podman tag <IMAGE ID> ${LOCAL_REPO}/operators/keepalived-operator:0.2.2
# podman tag <IMAGE ID> ${LOCAL_REPO}/catalogsource/${OPR_SORT_NAME}-catalog:latest
# podman push ${LOCAL_REPO}/operators/keepalived-operator:0.2.2
# podman push ${LOCAL_REPO}/catalogsource/${OPR_SORT_NAME}-catalog:latest

Change the registry for what every you are working on …

Multiple Images

In case you had created a temporary registry for multiple image , now is the time to use it :

# export REGISTRY_BASE="<your internal location>"
# podman load -i ${REGISTRY_BASE}/downloads/images/registry.tar 
# podman image list REPOSITORY   TAG      IMAGE ID       CREATED       SIZE
<none>       <none>   708bc6af7e5e   6 weeks ago   26.3 MB
# podman tag 708bc6af7e5e docker.io/library/registry:2
# {REGISTRY_BASE}/downloads/tools/start_registry.sh

You can use this registry or copy all the images to your enterprise registry with skopeo

# skopeo copy -a --authfile <authentication file> docker://temporary registry> docker://<destination registry>

IMPORTANT !!!

Now we will update our local registry in the ${OPR_SORT_NAME}-catalog.yaml file

Once that is done we can apply our source catalog in our cluster :

# oc create -f ${OPR_SORT_NAME}-catalog.yaml

Verify the CatalogSource and package manifest are created successfully:

# oc get pods -n openshift-marketplace
# oc get catalogsource -n openshift-marketplace
# oc get packagemanifest -n openshift-marketplace

Modify the keepalived-imageSource.yaml file And apply it :

# oc create -f keepalived-imageSource.yaml

And now you can intall the Operator using the OLM Console window…

If you have any question feel free to responed/ leave a comment.
You can find on linkedin at : https://www.linkedin.com/in/orenoichman
Or twitter at : https://twitter.com/ooichman

That is it
Have FUN …