Open in app

Sign in

Write

Sign in

OpenShift 4 Install — Mirroring images for an enterprise registry

Oren Oichman
8 min readMay 6, 2020

--

why this tutorial

I have being working with OpenShift 4 in a disconnected environment on multiple deployments and the same question keeps coming up. “Can I use my internal enterprise registry for the installation” and my answer is “Yes, and I think you should …”.

OpenShift needs access quay.io even after deployment for day 2 day work so you should treat you registry in the same impact tear as your OpenShift.

Sense your enterprise registry as a production top tear server ( at least you should) then it is a given choice to be the deployment registry.

What do you need

You don’t need to install your registry in an external server , you can use a simple docker registry with a few tweaks.
This tutorial already assume that you are running a quay/harbor/artifactory enterprise registry in your organization .

The hosts file

In a standard installation all we need is to setup to the word “registry” to be resolved to localhost (127.0.0.1). In this case we will need to provide to the registry an FQDN ( fully qualified domain name) of the internal domain.

Despite what it seams it is a very small security flow witch is affected only if your server gets hacked when you sync the images (very small chance) so that gives a very short time (between 20 to 40 minutes) in witch your internal domain may be compromised.

so go ahead and edit the file :

$ cat >> /etc/hosts << EOF
127.0.0.1      registry.example.local
EOF

After the sync is completed we will delete this row.

Registry

First let’s create a base directory for the repository on the external server.
For the purpose of this document I will refer to this server as “external”

On the external server run the following command :

$ mkdir /opt/registry
$ export REGISTRY_BASE="/opt/registry"

Now lets create the directories we need for the repository and everything we will want to take to the internal server

$ mkdir -p ${REGISTRY_BASE}/{auth,certs,data,downloads}
$ mkdir -p ${REGISTRY_BASE}/downloads/{images,tools,secrets}

The certificate

Next we will generate a self signed certificate with the required fqdn but first lets start a “screen” or tmux session in case we will get disconnected our session will continue

$ screen -S ocp
$ export REGISTRY_BASE="/opt/registry"

Our registry will need to work over SSL so we have 2 choices the long way (with a certificate request) or the short way (self signed certificate).
I will pick the short way because other then the sync itself will are not going to use this certificate (unless you do then I would prefer the long way)

$ cd ${REGISTRY_BASE}/certs/
$ cat >csr_answer.txt << EOF
[req]
default_bits = 4096
prompt = no
default_md = sha256
distinguished_name = dn
[ dn ]
C=US
ST=New York
L=New York
O=MyOrg
OU=MyOU
emailAddress=me@working.me
CN = registry.example.local
EOF

Change the values under the DN section as you see fit (here it does not really matter execpt for the CN ).

NOTE !
If your registry internal has a different name then you should apply it here and it the /etc/hosts section.

Now lets generate the self signed certificate:

$ openssl req -newkey rsa:4096 -nodes -sha256 -keyout domain.key -x509 -days 1825 -out domain.crt -config <( cat csr_answer.txt )

The output of this command will be 2 new files which we will use for our registry’s SSL certificate:

$ ls -al
total 20
drwxr-xr-x. 2 root root 4096 Jan 8 13:49 .
drwxr-xr-x. 7 root root 4096 Jan 8 09:57 ..
-rw-r — r — . 1 root root 175 Jan 8 13:48 csr_answer.txt
-rw-r — r — . 1 root root 1972 Jan 8 13:49 domain.crt
-rw-r — r — . 1 root root 3272 Jan 8 13:49 domain.key

First let’s create a username and password

Generate a username and password (must use bcrypt formatted passwords), for access to your registry.

$ htpasswd -bBc ${REGISTRY_BASE}/auth/htpasswd admin admin

This tutorial assume that you are running SElinux and firewalld as a secure server.
At this point we will make sure the port is open with the firewall-cmd tool:

$ export FIREWALLD_DEFAULT_ZONE=`firewall-cmd --get-default-zone`
$ echo ${FIREWALLD_DEFAULT_ZONE}
public

My output is “public” but it can be “dmz” , “internal” or “public” for you.

Make sure to open port 5000 on your host, as this is the default port for the registry.

$ firewall-cmd --add-service=https --zone=${FIREWALLD_DEFAULT_ZONE} --permanent
$ firewall-cmd --add-service=http --zone=${FIREWALLD_DEFAULT_ZONE} --permanent$ firewall-cmd --reload

Now we will build a startup script for the registry.

I would recommend you put this in a shell script under ${REGISTRY_BASE}/downloads/tools so it will be easy to run it again in the internal server:

$ echo 'podman run --name my-registry -d -p 443:5000 \
-v ${REGISTRY_BASE}/data:/var/lib/registry:z \
-v ${REGISTRY_BASE}/auth:/auth:z -e "REGISTRY_AUTH=htpasswd" \
-e "REGISTRY_AUTH_HTPASSWD_REALM=Registry" \
-e "REGISTRY_HTTP_SECRET=ALongRandomSecretForRegistry" \
-e REGISTRY_AUTH_HTPASSWD_PATH=/auth/htpasswd \
-v ${REGISTRY_BASE}/certs:/certs:z \
-e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt \
-e REGISTRY_HTTP_TLS_KEY=/certs/domain.key \
docker.io/library/registry:2' > ${REGISTRY_BASE}/downloads/tools/start_registry.sh

I am using “echo” here instead of “cut” because I want to preserve our variables with in the command.
The reason for that is to allow us to select a different Directory base for our internal registry.

Now change the file permission and run it :

$ chmod a+x ${REGISTRY_BASE}/downloads/tools/start_registry.sh
$ ${REGISTRY_BASE}/downloads/tools/start_registry.sh

make sure the external server has the certificate to validate with the registry :

$ cp ${REGISTRY_BASE}/certs/domain.crt /etc/pki/ca-trust/source/anchors/
$ update-ca-trust extract

NOTE
We are selecting to redirect port 443 (witch is the default for HTTPS) to port 5000 witch is the registry listening port.

Once the registry is running we need to make sure it working Verify connectivity to your registry with curl. Provide it the username and password you created.

$ curl -u admin:admin https://registry.example.local/v2/_catalog 
{"repositories":[]}

This should return an “empty” repository for now.

Next we will update the pull-secret.json file and add our registry.
(assuming you had already download it from try.openshift.com)First we will generate a base64 of the username and password:

$ REG_SECRET=`echo -n 'admin:admin' | base64 -w0`

and now lets create a bundle json file with all the registries.

$ export FQDN_REGISTRY="registry.example.local"$ cat pull-secret.json | jq '.auths += {"FQDN_REGISTRY": {"auth": "REG_SECRET","email": "me@working.me"}}' | sed "s/REG_SECRET/$REG_SECRET/" | sed "s/FQDN_REGISTRY/$FQDN_REGISTRY/" > pull-secret-bundle.json

Setup the right environment variables.

$ export LOCAL_REGISTRY='registry.example.local'$ export OCP_RELEASE="${OCP_RELEASE}-x86_64" $ export LOCAL_REPOSITORY='ocp/openshift4' $ export PRODUCT_REPO='openshift-release-dev' $ export LOCAL_SECRET_JSON="${REGISTRY_BASE}/downloads/secrets/pull-secret-bundle.json" $ export RELEASE_NAME="ocp-release"

And start the mirror process.

$ oc adm -a ${LOCAL_SECRET_JSON} release mirror \
--from=quay.io/${PRODUCT_REPO}/${RELEASE_NAME}:${OCP_RELEASE} \
--to=${LOCAL_REGISTRY}/${LOCAL_REPOSITORY} \
--to-release-image=${LOCAL_REGISTRY}/${LOCAL_REPOSITORY}:${OCP_RELEASE} \
2>&1 | tee ${REGISTRY_BASE}/downloads/secrets/mirror-output.txt

This process should take a between an hour or two , depending on your internet bandwidth.

Generating the openshift-install binary

This part is the most important part of the installation so don’t skip it !!!
In order to create an installation program which is based on the content and name of the registry you’ve just mirrored we will run the “oc” command which in result will generate the “openshift-install” binary to our needs.

$ cd ${REGISTRY_BASE}/downloads/tools/$ oc adm -a ${LOCAL_SECRET_JSON} release extract --command=openshift-install "${LOCAL_REGISTRY}/${LOCAL_REPOSITORY}:${OCP_RELEASE}"$ echo $?

This binary named “openshift-install” will be the command for the installation itself.
(basically we are telling openshift-install to work with our internal registry)
The “echo $?” at that point should print the “0” output which tell us that the command was succeeded.

Install config

Now we can create our install-config.yaml file which will be needed for our installation process, the reason that we are doing it now is to save us a few typos and to make sure we have everything we need from the internet to our Air Gaped environment

NOTE!!!

The file name must be "install-config.yaml".
This is the file our installation command expects to read from.
This is how the file should look like:

$ cd ${REGISTRY_BASE}/downloads/tools$ cat > install-config.yaml << EOF
apiVersion: v1
baseDomain: example.com
controlPlane:
  name: master
  hyperthreading: Disabled  
  replicas: 3
compute:
- name: worker
  hyperthreading: Disabled
  replicas: 3
metadata:
  name: test-cluster
networking:
  clusterNetworks:
  - cidr: 10.128.0.0/14
    hostPrefix: 23
  machineNetwork:
  - cidr: 172.18.0.0/16
  networkType: OpenShiftSDN
  serviceNetwork:
  - 172.30.0.0/16
platform:
  none: {} 
fips: false
pullSecret: '{"auths": ...}'
sshKey: 'ssh-ed25519 AAAA...'
additionalTrustBundle: |
     -----BEGIN CERTIFICATE-----
     <...base-64-encoded, DER - CA certificate>
     -----END CERTIFICATE-----
EOF

That is all we need for now , the rest we will generate from the output of our mirroring command and from our internal CA certificate and our SSH public key.

Saving the Registry

After we completed the export and generated the binary files the only thing that is left is making sure we are working with the same registry on the internal Server as we work with the the external server so far.
In order to achieve that we simple export the registry to a tar file and save it in our REGISTRY_BASE directory but first we will stop the registry:

$ podman stop my-registry
 
$ podman rm --force my-registry$ podman save docker.io/library/registry:2 -o ${REGISTRY_BASE}/downloads/images/registry.tar

Generating the 7zip files

It is much more easier to split the file with 7zip

$ 7za a -t7z -v1g -m0=lzma -mx=9 -mfb=64 -md=32m -ms=on ocp43-registry.7z ${BASE_REGISTRY}

That will generate a 1G files for the registry directory.

By the end of this part we are done with the external server.

Internally.

Once we introduce all of the zip files internally. We can unzip them ( make sure your destination directory has around 15 Giga byte of free space.

$ mkdir /opt/registry$ export REGISTRY_BASE=/opt/registry$ cd $REGISTRY_BASE$ 7za x ocp43-registry.7z*

Syncing internally.

For the syncing part we will use a tool called scopeo so let’s install it:

$ yum install skopeo

Now comes the tricky part. We will setup another registry (we will call it intermediate registry) and first sync all the images to that registry

Why?

Well it is simple and complicated …
We synced our images to the internal registry name so we can’t really change that and still use the registry so we will use an intermediate registry. Change the images names and then change them back again when we are resyncing to our enterprise registry…

So the steps are :

  1. External registry→ intermediate
  2. Intermediate→ real registry

First step

Modify the /etc/host in the internal server because it is the only server that needs access to it’s internal registry on the enterprise’s FQDN.

# cp /etc/hosts /tmp/hosts$ cat >> /etc/hosts << EOF
127.0.0.1      registry.example.local
EOF

firewall

$ export FIREWALLD_DEFAULT_ZONE=`firewall-cmd --get-default-zone`
$ echo ${FIREWALLD_DEFAULT_ZONE}
public

and open the ports:

$ firewall-cmd --add-port=5000/tcp --zone=${FIREWALLD_DEFAULT_ZONE} --permanent$ firewall-cmd --reload

Copy the CA certificate :

$ cp ${REGISTRY_BASE}/certs/domain.crt /etc/pki/ca-trust/source/anchors/$ update-ca-trust extract

start the registry

$ ${REGISTRY_BASE}/downloads/tools/start_registry.sh

SYNCING

First let’s make sure we are able to connect to both registries and create a file that holds the credentials :

$ mkdir $HOME/.containers/
$ echo '{"auths": {}}' > $HOME/.containers/auths.json
$ export REGISTRY_AUTH_FILE="$HOME/.containers/auths.json"

Login to all of them :

$ podman login registry:5000
....
$ podman login org-registry # replace with your Org registry

Now let’s sync all the image for the first time

$ skopeo copy --all --authfile $HOME/.containers/auths.json  docker://registry:5000/ocp/openshift4 docker://org-registry.example.com/ocp/openshift4

If you want to avoid TLS verify you can add to the command the following arguments :

--src-tls-verify=false --dest-tls-verify=false

Now we can stop the registry and sync everything to the real registry (modify your /etc/hosts again and delete the new rows

# cp /tmp/hosts /etc/hosts

the installation is looking for “registry.example.local” for image source. If your registry has a different FQDN then you need to make sure the SSL it is using is also answering the “registry.example.local” FQDN by adding it to it’s alter DNS names (unless you gave the registry name when you sync the images on the external server).

Stop the registry

run the podman command to stop the registry (NOT registry-int )

$ podman stop my-registry && podman rm my-registry

What next?

Continue with your normal installation and the deployment should go smoothly.

That is it.

If you have any question feel free to respond/ leave a comment.
You can find on linkedin at : https://www.linkedin.com/in/orenoichman
Or twitter at : https://twitter.com/ooichman

Openshift 4
Disconnect
Registry
Enterprise
Sync

--

--

Oren Oichman
Oren Oichman

Written by Oren Oichman

253 Followers
27 Following

Open Source contributer for the past 15 years

No responses yet

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams