Open in app

Sign in

Write

Sign in

Implementing MTLS with Apache and OpenSSL on OpenShift

Oren Oichman
8 min readApr 29, 2023

Security As Practice

It is a well known fact that today you need to apply (in any way) a Secure communication tunnel between the client and the server so that no one will be able to “tape” to you conversation and look at the data going through it.

Apply A Server Side encryption (E.G TLS) has become a commodity and even the very basic/lite HTTP server has the ability to provide TLS/SSL communication but is some case we need to make sure that the Server trust the client and not only the other way around.

What is MTLS

TLS stands for Tunnel Layer Secure which means that once the Client Authenticates the Server a Secure tunnel is initiated for the Session. MTLS stands for Mutual Tunnel Layer Secure which means that not only does the client needs to authentication the Server but the Server needs to authenticate the Client as well.

SSLv3 Extensions

For the purpose of defining the certificate intended usage SSL v3 (TLS 1.0 is a more advanced version based on SSL v3) comes with a set of extensions which we will go over some of them in this tutorial.

Where to Begin ?

The first basic assumption is that you are running a working OpenShift 4 Cluster and have the OCI tools (E.G podman, buildah ,skopeo…) available on a Linux WS/Server

The Steps we are going to take are :

  1. creating the CA certificate.
  2. creating a certificate for the Server Authentication
  3. creating a certificate for the Client Authentication
  4. creating an Apache container that will validated MTLS.
  5. running CURL with the Client certificate to tests the process

Creating the Certificates

In this part we will create the Server and the Client Certificates using an answer file for OpenSSL in order to create the certificates.

Creating the CA

In Order to Generate the CA certificate we first need to or create an answer file that will define the certificate as A CA certificate or put everything in a long command line.
In the v3 extension we can define a certificate whether it is a CA or not with the key value of :

basicConstraints=CA: {TRUE|FALSE}

In our example we will use the domain “example.com” and the CA will be called “ca.example.com”.

First we will create a directory structure that will help us with managing the certificates:

# mkdir {CA,Keys,Certs,CSR}

Let’s run a single command line to create our CA but first we need to create our CA key:

# openssl genrsa -out CA/ca.key 4096

And now Let’s run the command :

# openssl req -new -x509 -key CA/ca.key -days 730 -out CA/ca.crt  \
-subj "/CN=ca.example.com/O=Example/C=US/ST=Center/L=NY"  \
-addext "basicConstraints=CA:TRUE" -addext "subjectKeyIdentifier=hash" \
-addext "authorityKeyIdentifier=keyid,issuer"

Now we have 2 new files under the CA directory: “ca.crt” and “ca.key”

We can use the openssl command to explor_REF} \e the certificate and make sure it is marked as “CA:TRUE” with the following command :

# openssl x509 -in CA/ca.crt -text -noout | grep CA
                CA:TRUE

Now that we have our CA we can move on and start creating our certificates starting with the Server certificate and then the Client certificate.

Server Certificate

This part is still just like every Other TLS certificate so we will do the same as if we are going to sign a normal certificate.

The steps are create a new key for the Server, creating a “certificate request” and then use the CA we have created in our first step to sign the certificate request. By doing so we will create a certificate which is signed and approved by our CA.

As a good practice I like to create 2 environment variables that will help use the certificate signing. the first is the domain name and the second is the name of the server (AKA short name).

To define the 2 environment variables :

# export DOMAIN="example.com"
# export SHORT_NAME="server"

The main reason I am doing this is to make your job easier so you can set up the variable to suite your request and then just copy/paste the rest of the commands.

Let’s start by generating the Key :

# openssl genrsa -out Keys/${SHORT_NAME}.key 4096

For the next part we need to create a certificate request for our Server and then use the CA certificate (and key) and sign it.

To generate the request :

# openssl req -new -key Keys/${SHORT_NAME}.key -out CSR/${SHORT_NAME}.csr \
 -subj "/CN=${SHORT_NAME}/O=${DOMAIN}/C=US/ST=Center/L=NY"  \
 -addext "subjectAltName = DNS:${SHORT_NAME},DNS:${SHORT_NAME}.${DOMAIN}" \
 -addext "keyUsage=digitalSignature" -addext "basicConstraints=CA:FALSE" \
 -addext "nsCertType=server" -addext "extendedKeyUsage=serverAuth"

Now that we have our Certificate request in place we can go ahead and sign the certificate with our CA :

# openssl x509 -req -in CSR/${SHORT_NAME}.csr -CA CA/ca.crt -CAkey CA/ca.key \
 -CAcreateserial -out Certs/${SHORT_NAME}.crt -days 730 \
 -copy_extensions copy

We can verify that our server certificate was signed by the CA and not self signed with the following command :

# openssl verify -CAfile CA/ca.crt Certs/server.crt

Now we can view our certificate and make sure the certificate is a “TLS web server authentication certificate”

# openssl x509 -in Certs/server.crt -text -noout | grep TLS
                TLS Web Server Authentication

Client Certificate

We can move on to the fun part and create the client certificate for the second part of the MTLS authentication.

Same as before we need to redefine the SHORT_NAME environment variable and generate a private key for the client.

# export SHORT_NAME="client"
# export DOMAIN="example.com"

Let’s generate the key

# openssl genrsa -out Keys/${SHORT_NAME}.key 4096

Now that we have the key in place we need generate the client certificate which will be almost identical to the Server certificate with a few changes.
instead of “server” we will put “client” in the extensions part :

# openssl req -new -key Keys/${SHORT_NAME}.key -out CSR/${SHORT_NAME}.csr \
 -subj "/CN=${SHORT_NAME}.${DOMAIN}/O=Users"  \
 -addext "subjectAltName = DNS:${SHORT_NAME},DNS:${SHORT_NAME}.${DOMAIN}" \
 -addext "keyUsage=digitalSignature" -addext "basicConstraints=CA:FALSE" \
 -addext "nsCertType=client" -addext "extendedKeyUsage=clientAuth"

NOTE!

We did a few minor changes , we defined the CN (comma name ) as the short name with the Domain. and the Organization as “Users”.
The reason we did it is because of the way the Apache server is going to be configured . we will focus on that when we will reach that part.

Now , the same as before we will use our CA to sign the new client certificate :

# openssl x509 -req -in CSR/${SHORT_NAME}.csr -CA CA/ca.crt -CAkey CA/ca.key \
 -CAcreateserial -out Certs/${SHORT_NAME}.crt -days 730 \
 -copy_extensions copy

Now let’s see our new client certificate and we are going to see something a bit different then before :

# openssl x509 -in Certs/client.crt -text -noout | grep TLS
                TLS Web Client Authentication

As we can see the client certificate is a defined in the extension part as a “TLS Web Client Authentication” which is what we wanted.

This complete the MTLS configuration part. in our next section we will build an httpd container that will check the client CN and only allow access if the CN is a match and the CA is a trusted CA.

OpenShift

In order to test our new certificates we will setup an httpd Container , instruct in to use TLS and create an MTLS configuration to check the CN of the client certificate :

# mkdir httpd-mtls
# cd httpd-mtls

First create a file called ssl.conf with the following content :

# cat > ssl.conf << EOF
Listen 443 https


SSLPassPhraseDialog exec:/usr/libexec/httpd-ssl-pass-dialog

SSLSessionCache         shmcb:/run/httpd/sslcache(512000)
SSLSessionCacheTimeout  300

SSLRandomSeed startup file:/dev/urandom  256
SSLRandomSeed connect builtin

SSLCryptoDevice builtin
EOF

and a simple index.html file which will help us test our MTLS configuration :

# cat > index.html << EOF
<html>
<head>
<title> this is a test </title>
<body>
<p> this is the ${USER} page </p>
</body>
</html>
EOF

Create a Containerfile which the following content :

# cat > Containerfile << EOF
FROM centos:stream9
MAINTAINER ${USER} <apache SSL>
RUN dnf install -y httpd mod_ssl 
COPY run-httpd.sh /usr/sbin/run-httpd.sh
COPY ssl.conf /etc/httpd/conf.d/ssl.conf 

RUN echo "PidFile /tmp/http.pid" >> /etc/httpd/conf/httpd.conf
RUN sed -i "s/Listen\ 80/Listen\ 8080/g"  /etc/httpd/conf/httpd.conf
RUN sed -i "s/Listen\ 443/Listen\ 8443/g" /etc/httpd/conf.d/ssl.conf 
RUN sed -i "s/\"logs\/error_log\"/\/dev\/stderr/g" /etc/httpd/conf/httpd.conf
RUN sed -i "s/CustomLog \"logs\/access_log\"/CustomLog \/dev\/stdout/g" /etc/httpd/conf/httpd.conf

RUN echo 'IncludeOptional /opt/app-root/*.conf' >> /etc/httpd/conf/httpd.conf
RUN mkdir /opt/app-root/ && \
    chown apache:apache /opt/app-root/ && \
    chmod 777 /opt/app-root/ && \
    chmod a+x /usr/sbin/run-httpd.sh

COPY index.html /opt/app-root/

USER apache

EXPOSE 8080 8443
ENTRYPOINT ["/usr/sbin/run-httpd.sh"]
EOF

Now we need to create a configuration file which enables the modules we just installed but we need to it when the Service starts. The best way to do that is to create a startup scripts which generate the configuration file.

Let’s generate the the “run-httpd.sh” script :

#  echo '#!/bin/bash

if [ -z ${SSL_CERT} ]; then
        echo "Environment variable SSL_CERT undefined"
        exit 1
elif [[ -z ${SSL_KEY} ]]; then
        echo "Environment variable SSL_KEY undefined"
        exit 1
elif [[ -z ${CA_CERT} ]]; then
        echo "Environment variable CA_CERT undefined"
        exit 1
elif [[ -z ${ALLOWED_USER} ]]; then
        echo "Environment variable ALLOWED_USER undefined"
        exit 1
fi

echo "
<VirtualHost *:8443>
        DocumentRoot /opt/app-root
        SSLEngine on
        SSLCertificateFile ${SSL_CERT}
        SSLCertificateKeyFile ${SSL_KEY}
        SSLCACertificateFile ${CA_CERT}
        <Directory "/opt/app-root/">
                AllowOverride All
                Options +Indexes
                DirectoryIndex index.html
        </Directory>        
        <Location />
        SSLVerifyClient require
        SSLVerifyDepth 1
        SSLOptions +StdEnvVars
        <RequireAny>
           Require expr %{SSL_CLIENT_S_DN_CN} == \"${ALLOWED_USER}\"
        </RequireAny>
        </Location>
</VirtualHost>
" > /tmp/reverse.conf
mv /tmp/reverse.conf /opt/app-root/reverse.conf
/usr/sbin/httpd $OPTIONS -DFOREGROUND' > run-httpd.sh

We building the image let’s add our registry to the TAG we create for the image so we can push it as a follow up command :

# export IMAGE="registry.example.com/httpd-mtls:latest"
# buildah bud -f Containerfile -t ${IMAGE} && buildah push ${IMAGE}
# cd ..

Now that our image was created and pushed to our registry we can move on and the necessary configMap and secrets and finally we will create the deployment YAML file.

First Let’s create our CA as a configMap :

# oc create cm ca-cert --from-file=ca.crt=CA/ca.crt

For the TLS certificate we will create a secret with type TLS:

# oc create secret tls http-mtls-tls --cert=Certs/server.crt \
--key=Keys/server.key

For our next step we will generate a deployment template , add all the necessary parts and run the deployment :

# oc create deployment httpd-mtls --image=${IMAGE} \
 --port=8443 -o yaml --dry-run=client > httpd-mtls/deployment.yaml

in the YAML file remove the “timestamp” lines add the following

spec:
  ...
  template:
  ...
    spec:
      containers:
      - name: httpd-mtls
        env:
        - name: SSL_CERT
          value: /opt/app-root/ssl/tls.crt
        - name: SSL_KEY
          value: /opt/app-root/ssl/tls.key
        - name: CA_CERT
          value: /opt/app-root/CA/ca.crt
        - name: ALLOWED_USER
          value: "client.example.com"
        ...
        volumeMounts:
          - mountPath: "/opt/app-root/ssl"
            name: mtls-keys
            readOnly: true
          - name: ca-cert
            mountPath: /opt/app-root/CA/ca.crt
            subPath: ca.crt
      volumes:
        - name: mtls-keys
          secret:
            secretName: http-mtls-tls
        - name: ca-cert
          configMap:
            name: ca-cert

Finally let’s apply the deployment :

# oc apply -f httpd-mtls/deployment.yaml

Service and Route

create the service

# oc create service clusterip httpd-mtls --tcp=8443

and a route that should be TLS passthrough because we want the httpd to handle the ssl request:

# oc create route passthrough httpd-mtls \
 --service=httpd-mtls --insecure-policy=Redirect \
 --hostname server.example.com

Make sure that the hostname matches the CN or one of the alternate DNS names we have provided in the certificate.

Testing

To run the test we will use curl to run a test on our MTLS configuration

First grab the route to our MTLS_ROUTE variable:

# export ROUTE_MTLS=$(oc get route httpd-mtls -o jsonpath='{.spec.host}')

In order to make the MTLS is working we will first run the curl without the client certificate

# curl --cacert CA/ca.crt https://${ROUTE_MTLS}
curl: (56) OpenSSL SSL_read: error:1409445C:SSL routines:ssl3_read_bytes:tlsv13 alert certificate required, errno 0

This error message is actually Good for us as it indicates that the client needs a certificate to identify itself

Now Let’s run it with the client certificate

# curl --cacert CA/ca.crt --cert Certs/client.crt \
 --key Keys/client.key https://${ROUTE_MTLS}

If you see your content of our index.html file then you are good to go !!!

If you have any question feel free to responed/ leave a comment.
You can find me on linkedin at : https://www.linkedin.com/in/orenoichman
Or twitter at : https://twitter.com/ooichman

Mtls Authentication
Tls
Httpd
Openshift
Security

--

--

Oren Oichman

Written by Oren Oichman

229 Followers

Open Source contributer for the past 15 years

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams