HAproxy with X-forward-for pass to OpenShift 4

Into the thick of it

When we deploy OpenShift 4 in a full Open Source environment it is recommended to use HAproxy as the cluster external Load Balancer.
HAproxy has to ways of load balancing the traffic , The first and the most used is by TCP so the packet does not go through any inspection from the HAproxy side and just change to source IP which is sent onto it’s destination (the router or the API).

In some cases the end user wants to control which source IP addresses are allowed and which denied from using the application. For that they will need the original IP address of the client.

Since HAproxy in TCP mode will not pass the original source IP address ( while not using proxy protocol) we need to use HAproxy with HTTP mode and set it to forward the source IP address with X-forward-for.

Overview

In this tutorial we will set the HAproxy to send X-forward-for to OpenShift 4 and we will show how set up the OpenShift’s route to Allow/Deny the traffic from the source IP.

HAproxy Configuration

First we need to set up HAproxy with the right set of options on the back end and front end services which are relevant to the ingress operator.

In full stack configuration the HAproxy will handle traffic on ports : 80 , 443 , 6443 and 22623. In our case we will change only ports 80 and 443.

HTTP (Port 80)

Port 80 is simple then 443 (due to the SSL part which we will touch base on later). All we need to do is change the mode from TCP to HTTP and make sure the X-forward-for options are set.

The Following is an example of out relevant section in the haproxy.conf configuration file

We will switch From this :

frontend  openshift-app-http
    bind *:80
    default_backend openshift-app-http
    mode tcp
    option tcplog
backend openshift-app-http
    balance roundrobin
    mode tcp
    server      worker-01 192.168.1.9:80 check
    server      worker-02 192.168.1.10:80 check
    server      worker-03 192.168.1.11:80 check

To this :

frontend  openshift-app-http
    bind *:80  
    mode http  
    option httplog
    default_backend openshift-app-http
    option http-server-close
    option forwardfor
    http-request add-header X-Forwarded-Proto http
    http-request add-header X-Forwarded-Port 80
backend openshift-app-http
    balance roundrobin
    mode http
    server      worker-01 192.168.1.9:80 check
    server      worker-02 192.168.1.10:80 check
    server      worker-03 192.168.1.11:80 check

Now that we’ve finished with the HTTP part we can move on to the HTTPS part which will need use some SSL/TLS skills.

HTTPS (port 443)

For out HTTPS configuration we need to obtain the wildcard SSL/TLS certificate and add it to our HAproxy configuration to make sure we are addressing OpenShift with SSL enabled.

On our configuration file, We will change on the From this :

frontend  openshift-app-https
    bind *:443
    default_backend openshift-app-https
    mode tcp
    option tcplog
backend openshift-app-https
    balance roundrobin
    mode tcp
    server      worker-01 192.168.1.9:443 check
    server      worker-02 192.168.1.10:443 check
    server      worker-03 192.168.1.11:443 check

To the following :

frontend  openshift-app-https
    bind *:443 ssl crt /etc/haproxy/certs.d/wildcard.example.com.crt no-sslv3 
    mode http
    option httplog    
    option http-server-close
    option forwardfor
    http-request add-header X-Forwarded-Proto https
    http-request add-header X-Forwarded-Port 443
    # set HTTP Strict Transport Security (HTST) header (Optional)
    http-response add-header Strict-Transport-Security max-age=15768000
    # the Backend
    default_backend openshift-app-https
backend openshift-app-https
    balance roundrobin
    server      worker-01 192.168.1.9:443 ssl verify check
    server      worker-02 192.168.1.10:443 ssl verify check
    server      worker-03 192.168.1.11:443 ssl verify check

In the example above we have a bundle of the CA + certificate including the private key of the wildcard certificate.

In order to concatenate the needed files we can run the following :

#cat wildcard.crt ca.crt wildcard.key > /etc/haproxy/certs.d/wildcard.example.com.crt

In the above example the mentioned files are :

• wildcard.crt — the wildcard certificate file
• ca.crt — the certificate authority file
• wildcard.key — the wildcard certificate key file.

Now all we need to do is restart the haproxy service and make sure all the configuration are in order

Testing

In order to test our configuration we can add an annotation to the route and specified the source IP address

metadata:
  annotations:
    haproxy.router.openshift.io/ip_whitelist: 192.168.1.0/24 192.168.2.10

That’s it

If you have any question feel free to respond / leave a comment.
You can find me on linkedin at : https://www.linkedin.com/in/orenoichman
Or twitter at : https://twitter.com/ooichman