Aug 26, 2021

6 min read

OpenShift 4 with Metal-LB Layer2 for Service Public IP

Why This Article ?

Working with OpenShift/Kubernetes is an enterprise environment can present interesting challenges, one of them is make sure that none HTTP services are available outside of the OpenShift/Kubernetes Cluster SDN (Software defined Network).
for HTTP/HTTPS services we can use route/ingress to expose them outside of our cluster but for TCP services we need different approach …

The different approach is actually more then one.
The first option is to setup an external Load Balancer (HAproxy) and on the Cluster side setup a Service Load balancer with a NodePort. The second approach is to deploy an Operator named Metal-LB that will create a static NAT between an external IP and the internal IP of the service (more details later on)

In this tutorial we will configure the Metal-LB option.

NOTE!!

Currently Metal-LB is not with in the supported scope of OpenShift but in the near future of version 4.9–4.10 is will be GA and fully supported.

Architecture View

Deployment

in order to deploy and use the Metal-LB operator run the following steps.

First create the namespace with the following command :

# oc apply -f https://raw.githubusercontent.com/metallb/metallb/v0.11.0/manifests/namespace.yaml

Next , use wget to obtain the deployment YAML :

# wget https://raw.githubusercontent.com/metallb/metallb/v0.11.0/manifests/metallb.yaml

Once we obtain the file we need to run a few changes in it.

in the YAML file there are 2 issues we need to manage , first is the rages of the runAsUser and runAsGroup and modify the MAX and MIN values.

We want the lines ( numbers 14–30 ) to go from this :

   fsGroup:
     ranges:
     - max: 65535
       min: 1
     rule: MustRunAs
   hostIPC: false
   hostNetwork: false
   hostPID: false
   privileged: false
   readOnlyRootFilesystem: true
   requiredDropCapabilities:
   - ALL
   runAsUser:
     ranges:
     - max: 65535
       min: 1
     rule: MustRunAs
   seLinux:
     rule: RunAsAny
   supplementalGroups:
     ranges:
     - max: 65535
       min: 1
     rule: MustRunAs

To this :

fsGroup:
     ranges:
     - max: 1000719999
       min: 1000710000
     rule: MustRunAs
   hostIPC: false
   hostNetwork: false
   hostPID: false
   privileged: false
   readOnlyRootFilesystem: true
   requiredDropCapabilities:
   - ALL
   runAsUser:
     ranges:
     - max: 1000719999
       min: 1000710000
     rule: MustRunAs
   seLinux:
     rule: RunAsAny
   supplementalGroups:
     ranges:
     - max: 1000719999
       min: 1000710000
     rule: MustRunAs

NOTE!

the “max” and “min” range values my change from cluster installation to another , if by any case the “controller” Pod does not start go over the events and search for range reference :

# oc get events | grep range

Now go down to the deployment of the controller (line 431) and update the securityContext to set the UID for the deployment & we need to set it to run as non root as well :

After the change the Section should be as follow :

         securityContext:
           allowPrivilegeEscalation: false
           runAsNonRoot: true
           runAsUser: 1000710000
           capabilities:
             drop:
             - all

One you updated everything we need to give privileges Security Context to the speaker service account (the service account of the daemonset) :

# oc adm policy add-scc-to-user privileged -n metallb-system -z speaker

Finally, we can now apply it :

# oc apply -f metallb.yaml

Once you got it applied run ‘watch -n 1 “oc get all”’ to make sure all the pod are running :

# watch -n 1 "oc get all"
NAME                              READY   STATUS    RESTARTS   AGE
pod/controller-5cc59b7bb8-fvqjs   1/1     Running   0          16m
pod/speaker-6d9c8                 1/1     Running   0          110s
pod/speaker-c2nzm                 1/1     Running   0          2m9s
pod/speaker-cvv52                 1/1     Running   0          91sNAME                     DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
daemonset.apps/speaker   3         3         3       3            3           kubernetes.io/os=linux   27mNAME                         READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/controller   1/1     1            1           27mNAME                                    DESIRED   CURRENT   READY   AGE
replicaset.apps/controller-5cc59b7bb8   1         1         1       17m
replicaset.apps/controller-6b78bff7d9   0         0         0       27m

the Number of the speaker pod is depending on the number of your workers (you can change it to infra nodes by modifying the node Selector on the daemonset and given the right annotation to the metallb-system namspace).
If everything is running as required, we are good to go

NOTE!

n case you are hitting an error : secret “memberlist” not found
you need to create the “memberlist” secret with the following command :

# oc create secret generic -n metallb-system memberlist --from-literal=secretkey="$(openssl rand -base64 128)"

Configuration

in this tutorial we will setup on a Layer 2 configuration. the operator does allow us to use BGP but I will not cover this option at this tutorial.

Layer 2 configuration

Layer 2 mode is the simplest to configure: in many cases, you don’t need any protocol-specific configuration, only IP addresses.

Layer 2 mode does not require the IPs to be bound to the network interfaces of your worker nodes. It works by responding to ARP requests on your local network directly, to give the machine’s MAC address to clients.

For example, the following configuration gives MetalLB control over IPs from 192.168.1.140 to 192.168.1.150, and configures Layer 2 mode:

# cat > metal-lb-config.yaml << EOF
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: dev-public-ips
      protocol: layer2
      addresses:
      - 192.168.1.140-192.168.1.150
EOF

Next let’s go ahead and apply it :

# oc apply -f metal-lb-config.yaml

Usage

The usage of the operator is very simple , we are going to create a new namespace. On the new namespace we will deploy a new MariaDB database and then we will create a service with the right annotation to tell Metal-LB to assign our service a public IP.

first let’s create the namespace :

# oc new-project mariadb

Now that we have a new project we can deploy the MariaDB database.

First we will deploy the Persistent storage claim :

# cat > mariadb-pvc.yaml << EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mariadb-pv-claim
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 2Gi
EOF

Deploy it :

# oc apply -f mariadb-pvc.yaml

for the Mariadb we will create the following deployment file :

# cat > mariadb-deployment.yaml << EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mariadb
spec:
  selector:
    matchLabels:
      app: mariadb
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: mariadb
    spec:
      containers:
      - image: mariadb
        name: mariadb
        env:
        - name: MYSQL_ROOT_PASSWORD
          value: password
        ports:
        - containerPort: 3306
          name: mariadb
        volumeMounts:
        - name: mariadb-persistent-storage
          mountPath: /var/lib/mysql
      volumes:
      - name: mariadb-persistent-storage
        persistentVolumeClaim:
          claimName: mariadb-pv-claim
EOF

and apply it :

# oc apply -f mariadb-deployment.yaml

Once everything is running it’s time to create the service with the type of “loadbalancer” and get it a public IP address :

# cat > mariadb-service.yaml << EOF
apiVersion: v1
kind: Service
metadata:
  name: mariadb
  annotations:
    metallb.universe.tf/address-pool: dev-public-ips
spec:
  ports:
  - port: 3306
  selector:
    app: mariadb
  type: LoadBalancer
EOF

Note!

The annotation we set the address pool name is identical to name in the configmap we created under the metal-lb deployment for the network name.

Now that we look at our service we can see the public IP address :

# oc get svc
NAME      TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)          AGE
mariadb   LoadBalancer   10.43.24.171   192.168.1.140   3306:32517/TCP   2m14s

As we can see the external IP is part of the IP address pool we configured.

Run mysql with the External IP :

# mysql -u root -p -h 192.168.1.140
Enter password: 
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 5
Server version: 10.6.4-MariaDB-1:10.6.4+maria~focal mariadb.org binary distributionCopyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.MariaDB [(none)]>

If you have been prompted for a password (“password”) and then entered an mariaDB terminal then everything is working as expected

TCP/UDP Testing

the service external IP does not responses to ICMP (ping) request so in case we want to make sure the socket is open we can use netcat to test it :

For TCP connection:

# nc -vz <Public IP> <Port number>
(Example : nc -vz 192.168.1.140 3306)

For UDP connections:

# nc -vz -u <Public IP> <Port number>

If you have any question feel free to responed/ leave a comment.
You can find on linkedin at : https://www.linkedin.com/in/orenoichman
Or twitter at : https://twitter.com/ooichman