OpenShift 4 with Metal-LB Layer2 for Service Public IP

Why This Article ?

Working with OpenShift/Kubernetes is an enterprise environment can present interesting challenges, one of them is make sure that none HTTP services are available outside of the OpenShift/Kubernetes Cluster SDN (Software defined Network).
for HTTP/HTTPS services we can use route/ingress to expose them outside of our cluster but for TCP services we need different approach …

The different approach is actually more then one.
The first option is to setup an external Load Balancer (HAproxy) and on the Cluster side setup a Service Load balancer with a NodePort. The second approach is to deploy an Operator named Metal-LB that will create a static NAT between an external IP and the internal IP of the service (more details later on)

In this tutorial we will configure the Metal-LB option.

Currently Metal-LB is not with in the supported scope of OpenShift but in the near future of version 4.9–4.10 is will be GA and fully supported.

Architecture View

Deployment

in order to deploy and use the Metal-LB operator run the following steps.

First create the namespace with the following command :

# oc apply -f https://raw.githubusercontent.com/metallb/metallb/v0.11.0/manifests/namespace.yaml

Next , use wget to obtain the deployment YAML :

# wget https://raw.githubusercontent.com/metallb/metallb/v0.11.0/manifests/metallb.yaml

Once we obtain the file we need to run a few changes in it.

in the YAML file there are 2 issues we need to manage , first is the rages of the runAsUser and runAsGroup and modify the MAX and MIN values.

We want the lines ( numbers 14–30 ) to go from this :

   fsGroup:
ranges:
- max: 65535
min: 1
rule: MustRunAs
hostIPC: false
hostNetwork: false
hostPID: false
privileged: false
readOnlyRootFilesystem: true
requiredDropCapabilities:
- ALL
runAsUser:
ranges:
- max: 65535
min: 1
rule: MustRunAs
seLinux:
rule: RunAsAny
supplementalGroups:
ranges:
- max: 65535
min: 1
rule: MustRunAs

To this :

fsGroup:
ranges:
- max: 1000719999
min: 1000710000
rule: MustRunAs
hostIPC: false
hostNetwork: false
hostPID: false
privileged: false
readOnlyRootFilesystem: true
requiredDropCapabilities:
- ALL
runAsUser:
ranges:
- max: 1000719999
min: 1000710000
rule: MustRunAs
seLinux:
rule: RunAsAny
supplementalGroups:
ranges:
- max: 1000719999
min: 1000710000
rule: MustRunAs

NOTE!

the “max” and “min” range values my change from cluster installation to another , if by any case the “controller” Pod does not start go over the events and search for range reference :

# oc get events | grep range

Now go down to the deployment of the controller (line 431) and update the securityContext to set the UID for the deployment & we need to set it to run as non root as well :

After the change the Section should be as follow :

         securityContext:
allowPrivilegeEscalation: false
runAsNonRoot: true
runAsUser: 1000710000
capabilities:
drop:
- all

One you updated everything we need to give privileges Security Context to the speaker service account (the service account of the daemonset) :

# oc adm policy add-scc-to-user privileged -n metallb-system -z speaker

Finally, we can now apply it :

# oc apply -f metallb.yaml

Once you got it applied run ‘watch -n 1 “oc get all”’ to make sure all the pod are running :

# watch -n 1 "oc get all"
NAME READY STATUS RESTARTS AGE
pod/controller-5cc59b7bb8-fvqjs 1/1 Running 0 16m
pod/speaker-6d9c8 1/1 Running 0 110s
pod/speaker-c2nzm 1/1 Running 0 2m9s
pod/speaker-cvv52 1/1 Running 0 91s
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/speaker 3 3 3 3 3 kubernetes.io/os=linux 27m
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/controller 1/1 1 1 27m
NAME DESIRED CURRENT READY AGE
replicaset.apps/controller-5cc59b7bb8 1 1 1 17m
replicaset.apps/controller-6b78bff7d9 0 0 0 27m

the Number of the speaker pod is depending on the number of your workers (you can change it to infra nodes by modifying the node Selector on the daemonset and given the right annotation to the metallb-system namspace).
If everything is running as required, we are good to go

NOTE!

n case you are hitting an error : secret “memberlist” not found
you need to create the “memberlist” secret with the following command :

# oc create secret generic -n metallb-system memberlist --from-literal=secretkey="$(openssl rand -base64 128)"

Configuration

in this tutorial we will setup on a Layer 2 configuration. the operator does allow us to use BGP but I will not cover this option at this tutorial.

Layer 2 mode is the simplest to configure: in many cases, you don’t need any protocol-specific configuration, only IP addresses.

Layer 2 mode does not require the IPs to be bound to the network interfaces of your worker nodes. It works by responding to ARP requests on your local network directly, to give the machine’s MAC address to clients.

For example, the following configuration gives MetalLB control over IPs from 192.168.1.140 to 192.168.1.150, and configures Layer 2 mode:

# cat > metal-lb-config.yaml << EOF
apiVersion: v1
kind: ConfigMap
metadata:
namespace: metallb-system
name: config
data:
config: |
address-pools:
- name: dev-public-ips
protocol: layer2
addresses:
- 192.168.1.140-192.168.1.150
EOF

Next let’s go ahead and apply it :

# oc apply -f metal-lb-config.yaml

Usage

The usage of the operator is very simple , we are going to create a new namespace. On the new namespace we will deploy a new MariaDB database and then we will create a service with the right annotation to tell Metal-LB to assign our service a public IP.

first let’s create the namespace :

# oc new-project mariadb

Now that we have a new project we can deploy the MariaDB database.

First we will deploy the Persistent storage claim :

# cat > mariadb-pvc.yaml << EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: mariadb-pv-claim
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 2Gi
EOF

Deploy it :

# oc apply -f mariadb-pvc.yaml

for the Mariadb we will create the following deployment file :

# cat > mariadb-deployment.yaml << EOF
apiVersion: apps/v1
kind: Deployment
metadata:
name: mariadb
spec:
selector:
matchLabels:
app: mariadb
strategy:
type: Recreate
template:
metadata:
labels:
app: mariadb
spec:
containers:
- image: mariadb
name: mariadb
env:
- name: MYSQL_ROOT_PASSWORD
value: password
ports:
- containerPort: 3306
name: mariadb
volumeMounts:
- name: mariadb-persistent-storage
mountPath: /var/lib/mysql
volumes:
- name: mariadb-persistent-storage
persistentVolumeClaim:
claimName: mariadb-pv-claim
EOF

and apply it :

# oc apply -f mariadb-deployment.yaml

Once everything is running it’s time to create the service with the type of “loadbalancer” and get it a public IP address :

# cat > mariadb-service.yaml << EOF
apiVersion: v1
kind: Service
metadata:
name: mariadb
annotations:
metallb.universe.tf/address-pool: dev-public-ips
spec:
ports:
- port: 3306
selector:
app: mariadb
type: LoadBalancer
EOF

Note!

The annotation we set the address pool name is identical to name in the configmap we created under the metal-lb deployment for the network name.

Now that we look at our service we can see the public IP address :

# oc get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
mariadb LoadBalancer 10.43.24.171 192.168.1.140 3306:32517/TCP 2m14s

As we can see the external IP is part of the IP address pool we configured.

Run mysql with the External IP :

# mysql -u root -p -h 192.168.1.140
Enter password:
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 5
Server version: 10.6.4-MariaDB-1:10.6.4+maria~focal mariadb.org binary distribution
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.MariaDB [(none)]>

If you have been prompted for a password (“password”) and then entered an mariaDB terminal then everything is working as expected

the service external IP does not responses to ICMP (ping) request so in case we want to make sure the socket is open we can use netcat to test it :

For TCP connection:

# nc -vz <Public IP> <Port number>
(Example : nc -vz 192.168.1.140 3306)

For UDP connections:

# nc -vz -u <Public IP> <Port number>

If you have any question feel free to responed/ leave a comment.
You can find on linkedin at : https://www.linkedin.com/in/orenoichman
Or twitter at : https://twitter.com/ooichman

--

--

--

Open Source contributer for the past 15 years

Love podcasts or audiobooks? Learn on the go with our new app.

5 More Free Public APIs You Should Try

CS373 Spring 2022: Cole Weinman — Week 4

How One Week Sprints Pave the Way to Kanban

The image shows the article title: How one week sprint pave the way to Kanban

Clayton Waldock: How A Coding Bootcamp Changed My Life.

Repository Pattern In ASP.NET Core

Community update

SQL problems

Build a simple REST APIs with Java 8, Tomcat, JSP, MySQL & IntelliJ on Mac

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Oren Oichman

Oren Oichman

Open Source contributer for the past 15 years

More from Medium

Kubernetes Network Policies with Calico

Assign permissions to an user in Kubernetes. An overview of RBAC-based AuthZ in k8s

ArgoCD Image Updater with Digital Ocean Container Registry

Kubernetes Networking With Calico