OpenShift 4 with Metal-LB Layer2 for Service Public IP

Why This Article ?

Working with OpenShift/Kubernetes is an enterprise environment can present interesting challenges, one of them is make sure that none HTTP services are available outside of the OpenShift/Kubernetes Cluster SDN (Software defined Network).
for HTTP/HTTPS services we can use route/ingress to expose them outside of our cluster but for TCP services we need different approach …

NOTE!!

Currently Metal-LB is not with in the supported scope of OpenShift but in the near future of version 4.9–4.10 is will be GA and fully supported.

Architecture View

Deployment

in order to deploy and use the Metal-LB operator run the following steps.

# oc apply -f https://raw.githubusercontent.com/metallb/metallb/v0.11.0/manifests/namespace.yaml
# wget https://raw.githubusercontent.com/metallb/metallb/v0.11.0/manifests/metallb.yaml
   fsGroup:
ranges:
- max: 65535
min: 1
rule: MustRunAs
hostIPC: false
hostNetwork: false
hostPID: false
privileged: false
readOnlyRootFilesystem: true
requiredDropCapabilities:
- ALL
runAsUser:
ranges:
- max: 65535
min: 1
rule: MustRunAs
seLinux:
rule: RunAsAny
supplementalGroups:
ranges:
- max: 65535
min: 1
rule: MustRunAs
fsGroup:
ranges:
- max: 1000719999
min: 1000710000
rule: MustRunAs
hostIPC: false
hostNetwork: false
hostPID: false
privileged: false
readOnlyRootFilesystem: true
requiredDropCapabilities:
- ALL
runAsUser:
ranges:
- max: 1000719999
min: 1000710000
rule: MustRunAs
seLinux:
rule: RunAsAny
supplementalGroups:
ranges:
- max: 1000719999
min: 1000710000
rule: MustRunAs
# oc get events | grep range
         securityContext:
allowPrivilegeEscalation: false
runAsNonRoot: true
runAsUser: 1000710000
capabilities:
drop:
- all
# oc adm policy add-scc-to-user privileged -n metallb-system -z speaker
# oc apply -f metallb.yaml
# watch -n 1 "oc get all"
NAME READY STATUS RESTARTS AGE
pod/controller-5cc59b7bb8-fvqjs 1/1 Running 0 16m
pod/speaker-6d9c8 1/1 Running 0 110s
pod/speaker-c2nzm 1/1 Running 0 2m9s
pod/speaker-cvv52 1/1 Running 0 91s
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/speaker 3 3 3 3 3 kubernetes.io/os=linux 27m
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/controller 1/1 1 1 27m
NAME DESIRED CURRENT READY AGE
replicaset.apps/controller-5cc59b7bb8 1 1 1 17m
replicaset.apps/controller-6b78bff7d9 0 0 0 27m
# oc create secret generic -n metallb-system memberlist --from-literal=secretkey="$(openssl rand -base64 128)"

Configuration

in this tutorial we will setup on a Layer 2 configuration. the operator does allow us to use BGP but I will not cover this option at this tutorial.

Layer 2 configuration

Layer 2 mode is the simplest to configure: in many cases, you don’t need any protocol-specific configuration, only IP addresses.

# cat > metal-lb-config.yaml << EOF
apiVersion: v1
kind: ConfigMap
metadata:
namespace: metallb-system
name: config
data:
config: |
address-pools:
- name: dev-public-ips
protocol: layer2
addresses:
- 192.168.1.140-192.168.1.150
EOF
# oc apply -f metal-lb-config.yaml

Usage

The usage of the operator is very simple , we are going to create a new namespace. On the new namespace we will deploy a new MariaDB database and then we will create a service with the right annotation to tell Metal-LB to assign our service a public IP.

# oc new-project mariadb
# cat > mariadb-pvc.yaml << EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: mariadb-pv-claim
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 2Gi
EOF
# oc apply -f mariadb-pvc.yaml
# cat > mariadb-deployment.yaml << EOF
apiVersion: apps/v1
kind: Deployment
metadata:
name: mariadb
spec:
selector:
matchLabels:
app: mariadb
strategy:
type: Recreate
template:
metadata:
labels:
app: mariadb
spec:
containers:
- image: mariadb
name: mariadb
env:
- name: MYSQL_ROOT_PASSWORD
value: password
ports:
- containerPort: 3306
name: mariadb
volumeMounts:
- name: mariadb-persistent-storage
mountPath: /var/lib/mysql
volumes:
- name: mariadb-persistent-storage
persistentVolumeClaim:
claimName: mariadb-pv-claim
EOF
# oc apply -f mariadb-deployment.yaml
# cat > mariadb-service.yaml << EOF
apiVersion: v1
kind: Service
metadata:
name: mariadb
annotations:
metallb.universe.tf/address-pool: dev-public-ips
spec:
ports:
- port: 3306
selector:
app: mariadb
type: LoadBalancer
EOF
# oc get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
mariadb LoadBalancer 10.43.24.171 192.168.1.140 3306:32517/TCP 2m14s
# mysql -u root -p -h 192.168.1.140
Enter password:
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 5
Server version: 10.6.4-MariaDB-1:10.6.4+maria~focal mariadb.org binary distribution
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.MariaDB [(none)]>

TCP/UDP Testing

the service external IP does not responses to ICMP (ping) request so in case we want to make sure the socket is open we can use netcat to test it :

# nc -vz <Public IP> <Port number>
(Example : nc -vz 192.168.1.140 3306)
# nc -vz -u <Public IP> <Port number>

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store