Network Capture on OpenShift 4 with tcpdump and ksniff

Why this story

Sometimes when debug an application we need to go down to the network (TCP stuck) level and see how things work (or don’t work) in order to debug communication issues.

Scenario

In this scenario

Why Both

Well, In OpenShift there are 2 networks , one is the underlay which is the host network interface consistent of the host IP address.
The other network is the overlay network which is the network the pods communicate with each other.

Where to start ?

For the debugging to work on the node level and on the pod level we need 2 things.

1. debug image

2. ksniff module

The first is an image which contains the tcpdump binary so we can run it on the network interface node and the second is to install ksniff on our oc client machine to get it to work

Building the image

I like to be prepared and have all the tools I need with me when I am debugging , for that I will use a centos:8 base image and install a some CLI tools (even if some of them do not make sense).

first we need to build small daemon to run in the background when the pod is running.

# mkdir ~/admin-tools
# cd ~/admin-tools

We will build that small daemon first with a simple bush script :

# cat > run.sh << EOF
#!/bin/bash
tail -f /dev/null
EOF

Make sure it is an executable :

# chmod a+x run.sh

And now we need to build the image With A Dockerfile should look like this :

# cat > Dockerfile << EOF
FROM docker.io/centos:8
MAINTAINER Oren Oichman "Back to ROOT!!!!"
USER root
RUN yum install epel-release -y && yum install -y jq ftp openssl p7zip curl wget tftp buildah telnet podman httpd-tools tcpdump nmap tmux bind-utils sg3_utils nmap-ncat rlwrap openldap-clients && yum clean all
WORKDIR /opt/app-root/
COPY run.sh .
ENTRYPOINT ["/opt/app-root/run.sh"]
EOF

(this procedure will work with ubi8 and local repository as well)

Now let’s build the image :

# buildah bud -f Dockerfile -t admin-tools

Now we need to push the image to a registry which is available

# HOST=$(oc get route default-route -n openshift-image-registry --template='{{ .spec.host }}')
# podman tag localhost/admin-tools ${HOST}/default/admin-tools

First let’s make sure we have all the CA certificates are available on the current node.

# cd /etc/pki/ca-trust/source/anchors/

And now export the CA for the ingress name space :

# oc -n openshift-authentication  rsh `oc get pods -n openshift-authentication -o name | head -1 `  cat /run/secrets/kubernetes.io/serviceaccount/ca.crt > ingress-ca.crt

and update the CA database :

# update-ca-trust extact

Let’s login to the registry:

# podman login -u $(oc whoami) -p (oc whoami -t) $HOST

And push the image to the registry

# podman push ${HOST}/default/admin-tools

Once the process is complete we can use this image on the node we want to debug.

Running the image

Before we run the command we need to see on which IP address the node is listening on:

# oc get nodes -o wide

The Server IP should be on the Internal IP column so we can catch it with a simple grep and awk command with the node we want to debug:

# oc get nodes -o wide | grep <node> | awk '{print $6}'

Save the output a side (we will need it later).

Now we can run the debug with our new image :

# oc debug node/<node> --image=${HOST}/default/admin-tools

we can look interfaces with a simple ip command :

# ip addr show | grep -B2 <IP Address>

Now that we have the interface name we can start running the network debug on that interface.

# tcpdump -i <interface> 'tcpdump filgters'

oc with ksniff

Up until now we have worked on the node which is hosting the POD (A.K.A the underlay) but now we want see the traffic going over the POD 2 POD communication , for that we can run the debug command from the previous run or we can use ksniff.

Ksniff for those among us whom don’t know is a plug-in for kubectl (and oc) which enable us to run a interface capture from the pod itself to our running terminal.

Installing ksniff

The Install process is fairly simple , all we need to do it to download the zip file (the corrent version is 1.6)

# wget https://github.com/eldadru/ksniff/releases/download/v1.6.0/ksniff.zip

and then unzip it and install it with the make command :

# unzip ksniff.zip
# make install

and you are good to go to use it.

Exmaple :

if we want to run ksniff and capture all the traffic on port 80 we will run the command as follow :

# oc sniff <pod name> -f "port 80"

Redirect to Wireshark

another great usage is to direct all the output to our wireshark (needs to be installed on the same machine where the “oc” command is running).
If you are running a Linux desktop the command is pretty simple :

# oc sniff <pod name> -f "port 80" -o - | wireshark -r -

Non-Privileged and Scratch Pods

To reduce attack surface and have small and lean containers, many production-ready containers runs as non-privileged user or even as a scratch container.

To support those containers as well, ksniff now ships with the “-p” (privileged) mode. When executed with the -p flag, ksniff will create a new pod on the remote kubernetes cluster that will have access to the node docker daemon.

ksniff will than use that pod to execute a container attached to the target container network namespace and perform the actual network capture.

More information can be found on the github tool website here

That is it
Have fun

If you have any question feel free to responed/ leave a comment.
You can find on linkedin at : https://www.linkedin.com/in/orenoichman
Or twitter at : https://twitter.com/ooichman