Search:

Kubectl New Feature: debug

Kubectl New Feature: debug

In this post, I will talk about the background of debug, a new alpha feature in the kubectl.

In Kubernetes, each pod runs as a docker container and these containers are isolated through the Linux namespace isolation feature. 

Each container has its own process filesystem, as shown below.

$ docker exec nginx ps aux

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND

root         1  0.1  0.0  10632  5344 ?        Ss   13:53   0:00 nginx: master process nginx -g daemon off;

nginx        6  0.0  0.0  11088  2664 ?        S    13:53   0:00 nginx: worker process

root        18  0.1  0.0   2388   696 pts/0    Ss   13:54   0:00 sh

This is the output of a healthy process and everything looks good and we can attach pods via exec command .

As you can see in the output, nginx process works as PID 1. If this process crashes, it is impossible to attach to this container via kubectl exec

So how can we debug our applications when their main process crashes?

Solution of Kubernetes

Kubernetes 1.17 supports ephemeral containers, and this capability makes it possible to share PID namespaces.

Ephemeral containers allow injections to IPC namespaces and you can run debug commands on your crashed pods.

 

kubernetes-1-18-debug

 

To use this capability, you need an ephemeral container definition, which is shown below:

{

   "apiVersion": "v1",

   "kind": "EphemeralContainers",

   "metadata": {

           "name": "example-pod"

   },

   "ephemeralContainers": [{

       "command": [

           "sh"

       ],

       "image": "busybox",

       "imagePullPolicy": "IfNotPresent",

       "name": "debugger",

       "stdin": true,

       "tty": true,

       "terminationMessagePolicy": "File"

   }]

}

Then, you need to attach to the IPC namespace of your container with this command: 

kubectl replace --raw /api/v1/namespaces/default/pods/java-app/ephemeralcontainers  -f ephemeral.json

What does `kubectl debug` do ?

The new debug feature automates ephemeral container attaching and namespace sharing with a single command and without any manifest files. Even if your cluster version is 1.15, 1.16 or 1.17, you should enable the ephemeral container feature. But with version 1.18, this feature comes by default.

I have a created sample cluster configuration file for kubeadm that enables feature-gates on Kubernetes clusters.

apiVersion: kubeadm.k8s.io/v1beta2

kind: InitConfiguration

nodeRegistration:

 featureGates:

   EphemeralContainers: true

---

apiVersion: kubeadm.k8s.io/v1beta2

kind: ClusterConfiguration

kubernetesVersion: v1.18.0

apiServer:

 featureGates:

   EphemeralContainers: true

---

apiVersion: kubeadm.k8s.io/v1beta2

kind: ClusterConfiguration

kubernetesVersion: v1.18.0

controllerManager:

 featureGates:

   EphemeralContainers: true

Summary

As you can see at above, you need to enable EphemeralContainer and write another JSON manifest file to use debug containers in previous versions. But on version 1.18 kubectl, you can officially complete this flow with only one CLI argument.

kubectl alpha debug -it java-app-failing --image=busybox --target=failing-java-app

Kubernetes environment is improving itself day by day. These new debug capabilities make  debugging easy on your pods. We are excited for the upcoming new features in Kubernetes.

Have a nice day, stay at home and support the Kubernetes community!

Emir Ozbir

Software Developer at kloia