Troubleshooting Kubernetes Master Nodes

Some distributions of Kubernetes hide the master nodes away from you so you don't need to worry about them.  Some examples of this are Azure AKS or Google Kubernetes Engine.  In those instances, you're paying for the vendor to manage the master nodes for you, so there's no need for you to monitor or troubleshoot them.

However, in some instances you will be responsible for the master nodes.  For example, if you've deployed a cluster using acs-engine, or built your own kubernetes cluster from scratch.  This post will cover how to troubleshoot the main components of the master nodes.

What are the different components on the master nodes and what do they do:

  • kubelet
    • The kubelet is one of the main kubernetes components. This is responsible for reporting node ready status and actually interacting with the container runtime to execute pods
  • etcd
    • etcd is the key-value store that stores the state of the kubernetes cluster. The kube-apiserver uses this to read/write and store it's data
  • kube-apiserver
    • This is the API web server that kubectl and other kubernetes components interact with
  • kube-controller-manager
    • This component is used by kubernetes to ensure the system is in the correct desired states. For example, if you request 3 pods of a certain type, and one pod fails, the controller-manager will ensure another pod is started up
  • kube-scheduler
    • This component is used by kubernetes to determine the appropriate node(s) to deploy pods to

There are a few reasons you may notice something is wrong with your master nodes.  Some possible scenarios that could occur if one or more components are down are:

  • Unable to access kube-apiserver via kubectl
  • Pods are not getting deployed to nodes
  • Deployments are not creating an appropriate number of pods
  • One or more nodes are not showing as Ready via kubectl

It's important to keep in mind that there are multiple different ways that a master node can be deployed.  However, the most common way you'll see today is you'll have a kubelet that's managed via the systemd startup process, and then all other components are launched as static pod manifests and actually run as containers themselves.   That configuration is what I'll talk about below:

Checking node status

First of all you'll want to check the node status - you can do that via kubectl:

# Get the status for all known nodes
kubectl get nodes
# Or get detailed information for a specific node
kubectl describe node NODE_NAME

Here's some sample output you might get - while some components could fail and still leave the node in a "Ready" state, if you see one as "NotReady", there's definitely a problem to look into.

kubectl get nodes
NAME                    STATUS    ROLES     AGE       VERSION
1348k8s000              Ready     agent     11d       v1.12.2
1348k8s001              Ready     agent     11d       v1.12.2
1348k8s002              Ready     agent     11d       v1.12.2
k8s-master-13487264-0   Ready     master    11d       v1.12.2
k8s-master-13487264-1   NotReady  master    11d       v1.12.2
k8s-master-13487264-2   Ready     master    11d       v1.12.2

Digging in to a specific node can give you some more information - you're most interested in the "Conditions" and "Events" output of this command:

kubectl describe node k8s-master-13487264-1
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  OutOfDisk        False   Wed, 19 Dec 2018 07:53:01 -0600   Fri, 07 Dec 2018 08:13:33 -0600   KubeletHasSufficientDisk     kubelet has sufficient disk space available
  MemoryPressure   False   Wed, 19 Dec 2018 07:53:01 -0600   Fri, 07 Dec 2018 08:13:33 -0600   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     False   Wed, 19 Dec 2018 07:53:01 -0600   Fri, 07 Dec 2018 08:13:33 -0600   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure      False   Wed, 19 Dec 2018 07:53:01 -0600   Fri, 07 Dec 2018 08:13:33 -0600   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready            True    Wed, 19 Dec 2018 07:53:01 -0600   Fri, 07 Dec 2018 08:13:33 -0600   KubeletReady                 kubelet is posting ready status. AppArmor enabled
Events:         none

Checking the status of the kubelet

The kubelet will almost always be deployed as a systemd process.  That means we can use the systemctl commands to see the state:

systemctl status kubelet.service
● kubelet.service - Kubelet
   Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2018-12-18 20:17:39 UTC; 17h ago

The most important thing is that you see the status of "loaded" and "active (running)" in the output.  If you don't see that, you can always try to stop/restart/start the service with the following systemctl commands:

# Start the service
systemctl start kubelet.service

# Stop the service
systemctl stop kubelet.service

# Restart the service
systemctl restart kubelet.service

To check the configuration values passed to the kubelet, you'll want to note the .service file it's using.  This was noted in the following line from the systemctl status output you ran earlier:

Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: enabled)

cat /etc/systemd/system/kubelet.service

ExecStartPre=/bin/bash /opt/azure/containers/
ExecStartPre=/bin/mkdir -p /var/lib/kubelet
ExecStartPre=/bin/mkdir -p /var/lib/cni
ExecStartPre=/bin/bash -c "if [ $(mount | grep \"/var/lib/kubelet\" | wc -l) -le 0 ] ; then /bin/mount --bind /var/lib/kubelet /var/lib/kubelet ; fi"
ExecStartPre=/bin/mount --make-shared /var/lib/kubelet
# This is a partial workaround to this upstream Kubernetes issue:
ExecStartPre=/sbin/sysctl -w net.ipv4.tcp_retries2=8
ExecStartPre=-/sbin/ebtables -t nat --list
ExecStartPre=-/sbin/iptables -t nat --list
ExecStart=/usr/local/bin/kubelet \
        --enable-server \
        --node-labels="${KUBELET_NODE_LABELS}" \
        --v=2 \
        --volume-plugin-dir=/etc/kubernetes/volumeplugins \


Down in the "ExecStart" you'll see what parameters are being used for the kubelet, and you'll notice there's also an "EnvironmentFile".  This is what's likely to have the configuration values we need, so let's take a look at that:

cat /etc/default/kubelet


KUBELET_CONFIG=--address= --allow-privileged=true --anonymous-auth=false --authorization-mode=Webhook --azure-container-registry-config=/etc/kubernetes/azure.json --cgroups-per-qos=true --client-ca-file=/etc/kubernetes/certs/ca.crt --cloud-config=/etc/kubernetes/azure.json --cloud-provider=azure --cluster-dns= --cluster-domain=cluster.local --enforce-node-allocatable=pods --event-qps=0 --feature-gates=PodPriority=true --image-gc-high-threshold=85 --image-gc-low-threshold=80 --image-pull-progress-deadline=30m --keep-terminated-pod-volumes=false --kubeconfig=/var/lib/kubelet/kubeconfig --max-pods=30 --network-plugin=cni --node-status-update-frequency=10s --non-masquerade-cidr= --pod-manifest-path=/etc/kubernetes/manifests --pod-max-pids=100,,

Make a note of the value for the "--pod-manifest-path" (set to /etc/kubernetes/manifests) as we'll be using that later to troubleshoot the other components.

Viewing kubelet logs

Units deployed via systemd use journald for their logging.  This means we have some simple command line tools we can use to see the log details for our kubelet

# View ALL known logs for the kubelet
journalctl -u kubelet.service

# View all logs for the kubelet since the last system boot
journalctl -b -u kubelet.service

# View all logs for the kubelet in the last hour
journalctl -b -u kubelet.service --since "1 hour ago"

# View all logs for the kubelet in the last fifteen minutes
journalctl -b -u kubelet.service --since "15 minutes ago"

# View all logs for the kubelet since a given date/time
journalctl -b -u kubelet.service --since "2018-12-10 15:30:00"

For some more details on the command syntax while using systemctl and journalctl, check out these helpful guides:

Checking the status of other components running as static pods

As noted earlier, the most common kubernetes deployment involves the kubelet being managed by systemd, and the other components being run as static pods by the kubelet.  If you remember before, we saw that our kubelet was being passed a parameter of "--pod-manifest-path", with a value of "/etc/kubernetes/manifests".  So let's take a look at what we find in that directory:

cd /etc/kubernetes/manifests
kube-apiserver.yaml  kube-controller-manager.yaml  kube-scheduler.yaml

If you open any of these YAML files, you'll see they are just Pod definitions that can be understood by kubernetes.  Open up and view any of the files to see the parameters and value being sent to that particular component.

Checking the status of the static pods and viewing logs

In almost every case, these pods will be running in the "kube-system" namespace.  You can verify this by evaluating the metadata.namespace value in the YAML files as well.  We can then use kubectl to see the status of the pods, as well as gathering their logs:

# See the status of the pod, and verify what node it's running on
kubectl get pods -n kube-system -o wide

# Once you've found a specific pod you want to review, describe it
kubectl describe pod POD_NAME -n kube-system

# Review the logs for your pod
kubectl logs POD_NAME -n kube-system

What if kubectl isn't working and I need to view logs?

Let's say that you're unable to execute kubectl to view logs - perhaps the kube-apiserver isn't running and as such you can't even use any kubectl commands.  You can validate the logs of these components by SSH'ing directly to the given node and looking at the docker logs themselves.

First we need to find our container ID:

# Replace "kube-apiserver" with the name of the pod you're looking for
docker ps -a | grep kube-apiserver

You'll likely see two containers returned here.  This is because every pod in Kubernetes actually launches 1 additional container, in addition to however many containers you specify.  This is the "pause" container, and the purpose of it is to glue together the network/storage stack of all the containers in your pod.  For our purposes, we don't really care about it.  So you'll want to ignore the container that's running the "/pause" command, and instead get the container ID of the other one.

Once you have the ID, you can just use the Docker CLI to query the logs:

docker logs CONTAINER_ID
docker logs CONTAINER_ID --tail 500

If for whatever reason the docker commands aren't working, you can always view the raw log files directly.  These are usually located in /var/lib/docker/containers/CONTAINER_ID.


Hopefully this helps you get started in digging into your master nodes.  As time goes on, it's going to be more and more likely that these will be abstracted from the consumer (as is the case with GKE and AKS solutions), but having an understanding of how they work and how to troubleshoot them will still be an important skill until then.   If you have any questions or run into any problems, feel free to reach out!


Justin Carlson

Read more posts by this author.