Some distributions of Kubernetes hide the master nodes away from you so you don't need to worry about them. Some examples of this are Azure AKS or Google Kubernetes Engine. In those instances, you're paying for the vendor to manage the master nodes for you, so there's no need for you to monitor or troubleshoot them.
However, in some instances you will be responsible for the master nodes. For example, if you've deployed a cluster using acs-engine, or built your own kubernetes cluster from scratch. This post will cover how to troubleshoot the main components of the master nodes.
What are the different components on the master nodes and what do they do:
- kubelet
- The kubelet is one of the main kubernetes components. This is responsible for reporting node ready status and actually interacting with the container runtime to execute pods
- etcd
- etcd is the key-value store that stores the state of the kubernetes cluster. The kube-apiserver uses this to read/write and store it's data
- kube-apiserver
- This is the API web server that kubectl and other kubernetes components interact with
- kube-controller-manager
- This component is used by kubernetes to ensure the system is in the correct desired states. For example, if you request 3 pods of a certain type, and one pod fails, the controller-manager will ensure another pod is started up
- kube-scheduler
- This component is used by kubernetes to determine the appropriate node(s) to deploy pods to
There are a few reasons you may notice something is wrong with your master nodes. Some possible scenarios that could occur if one or more components are down are:
- Unable to access kube-apiserver via kubectl
- Pods are not getting deployed to nodes
- Deployments are not creating an appropriate number of pods
- One or more nodes are not showing as Ready via kubectl
It's important to keep in mind that there are multiple different ways that a master node can be deployed. However, the most common way you'll see today is you'll have a kubelet that's managed via the systemd startup process, and then all other components are launched as static pod manifests and actually run as containers themselves. That configuration is what I'll talk about below:
Checking node status
First of all you'll want to check the node status - you can do that via kubectl:
# Get the status for all known nodes
kubectl get nodes
# Or get detailed information for a specific node
kubectl describe node NODE_NAME
Here's some sample output you might get - while some components could fail and still leave the node in a "Ready" state, if you see one as "NotReady", there's definitely a problem to look into.
kubectl get nodes
NAME STATUS ROLES AGE VERSION
1348k8s000 Ready agent 11d v1.12.2
1348k8s001 Ready agent 11d v1.12.2
1348k8s002 Ready agent 11d v1.12.2
k8s-master-13487264-0 Ready master 11d v1.12.2
k8s-master-13487264-1 NotReady master 11d v1.12.2
k8s-master-13487264-2 Ready master 11d v1.12.2Digging in to a specific node can give you some more information - you're most interested in the "Conditions" and "Events" output of this command:
kubectl describe node k8s-master-13487264-1
...
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
OutOfDisk False Wed, 19 Dec 2018 07:53:01 -0600 Fri, 07 Dec 2018 08:13:33 -0600 KubeletHasSufficientDisk kubelet has sufficient disk space available
MemoryPressure False Wed, 19 Dec 2018 07:53:01 -0600 Fri, 07 Dec 2018 08:13:33 -0600 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Wed, 19 Dec 2018 07:53:01 -0600 Fri, 07 Dec 2018 08:13:33 -0600 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Wed, 19 Dec 2018 07:53:01 -0600 Fri, 07 Dec 2018 08:13:33 -0600 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Wed, 19 Dec 2018 07:53:01 -0600 Fri, 07 Dec 2018 08:13:33 -0600 KubeletReady kubelet is posting ready status. AppArmor enabled
...
Events: none
Checking the status of the kubelet
The kubelet will almost always be deployed as a systemd process. That means we can use the systemctl commands to see the state:
systemctl status kubelet.service
● kubelet.service - Kubelet
Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2018-12-18 20:17:39 UTC; 17h ago
The most important thing is that you see the status of "loaded" and "active (running)" in the output. If you don't see that, you can always try to stop/restart/start the service with the following systemctl commands:
# Start the service
systemctl start kubelet.service
# Stop the service
systemctl stop kubelet.service
# Restart the service
systemctl restart kubelet.service
To check the configuration values passed to the kubelet, you'll want to note the .service file it's using. This was noted in the following line from the systemctl status output you ran earlier:
Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: enabled)
cat /etc/systemd/system/kubelet.service
[Unit]
Description=Kubelet
ConditionPathExists=/usr/local/bin/kubelet
Requires=kms.service
[Service]
Restart=always
EnvironmentFile=/etc/default/kubelet
SuccessExitStatus=143
ExecStartPre=/bin/bash /opt/azure/containers/kubelet.sh
ExecStartPre=/bin/mkdir -p /var/lib/kubelet
ExecStartPre=/bin/mkdir -p /var/lib/cni
ExecStartPre=/bin/bash -c "if [ $(mount | grep \"/var/lib/kubelet\" | wc -l) -le 0 ] ; then /bin/mount --bind /var/lib/kubelet /var/lib/kubelet ; fi"
ExecStartPre=/bin/mount --make-shared /var/lib/kubelet
# This is a partial workaround to this upstream Kubernetes issue:
# https://github.com/kubernetes/kubernetes/issues/41916#issuecomment-312428731
ExecStartPre=/sbin/sysctl -w net.ipv4.tcp_retries2=8
ExecStartPre=-/sbin/ebtables -t nat --list
ExecStartPre=-/sbin/iptables -t nat --list
ExecStart=/usr/local/bin/kubelet \
--enable-server \
--node-labels="${KUBELET_NODE_LABELS}" \
--v=2 \
--volume-plugin-dir=/etc/kubernetes/volumeplugins \
$KUBELET_CONFIG $KUBELET_OPTS \
$KUBELET_REGISTER_NODE $KUBELET_REGISTER_WITH_TAINTS
[Install]
WantedBy=multi-user.target
Down in the "ExecStart" you'll see what parameters are being used for the kubelet, and you'll notice there's also an "EnvironmentFile". This is what's likely to have the configuration values we need, so let's take a look at that:
cat /etc/default/kubelet
KUBELET_OPTS=
KUBELET_CONFIG=--address=0.0.0.0 --allow-privileged=true --anonymous-auth=false --authorization-mode=Webhook --azure-container-registry-config=/etc/kubernetes/azure.json --cgroups-per-qos=true --client-ca-file=/etc/kubernetes/certs/ca.crt --cloud-config=/etc/kubernetes/azure.json --cloud-provider=azure --cluster-dns=10.0.0.10 --cluster-domain=cluster.local --enforce-node-allocatable=pods --event-qps=0 --feature-gates=PodPriority=true --image-gc-high-threshold=85 --image-gc-low-threshold=80 --image-pull-progress-deadline=30m --keep-terminated-pod-volumes=false --kubeconfig=/var/lib/kubelet/kubeconfig --max-pods=30 --network-plugin=cni --node-status-update-frequency=10s --non-masquerade-cidr=0.0.0.0 --pod-infra-container-image=k8s.gcr.io/pause-amd64:3.1 --pod-manifest-path=/etc/kubernetes/manifests --pod-max-pids=100
KUBELET_IMAGE=k8s.gcr.io/hyperkube-amd64:v1.12.2
KUBELET_NODE_LABELS=kubernetes.io/role=master,node-role.kubernetes.io/master=,kubernetes.azure.com/cluster=jdc-k8s-poc
Make a note of the value for the "--pod-manifest-path" (set to /etc/kubernetes/manifests) as we'll be using that later to troubleshoot the other components.
Viewing kubelet logs
Units deployed via systemd use journald for their logging. This means we have some simple command line tools we can use to see the log details for our kubelet
# View ALL known logs for the kubelet
journalctl -u kubelet.service
# View all logs for the kubelet since the last system boot
journalctl -b -u kubelet.service
# View all logs for the kubelet in the last hour
journalctl -b -u kubelet.service --since "1 hour ago"
# View all logs for the kubelet in the last fifteen minutes
journalctl -b -u kubelet.service --since "15 minutes ago"
# View all logs for the kubelet since a given date/time
journalctl -b -u kubelet.service --since "2018-12-10 15:30:00"
For some more details on the command syntax while using systemctl and journalctl, check out these helpful guides:
- https://www.digitalocean.com/community/tutorials/systemd-essentials-working-with-services-units-and-the-journal
- https://www.digitalocean.com/community/tutorials/how-to-use-journalctl-to-view-and-manipulate-systemd-logs
- https://www.digitalocean.com/community/tutorials/understanding-systemd-units-and-unit-files
Checking the status of other components running as static pods
As noted earlier, the most common kubernetes deployment involves the kubelet being managed by systemd, and the other components being run as static pods by the kubelet. If you remember before, we saw that our kubelet was being passed a parameter of "--pod-manifest-path", with a value of "/etc/kubernetes/manifests". So let's take a look at what we find in that directory:
cd /etc/kubernetes/manifests
ls
kube-apiserver.yaml kube-controller-manager.yaml kube-scheduler.yaml
If you open any of these YAML files, you'll see they are just Pod definitions that can be understood by kubernetes. Open up and view any of the files to see the parameters and value being sent to that particular component.
Checking the status of the static pods and viewing logs
In almost every case, these pods will be running in the "kube-system" namespace. You can verify this by evaluating the metadata.namespace value in the YAML files as well. We can then use kubectl to see the status of the pods, as well as gathering their logs:
# See the status of the pod, and verify what node it's running on
kubectl get pods -n kube-system -o wide
# Once you've found a specific pod you want to review, describe it
kubectl describe pod POD_NAME -n kube-system
# Review the logs for your pod
kubectl logs POD_NAME -n kube-system
What if kubectl isn't working and I need to view logs?
Let's say that you're unable to execute kubectl to view logs - perhaps the kube-apiserver isn't running and as such you can't even use any kubectl commands. You can validate the logs of these components by SSH'ing directly to the given node and looking at the docker logs themselves.
First we need to find our container ID:
# Replace "kube-apiserver" with the name of the pod you're looking for
docker ps -a | grep kube-apiserver
You'll likely see two containers returned here. This is because every pod in Kubernetes actually launches 1 additional container, in addition to however many containers you specify. This is the "pause" container, and the purpose of it is to glue together the network/storage stack of all the containers in your pod. For our purposes, we don't really care about it. So you'll want to ignore the container that's running the "/pause" command, and instead get the container ID of the other one.
Once you have the ID, you can just use the Docker CLI to query the logs:
docker logs CONTAINER_ID
docker logs CONTAINER_ID --tail 500
If for whatever reason the docker commands aren't working, you can always view the raw log files directly. These are usually located in /var/lib/docker/containers/CONTAINER_ID.
Summary
Hopefully this helps you get started in digging into your master nodes. As time goes on, it's going to be more and more likely that these will be abstracted from the consumer (as is the case with GKE and AKS solutions), but having an understanding of how they work and how to troubleshoot them will still be an important skill until then. If you have any questions or run into any problems, feel free to reach out!
-Justin