I recently was looking into auditing available for Kuberentes. Kubernetes has the option to enable built-in auditing that can show almost anything that is done to the system - who added pods, who deleted services, who viewed secrets, etc. You can see the changes done by users via the kubectl CLI, as well as changes that the system itself (such as the kube-scheduler or kube-controller) made.
There are two main components to enabling auditing in Kubernetes. First of all, you need an "Audit Policy", which essentially tells the kube-apiserver component what you want audited. Secondly, you need to define WHERE you want the audits reported to. For the purposes of this example, I'm using a cluster deployed by aks-engine. In the default configuration, auditing is already turned on and writes the logs to disk on the kube-apiserver component. You can read more about how to do this here.
So, I already have auditing enabled and writing to a log file on the local Linux master node. However, there's a pretty neat feature called the "webhook backend", where you can have the kube-apiserver pass the audits to a web endpoint. I was struggling to find some good documentation on how to do this, so I figured I would setup an example and get it working.
Essentially - how it works is as follows:
- Enable auditing in your cluster via an Audit Policy (already done for us in our aks-engine deployment)
- Deploy a web endpoint somewhere (language doesn't matter - I wrote mine in ASP.NET core, but you can use anything) that is capable of accepting a POST with a JSON body (and then doing something with it)
- Creating a kubeconfig file that has the address of your endpoint in it
- Passing in the "--audit-webhook-config-file" parameter to the kube-apiserver startup parameters pointing to the kubeconfig file you just created
Let's walk through an example!
First of all - this is what will be posted to your webhook endpoint - an "EventList" in the audit.k8s.io/v1 namespace, which has an "items" array, where each object in that array is an "Event" type in the audit.k8s.io/v1 namespace:
(If you're interested in seeing the structure of the actual class used, you can see it in the kube-apiserver source code here)
{
"kind": "EventList",
"apiVersion": "audit.k8s.io/v1",
"metadata": {},
"items": [
{
"level": "Request",
"auditID": "b7699b7e-e876-4c97-9b18-f7e7adc18841",
"stage": "ResponseComplete",
"requestURI": "/api/v1/nodes?limit=500",
"verb": "list",
"user": {
"username": "https://sts.windows.net/147a2b71-5ce9-4933-94c4-2054328de565/#a7b4eb91-181f-4b91-a405-c4d904f1af0f",
"groups": [
"29243834-aec2-4872-b903-661512b6ec08",
"0e22bf18-6762-4e50-b489-6eb39b652962",
"system:authenticated"
]
},
"sourceIPs": [
"74.203.144.5"
],
"userAgent": "kubectl.exe/v1.10.11 (windows/amd64) kubernetes/637c7e2",
"objectRef": {
"resource": "nodes",
"apiVersion": "v1"
},
"responseStatus": {
"metadata": {},
"status": "Failure",
"reason": "Forbidden",
"code": 403
},
"requestReceivedTimestamp": "2019-02-06T14:45:25.277447Z",
"stageTimestamp": "2019-02-06T14:45:25.277756Z",
"annotations": {
"authorization.k8s.io/decision": "forbid",
"authorization.k8s.io/reason": ""
}
},
{
"level": "Request",
"auditID": "2cf8ad1e-25b1-49d7-bb0d-b56341587c12",
"stage": "ResponseComplete",
"requestURI": "/api/v1/nodes?limit=500",
"verb": "list",
"user": {
"username": "https://sts.windows.net/147a2b71-5ce9-4933-94c4-2054328de565/#a7b4eb91-181f-4b91-a405-c4d904f1af0f",
"groups": [
"29243834-aec2-4872-b903-661512b6ec08",
"0e22bf18-6762-4e50-b489-6eb39b652962",
"system:authenticated"
]
},
"sourceIPs": [
"74.203.144.5"
],
"userAgent": "kubectl.exe/v1.10.11 (windows/amd64) kubernetes/637c7e2",
"objectRef": {
"resource": "nodes",
"apiVersion": "v1"
},
"responseStatus": {
"metadata": {},
"status": "Failure",
"reason": "Forbidden",
"code": 403
},
"requestReceivedTimestamp": "2019-02-06T14:46:36.735833Z",
"stageTimestamp": "2019-02-06T14:46:36.735963Z",
"annotations": {
"authorization.k8s.io/decision": "forbid",
"authorization.k8s.io/reason": ""
}
}
]
}
For my web backend, I created a simple ASP.NET WebAPI application that has a single endpoint, which takes a POST with an input parameter of "dynamic", as shown here:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading.Tasks;
using Microsoft.AspNetCore.Mvc;
using Newtonsoft.Json;
using Newtonsoft.Json.Linq;
namespace kubernetes_audit_webhook.Controllers
{
[Route("api/[controller]")]
[ApiController]
public class AuditsController : ControllerBase
{
// POST api/audits
[HttpPost]
public void Post(dynamic auditBody)
{
Console.WriteLine($"Received Audit Webhook Post at {DateTime.UtcNow.ToString("yyyy-MM-dd HH:mm:ss")}");
try
{
var parsedJson = JObject.Parse(auditBody.ToString());
JArray itemsArray = (JArray)parsedJson["items"];
foreach (var auditEvent in itemsArray.Children())
{
var timestamp = auditEvent["stageTimestamp"];
var level = auditEvent["level"];
var stage = auditEvent["stage"];
var requestURI = auditEvent["requestURI"];
var verb = auditEvent["verb"];
var username = "UNKNOWN";
var userElement = auditEvent["user"];
if (userElement != null)
{
var userNameElement = userElement["username"];
if (userNameElement != null)
{
username = userNameElement.ToString();
}
}
string sourceIPValue = "";
JArray sourceIPArray = (JArray)auditEvent["sourceIPs"];
if (sourceIPArray.Count == 0)
{
sourceIPValue = "UNKNOWN";
}
else if (sourceIPArray.Count == 1)
{
sourceIPValue = sourceIPArray[0].ToString();
}
else
{
var ipAddresses = new List();
foreach (JToken ipAddress in sourceIPArray)
{
ipAddresses.Add(ipAddress.ToString());
}
sourceIPValue = String.Join(",", ipAddresses);
}
var resourceType = "UNKNOWN";
var resourceName = "UNKNOWN";
var objectRefElement = auditEvent["objectRef"];
if (objectRefElement != null)
{
var resourceTypeElement = objectRefElement["resource"];
if (resourceTypeElement != null)
{
resourceType = resourceTypeElement.ToString();
}
var resourceNameElement = objectRefElement["name"];
if (resourceNameElement != null)
{
resourceName = resourceNameElement.ToString();
}
}
var authorizationDecision = "UNKNOWN";
var authorizationReason = "UNKNOWN";
var annotationsElement = auditEvent["annotations"];
if (annotationsElement != null)
{
authorizationDecision = annotationsElement["authorization.k8s.io/decision"].ToString();
authorizationReason = annotationsElement["authorization.k8s.io/reason"].ToString();
}
// starting each actual record line with "###" so I can distinguish them in the log file from any errors
Console.WriteLine($"###{timestamp},{level},{stage},{requestURI},{verb},{username},{sourceIPValue},{resourceType},{resourceName},{authorizationDecision},{authorizationReason}");
}
}
catch (Exception ex)
{
Console.WriteLine($"UNABLE TO PROCESS RECORD DUE TO ERROR: {ex.Message} - THE FULL POST BODY WILL BE SHOWN BELOW:");
Console.WriteLine(auditBody.ToString());
}
}
}
}
You can see the whole source code for the project here.
Basically, this will take that JSON that was shown earlier, use Newtonsoft.Json to parse it out into values I care about, and just write out a line to STDOUT. You could build and deploy this wherever you want, but I chose to deploy it into my actual Kubernetes cluster that I'm auditing.
First, clone the repo and build the Docker image:
(This is built with on Windows Server Build 1803, so you'll need a server 1803 box to build it, but it's .NET core so you should be able to change it to build on Linux if you desire - just change the Dockerfile to use the appropriate tags)
git clone https://github.com/carlsoncoder/kubernetes-audit-webhook.git
cd kubernetes-audit-webhook
docker build -t kubernetes-audit-image .
Then, you'll want to upload this to a container registry somewhere so you can pull it down. I chose to use Azure Container Registry:
docker login yourcontainerregistry.azurecr.io -u userName -p password
$auditImage = yourcontainerregistry.azurecr.io/k8s-audit-webhook:build-1803-v1
docker tag kubernetes-audit-image $auditImage
docker push $auditImage
Now we can deploy our image into a service and deployment in Kubernetes via kubectl:
First create a namespace to put everything in:
kubectl create namespace auditing
Then, make and submit a YAML file to create the service that we'll use to access the pods:
apiVersion: v1
kind: Service
metadata:
name: audit-webhook-service
namespace: auditing
spec:
selector:
app: audit-webhook
ports:
- protocol: TCP
port: 80
Lastly, create the deployment YAML file to make the deployment and submit it via kubectl. Note that you'll want to modify the "imagePullSecrets" to be the actual secret name that you have in your cluster to reach your container registry:
(Side note - the "command" portion in the spec really shouldn't be necessary - it's exactly the same as what's defined in the ENTRYPOINT of the Dockerfile for the image. But for some reason .NET core would fail without this...?)
apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: audit-webhook-deployment
namespace: auditing
labels:
app: audit-webhook
spec:
replicas: 1
selector:
matchLabels:
app: audit-webhook
template:
metadata:
labels:
app: audit-webhook
spec:
containers:
- name: audit-webhook-application
image: yourcontainerregistry.azurecr.io/k8s-audit-webhook:build-1803-v1
command: ["dotnet.exe", "kubernetes-audit-webhook.dll"]
ports:
- containerPort: 80
imagePullSecrets:
- name: regcred
Now we have our deployment backed by a service up and running. If you recall from the code sample, the POST endpoint is available at /api/audits. We have a service called "audit-webhook-service" in the "auditing" namespace, so with kube-dns running in our cluster, the full address for a pod to reach our service is:
http://audit-webhook-service.auditing.svc.cluster.local/api/audits
We're almost there! Now create a file called "audit-webhook-kubeconfig" in the /etc/kubernetes directory on your master node, and use vim or a text editor to put the following text in it:
apiVersion: v1
clusters:
- cluster:
server: http://audit-webhook-service.auditing.svc.cluster.local/api/audits
name: audit-webhook-service
contexts:
- context:
cluster: audit-webhook-service
user: ""
name: default-context
current-context: default-context
kind: Config
preferences: {}
users: []
NOTE: For some reason, kube-dns doesn't seem to always work - I've run into issues where the kube-apiserver logs show that the kube-apiserver cannot resolve the "audit-webhook-service.auditing.svc.cluster.local" endpoint, while the host node can. If you run into this issue, you can just replace the FQDN host name with the IP of your "audit-webhook-service" service entry.
All we're really doing here is defining the endpoint. I believe you can also use the "users" section to do some basic auth if your endpoint requires it, but I really didn't look into that.
Next we need to set the "--audit-webhook-config-file" parameter for the kube-apiserver. In aks-engine (and pretty much all other deployments), kube-apiserver is launched by the kubelet as a static pod. This means the config for kube-apiserver will likely be in a YAML file, somewhere in a directory like /etc/kubernetes/manifests. Find the YAML definition, and add the following line there to the startup parameters:
"--audit-webhook-config-file=/etc/kubernetes/audit-webhook-kubeconfig"
And lastly we need to restart the kubelet service and delete the kube-apiserver pod to make the changes take effect:
sudo systemctl restart kubelet.service
kubectl get pods -n kube-system # locate the name of the kube-apiserver pod
kubectl delete pod name-of-apiserver-pod -n kube-system
Deleting the pod will force the kubelet to recreate it with our new parameters.
Since all our application does is parse the JSON and then output it to STDOUT, we can just view the kubectl logs for that pod to see that it's working:
kubectl get pods -n auditing # locate the name of the audit-webhook pod
kubectl logs audit-webhook-pod-name -n auditing
You should see some output like this:
###2/18/2019 6:55:56 PM,Metadata,ResponseComplete,/apis/events.k8s.io/v1beta1/events?resourceVersion=1442977&timeout=7m55s&timeoutSeconds=475&watch=true,watch,client,10.255.255.5,events,UNKNOWN,allow,
###2/18/2019 6:55:57 PM,Request,ResponseComplete,/api/v1/namespaces/kube-system/endpoints/kube-controller-manager?timeout=10s,get,client,10.255.255.5,endpoints,kube-controller-manager,allow,
###2/18/2019 6:55:57 PM,Request,ResponseComplete,/api/v1/namespaces/kube-system/endpoints/kube-scheduler?timeout=10s,get,client,10.255.255.5,endpoints,kube-scheduler,allow,
###2/18/2019 6:55:58 PM,Metadata,ResponseComplete,/apis/policy/v1beta1/poddisruptionbudgets?resourceVersion=1095915&timeout=9m38s&timeoutSeconds=578&watch=true,watch,client,10.255.255.5,poddisruptionbudgets,UNKNOWN,allow,
###2/18/2019 6:55:59 PM,Request,ResponseComplete,/api/v1/namespaces/kube-system/endpoints/kube-controller-manager?timeout=10s,get,client,10.255.255.5,endpoints,kube-controller-manager,allow,
And that's all you need to get it going! So what are some other things that could be done to further improve this process?
- Configure the webhook service to run with TLS/SSL instead of just plain HTTP over port 80
- Use credentials or a certificate to authenticate to your web endpoint prior to POSTing
- Integrate one of the kubernetes client libraries so you can cast the JSON object directly to an object spec instead of doing all the JSON parsing yourself
- Have your pod image post the data to OMS instead of writing to STDOUT
- Other fun things!
There's so many cool things you can do with Kubernetes. It's really exciting every time I find out about some new feature like this and figure out how to make it work. I hope you enjoyed the walkthrough, and feel free to reach out if you have any questions!
-Justin