Securing your hosted ELK stack

In my previous post, I walked through the process to deploy an ELK stack in AKS to play around with / learn / test stuff.  In that example there was NO TLS/SSL communication, and nothing was protected by passwords.

I went through and read all the documentation and figured out how to do the same thing, but to deploy a fully-secured cluster for both Elasticsearch and Kibana, and wanted to share that process as well.

You would want to read through the previous post to deploy an AKS instance to use and get chocolatey/helm installed, unless you've already done so.  For reference, you can check back here:  https://talkcloudlytome.com/setting-up-your-own-elk-stack-in-kubernetes-with-azure-aks/

Once we have an AKS cluster to deploy to, we're going to have to create some certificates to secure our ELK stack with.  The point of my blog post here is to show how to setup and secure your ELK stack.  However - it would take far longer than I want to spend to setup an AppGateway/LoadBalancer with the proper DNS names so I could get correct certificate validation.  So with this example, everything will be secured via TLS/SSL, but if you try to access it via the external IP from outside of the cluster, you WILL get certificate validation errors (as expected).

On other thing to note is that I'm using .pem certificates.  If you're using PKCS#12 certificates, there are some different settings that you use for Elasticsearch, which is all covered in the documentation listed in the reference section below.

We're going to use cfssl and cfssljson to create our certs.  Run this and save off the ca-key.pem, ca.pem, security-master-key.pem, and security-master.pem files that are created:

	
cat > ca-config.json <<EOF
{
  "signing": {
    "default": {
      "expiry": "8760h"
    },
    "profiles": {
      "elasticsearch": {
        "usages": ["signing", "key encipherment", "server auth", "client auth"],
        "expiry": "8760h"
      }
    }
  }
}
EOF

cat > ca-csr.json <<EOF
{
  "CN": "Elasticsearch",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "US",
      "L": "Hartland",
      "O": "Elasticsearch",
      "OU": "CA",
      "ST": "Wisconsin"
    }
  ]
}
EOF

cfssl gencert -initca ca-csr.json | cfssljson -bare ca

cat > security-master-csr.json <<EOF
{
  "CN": "security-master",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "US",
      "L": "Hartland",
      "O": "Elasticsearch",
      "OU": "Elasticsearch",
      "ST": "Wisconsin"
    }
  ]
}
EOF

cfssl gencert \
  -ca=ca.pem \
  -ca-key=ca-key.pem \
  -config=ca-config.json \
  -profile=elasticsearch \
  security-master-csr.json | cfssljson -bare security-master
	

Now create a file in your current directory (containing all the certificates you just created) called "kustomization.yaml" with the following contents:

	
secretGenerator:
- name: elastic-certificates
  files:
  - ca.pem
  - security-master.pem
  - security-master-key.pem
	

Run the following command to apply this secret to Kubernetes:

	
kubectl apply -k .
	

Make a note of the secret name that is output here, as you'll need it later.  You can see it in the output - it will look something like this:

	
PS > kubectl apply -k .
secret/elastic-certificates-7ft8hkbftk created
	

Next we need to create a secret that will hold the credentials that Elasticsearch will run under.  Replace "##PASSWORD##" with your actual password you want to use:

	
kubectl create secret generic elastic-credentials --from-literal=password=##PASSWORD## --from-literal=username=elastic
	

Now we're going to make and modify a values.yml file for Helm to deploy Elasticsearch with our security configuration settings.  Previously we didn't bother with a values file for our deployment, since we were only overriding one value (service.type).  However, we have to modify quite a few values here, so we'll put them all into one file.

Add the following into a file and save it as "elasticsearch-helm-values.yml".  Make sure you replace "elastic-certificates-7ft8hkbftk" in the secretMounts.secretName section with the name of the secret you created above.

	
---
clusterName: "security"
nodeGroup: "master"

roles:
  master: "true"
  ingest: "true"
  data: "true"

protocol: https

esConfig:
  elasticsearch.yml: |
    xpack.security.enabled: true
    xpack.security.transport.ssl.enabled: true
    xpack.security.transport.ssl.verification_mode: certificate
    xpack.security.transport.ssl.key: /usr/share/elasticsearch/config/certs/security-master-key.pem
    xpack.security.transport.ssl.certificate: /usr/share/elasticsearch/config/certs/security-master.pem
    xpack.security.transport.ssl.certificate_authorities: [ "/usr/share/elasticsearch/config/certs/ca.pem" ]
    xpack.security.http.ssl.enabled: true
    xpack.security.http.ssl.key: /usr/share/elasticsearch/config/certs/security-master-key.pem
    xpack.security.http.ssl.certificate: /usr/share/elasticsearch/config/certs/security-master.pem
    xpack.security.http.ssl.certificate_authorities: [ "/usr/share/elasticsearch/config/certs/ca.pem" ]
    
extraEnvs:
  - name: ELASTIC_PASSWORD
    valueFrom:
      secretKeyRef:
        name: elastic-credentials
        key: password
  - name: ELASTIC_USERNAME
    valueFrom:
      secretKeyRef:
        name: elastic-credentials
        key: username

secretMounts:
  - name: elastic-certificates
    secretName: elastic-certificates-7ft8hkbftk
    path: /usr/share/elasticsearch/config/certs
    
service:
  labels: {}
  labelsHeadless: {}
  type: LoadBalancer
  nodePort: ""
  annotations: {}
  httpPortName: http
  transportPortName: transport
	

Now we're good to install it via Helm with these values:

	
helm install elasticsearch elastic/elasticsearch --values elasticsearch-helm-values.yml
	

You should get a similar output as in the previous post:

	
NAME: elasticsearch
LAST DEPLOYED: Fri Feb 21 19:26:40 2020
NAMESPACE: default
STATUS: deployed
REVISION: 1
NOTES:
1. Watch all cluster members come up.
  $ kubectl get pods --namespace=default -l app=elasticsearch-master -w
2. Test cluster health using Helm test.
  $ helm test elasticsearch 
	

Run the "get pods" command until all the pods have a "READY" value of "1/1" and a STATUS of "Running".

Find out the External IP address of your elasticsearch service:

	
kubectl get svc security-master
	

Make a note of the "EXTERNAL-IP" that you find here.  Then go over to a browser and go to https://<YOUR-EXTERNAL-IP>:9200.  Note that we're specifying "https" instead of "http" for the protocol now!  You should see something like this:

TLS and Authentication credentials for Elasticsearch

Note that it does say "Not secure".  It IS secured with TLS, however, due to how we setup our load balancer infrastructure, we currently have to access it via IP.  If we were to access it with the host name (security-master), like it does with in-cluster communication, it would work fine and show as secure.   In a production environment, you would probably configure this a little differently.  But you can click on the "Not secure" link to see the certificate and validate that we are in fact securing communications with TLS:

The elasticsearch certificate 

Another thing you'll note that's different from our unsecured installation - we're now prompted for credentials when logging in!   This is due to us setting "xpack.security.enabled" to true in our elasticsearch.yml configuration.  To log in, use "elastic" for the username, and whatever you specified for your "elastic-credentials" secret password you created earlier.

Elasticsearch web page

Success!  We now have an elasticsearch installation that's using encrypted TLS both for node-to-node and HTTP communications, and also requires a login to authenticate.

Next we're going to setup Kibana to talk to this instance.  This is going to require additional configuration as well since we want Kibana to be accessed over HTTPS from the web browser, and we'll also need to configure it to communicate with our Elasticsearch instance over HTTPS as well.

We're going to go back to the same directory where we created our elasticsearch certs and make one for Kibana.  Run this to create the certs.  Save off the kibana-kibana.pem and kibana-kibana-key.pem files that are created:

	
cat > kibana-kibana-csr.json <<EOF
{
  "CN": "kibana-kibana",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "US",
      "L": "Hartland",
      "O": "Kibana",
      "OU": "Kibana",
      "ST": "Wisconsin"
    }
  ]
}
EOF

cfssl gencert \
  -ca=ca.pem \
  -ca-key=ca-key.pem \
  -config=ca-config.json \
  -profile=elasticsearch \
  kibana-kibana-csr.json | cfssljson -bare kibana-kibana
	

Copy those .pem files into the same directory as all your other certificates, and modify the "kustomization.yaml" file you created earlier to contain only the following contents:

	
secretGenerator:
- name: kibana-certificates
  files:
  - kibana-kibana.pem
  - kibana-kibana-key.pem
	

Run the following command to apply this secret to Kubernetes:

	
kubectl apply -k .
	

Make a note of the secret name that is output here, as you'll need it later.  You can see it in the output - it will look something like this:

	
PS > kubectl apply -k .
secret/kibana-certificates-mkt5m8644t created
	

Now we're going to make and modify a values.yml file to deploy Kibana with our security config settings. Add the following into a file and save it as "kibana-helm-values.yml".  Make sure you replace "elastic-certificates-7ft8hkbftk" and "kibana-certificates-mkt5m8644t" secret names with the respective names of the secrets you've created so far in this process!

	
---
elasticsearchHosts: "https://security-master:9200"

extraEnvs:
  - name: 'ELASTICSEARCH_USERNAME'
    valueFrom:
      secretKeyRef:
        name: elastic-credentials
        key: username
  - name: 'ELASTICSEARCH_PASSWORD'
    valueFrom:
      secretKeyRef:
        name: elastic-credentials
        key: password

kibanaConfig:
  kibana.yml: |
    server.ssl:
      enabled: true
      key: /usr/share/kibana/config/certs/kibana-kibana-key.pem
      certificate: /usr/share/kibana/config/certs/kibana-kibana.pem
    elasticsearch.ssl:
      certificateAuthorities: /usr/share/kibana/config/elasticsearchcerts/ca.pem
      verificationMode: certificate

protocol: https

secretMounts:
  - name: kibana-certificates
    secretName: kibana-certificates-mkt5m8644t
    path: /usr/share/kibana/config/certs
  - name: elastic-certificates
    secretName: elastic-certificates-7ft8hkbftk
    path: /usr/share/kibana/config/elasticsearchcerts
    
service:
  type: LoadBalancer
  port: 5601
  nodePort: ""
  labels: {}
  annotations: {}
  loadBalancerSourceRanges: []
	

Now we're good to install it via Helm with these values:

	
helm install kibana elastic/kibana --values kibana-helm-values.yml
	

You can run the following to watch the status of the Kibana pod:

	
kubectl get pods --namespace=default -l app=kibana -w
	

Once this command shows the pod(s) with a "READY" value of "1/1" and a STATUS of "Running", we're all good to go.

Find out the External IP address of the Kibana service:

	
kubectl get svc kibana-kibana
	

Make a note of the "EXTERNAL-IP" that you find here.  Then go over to a browser and go to https://<YOUR-EXTERNAL-IP>:5601.  Note that we're specifying "https" instead of "http" for the protocol now!  All the same comments about the certificate being 'wrong' are still the same here, but you can view the certificate itself to see that we're indeed using TLS:

Kibana certificate

If you click through the security warnings, you'll see this:

Kibana login page

Kibana uses the same credentials for login that Elasticsearch does, so you can use the same username/password you used to login to Elasticsearch earlier:

And we're in!

Kibana web portal

Lastly - I didn't go through and setup any Beats with this instance, but the changes are relatively minor to setup.  Basically you just need to specify an "https" when you're setting up your elasticsearch output and provide a path to the CA of your elasticsearch certificate.  You can read up some examples here if you're interested: https://www.elastic.co/guide/en/beats/filebeat/current/elasticsearch-output.html

Additional Resources:

I used the following resources when researching this process and building out this blog post:

I hope this helps you out - thanks for reading!

-Justin

Justin Carlson

Read more posts by this author.

Wisconsin