As Kubernetes continues to grow in popularity and has become the de facto standard for orchestrating container workloads, no wonder Kubernetes and Helm are now a standard pairing when it comes to describing a company’s monitoring stack.

The Elastic Stack (also known as ELK) integrates natively with Kubernetes and is a popular open-source solution for collecting, storing and analyzing Kubernetes telemetry data.

While deploying the ELK Stack using Kubernetes might seem like a complex task, there are more and more best practices around this scenario as well as Kubernetes-native solutions. One of these solutions is using Helm charts.

Overview of the standard ELK logging solution

Logs : Server logs to be analyzed are identified

: Server logs to be analyzed are identified Logstash : Data aggregation and processing

: Data aggregation and processing Elasticsearch : Indexing and storage

: Indexing and storage Kibana: Analysis and visialization

Before we dive into the actual setup, let’s quickly explore the possible alternatives of running the ELK stack on Kubernetes.

Elasticserach setup — SaaS or self-managed?

Many cloud providers offer Elasticsearch as a Service and it may seem appealing for a company to minimize their effort in building the solutions. The ease of maintenance, however, comes with the following considerations:

Elasticsearch as a Service

Security: cloud-based Elasticsearch solutions often lack basic ELK security functionalities like RBAC — most notably the X-Pack plugin is not supported on the AWS Elasticsearch offering. Therefore the flexibility for doing security right is non-existent and would most certainly require additional effort in the long term.

cloud-based Elasticsearch solutions often lack basic ELK security functionalities like RBAC — most notably the X-Pack plugin is not supported on the AWS Elasticsearch offering. Therefore the flexibility for doing security right is non-existent and would most certainly require additional effort in the long term. Functionality: the most popular Elasticsearch cloud solutions lack shard rebalancing capabilities which are a critical bit in a large production environment, thus requiring some manual effort to move the indexes to a new node in case of a node failure.

Additional plugins such as analyzer plugins and ingest plugins are also not supported on the SaaS options.

the most popular Elasticsearch cloud solutions lack shard rebalancing capabilities which are a critical bit in a large production environment, thus requiring some manual effort to move the indexes to a new node in case of a node failure. Additional plugins such as analyzer plugins and ingest plugins are also not supported on the SaaS options. Control: managed solutions rarely provide full control over the Elasticsearch settings. There is usually very limited support for configuration changes and performance optimizations.

managed solutions rarely provide full control over the Elasticsearch settings. There is usually very limited support for configuration changes and performance optimizations. Maintenance: backup frequency options are usually limited to once a day. New versions are released quite late compared to the official release date from Elastic. Upgrades are usually a painful process as they often require setting up a brand new cluster for the new version.

backup frequency options are usually limited to once a day. New versions are released quite late compared to the official release date from Elastic. Upgrades are usually a painful process as they often require setting up a brand new cluster for the new version. Visibility: monitoring and cluster metrics are quite limited too. Logs such as complaints, warnings, GC slow logs are not available

monitoring and cluster metrics are quite limited too. Logs such as complaints, warnings, GC slow logs are not available Cost: managed cloud services come with predefined options for instance type and number which results in increased cost compared with the self set up solution.

Elasticsearch on Kubernetes

According to Elastic, the company that powers the ELK stack, these are just a few of the benefits this setup presents:

Straightforward deployment and management of multiple Elasticsearch clusters, including Kibana

of multiple Elasticsearch clusters, including Kibana Seamless upgrades to new versions of the Elastic Stack

to new versions of the Elastic Stack Simple scaling that allows you to grow with your use cases

that allows you to grow with your use cases Default security on every cluster

Elastic provide the option to have the Elasticsearch & Kibana on Kubernetes running on the Elastic Cloud managed solution, you can review their offering if SaaS is the preferred option for you.

If you are willing to explore the Elastic setup on Kubernetes with Helm 3, now that we have an understanding of the potential benefits of the Elasticsearch setup on Kubernetes, you can proceed with the steps of this article.

Below is a overview of how this setup would look like.

Step 1: Creating the Kubernetes Cluster

For testing purposes, you can use minikube-version 1.15.6 which ban be installed as shown in the Kubernetes Minicube documentation.

Alternatively, the cluster can be deployed on one of the major Cloud providers offering this service (EKS or GKE). Our setup will be running on EKS version 1.15. You can refer to the official Cloud Provider Documentation for setting this up easily.

Step 2: Helm 3 installation

As simple as following the official installation guides

Step 3: Deploying the Elasticsearch cluster

For that , we will be using the official Elastic Helm charts, available on Github.

$ helm repo add elastic https://helm.elastic.co

You may refer to the numerous installation and setup examples available in the repository.

For our setup, we will create a directory structure similar to the one below and create the values.yaml file in the relevant directory for each component:

├── elasticsearch

│ └── values.yaml

├── kibana

│ └── values.yaml

└── metricbeat

└─── values.yaml

Pay special attention to the resources config for CPU and Memory usage and make sure the values.yaml are properly updated for a prod-grade deployment once you are done with the testing.

By default, for elasticsearch we’ll be creating PersistentVolumeClaim with a capacity of 1Gi for each pod in the cluster to prevent data loss in case of accidental pod deletion. For production workloads, the volume claim template should be defined with the desired storage capacity and (optionally) the Kubernetes storage class to associate with the persistent volume. The name of the volume claim must always be elasticsearch-data .

Elasticsearch values.yaml

For the actual installation , we will use the following script . Place it into the root directory of your project, so that it has access to the 3 sub-directories.

Feel free to enhance the script or change the provided defaults to meet your needs, as we’ve tried to keep it to minimum for simplicity:

ELK Installation script

This script must be run with 2 parameters Usage:

./elk-setup.sh [install|update] [elasticsearch|kibana|metricbeat|filebeat] Example:

./elk-setup.sh install kibana

Running the above script with the install elasticsearch option should produce the following output:

NAME: elasticsearch

LAST DEPLOYED: Thu Apr 1 17:28:20 2020

NAMESPACE: default

STATUS: DEPLOYED

NOTES:

1. Watch all cluster members come up.

$ kubectl get pods --namespace=default -l app=elasticsearch-master -w

2. Test cluster health using Helm test.

$ Helm test elasticsearch

As shown on the hint above, you can watch the status of the cluster with the following commands:

$ kubectl get pods --namespace=default -l app=elasticsearch-master -w $ helm test elasticsearch

Once the deployment is complete, would see the elasticsearch pods appear:

NAME READY STATUS RESTARTS AGE

elasticsearch-master-0 1/1 Running 0 1m

elasticsearch-master-2 1/1 Running 0 1m

elasticsearch-master-1 1/1 Running 0 1m

Complete the setup by testing it locally. You can access the cluster by performing a port forward to your local machine.

$ kubectl port-forward svc/elasticsearch-master 9200

And the output:

Step 4: Deploying Kibana

For the Kibana deployment, we will proceed with the exact same approach, only updating the relevant chart and values.yaml for Kibana:

.. and running the install script with the Kibana option:

$ ./elk-setup install elasticsearch

Output below:

NAME: kibana

LAST DEPLOYED: Thu Apr 1 09:52:21 2020

NAMESPACE: default

STATUS: DEPLOYED

.. and a list of the running pods

kubectl get pods

NAME READY STATUS RESTARTS AGE

elasticsearch-master-0 1/1 Running 0 15m

elasticsearch-master-1 1/1 Running 0 15m

elasticsearch-master-2 1/1 Running 0 15m

kibana-kibana-6d7466b9b9-fbmsz 1/1 Running 0 2m

Port-forward:

$ kubectl port-forward deployment/kibana-kibana 5601

Now you can access Kibana in your browser at: http://localhost:5601

Step 5: Deploying metricbeat

The next step will be to deploy metricbeat.

It is one of the commonly used beats provided by Elastic, responsible for sending metrics data. It can send data directly to Elasticsearch or via Logstash, where you can further process and enhance the data, before visualizing it in Kibana.

Below is a full list of all beats, provided by Elastic. Each one of them is responsible for gathering and shipping certain kind of data to the elasticsearch cluster. Take a moment to familiarize yourself with the Beats Documentation and as we won’t go in too much detail regarding each one of them in this article.

$ ./elk-setup install metricbeat

metricbeat-values.yaml

If we inspect the metricbeat-values.yaml file closely, we will notice that it configures various metric-collecting modules (kubernetes, system, etc). Metricbeat modules define what specific metrics to collect from the service, defines the frequency in which to collect them, and how to connect to it. Modules are comprised of one or multiple metricsets — literally, a set of related metrics to be collected and shipped.

Modules breakdown:

metricbeat.yml file that provides a general configuration for Metricbeat (contains system and kubernetes modules collected from the kubelet on each node)

file that provides a general configuration for Metricbeat (contains system and kubernetes modules collected from the kubelet on each node) kube-state-metrics-metricbeat.yml — kube-state-metrics is a simple service that listens to the Kubernetes API server and generates metrics about the state and health of the various objects inside Kubernetes, such as deployments, nodes and pods.

We’ll deploy Metricbeat as a DaemonSet on each node of the k8s cluster. By deploying metricbeat as a DaemonSet we ensure we get a running metricbeat daemon on each node of the cluster.

Please note that in the configuration, all metricsets with the state_ prefix require hosts field pointing to kube-state-metrics service within the cluster, while the rest should be pointed to kubelet service.

Let’s test the setup:

$ curl localhost:9200/_cat/indices



Handling connection for 9200

green open .kibana_task_manager_1 1 1 2 2 55.5kb 27.6kb

green open .apm-agent-configuration 1 1 0 0 566b 283b

green open ilm-history-1-000001 1 1 18 0 50.9kb 25.4kb

green open .kibana_1 1 1 77 6 2.2mb 1.1mb

green open metricbeat-7.6.1 1 1 1512060 0 1.7gb 910.5mb

By default, the Prometheus module for Metricbeat exposes the /metrics endpoint which means that these metrics can also be collected by an existing Prometheus setup.

Step 6: Deploying Filebeat

$ ./elk-setup install filebeat Release "filebeat-1586246704" has been upgraded. Happy Helming!

NAME: filebeat-1586246704

LAST DEPLOYED: Wed Apr 8 12:48:24 2020

NAMESPACE: log

STATUS: deployed

REVISION: 4

TEST SUITE: None

NOTES:

1. Watch all containers come up.

$ kubectl get pods --namespace=log -l app=filebeat-1586246704-filebeat -w

Updated filebeat

Step 7: Putting it all together

If we inspect the created setup, we can inspect the Helm release status (some fields have been removed for simplicity):

$ helm ls NAME NAMESPACE REVISION STATUS CHART APP VERSION

elasticsearch log 1 deployed elasticsearch-7.6.1 7.6.1

kibana log 1 deployed kibana-7.6.1 7.6.1

metricbeat log 1 deployed metricbeat-7.6.1 7.6.1

filebeat log 1 deployed filebeat-7.6.1 7.6.1

To see the detailed information about created Kubernetes resources, you can run the following command:

$ helm get manifest <RELEASE-NAME> | kubectl get -f -

Sample output:

$ helm get manifest kibana-1585817443 | kubectl get -f -

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE

service/kibana-1585817443-kibana ClusterIP 10.20.5.20 <none> 5601/TCP 25h NAME READY UP-TO-DATE AVAILABLE AGE

deployment.apps/kibana-1585817443-kibana 1/1 1 1 25h

Step 9: Analyzing data in Kibana

Now that we have all components deployed, in Kibana we can go to the Management → Kibana → Index Patterns page, and click Create index pattern. Kibana will automatically identify and display the Metricbeat index.

Enter ‘metricbeat-*’ and on the next step select the @timestamp field to finalize the creation of the index pattern in Kibana.

Hop on over to the Discover page. You’ll see all the metrics being collected from your Kubernetes cluster by Metricbeat displayed:

If you navigate through the Kibana UI, you can explore the various fields, dashboards and statistics it provides.

Jump to the Machine Learning tab and select the metricbeat-* index to visualize plenty of Kubernetes ans System metrics which we configured earlier.

http://localhost:5601/app/infra#/infrastructure/inventory

Check out the infrastructure inventory and select the resource of interest in order to explore its metrics and view its log files.

Kubernetes service discovery

A word regarding the Kibana dashboards:

Kibana dashboards for Kubernetes are quite useful as they give us a good overview of our Kubernetes cluster and its components. A very nice Dashboard explorer is available on Elastic Demo Page.

At this point, the Elastic Helm charts for not enable the Filebeat and Metricbeat dashboards by default. How this can be achieved, is available in the official documentation and would require some additions to the default values.yaml files:

Metricbeat Dashboard Setup

Filebeat Dashboard Setup

Final words

You have been presented with a simple setup for deploying the Elastic Stack (Elasticsearch, Kibana, Metricbeat, Filebeat) on Kubernetes using the official Elastic Helm Charts.

Now that we have all these components up and running on our cluster, there are endless possibilities for potential enhancements: high availability and high performance cluster configurations, customized retention strategies, security enhancements, user accessibility improvements and many more. You are free to explore the endless options and experiment with them in the same manner as if this was a traditional deployment on a VM hosted locally or somewhere in the cloud.

Resources