Credits: elastic.co

In past few months, I have had wonderful opportunities to explore or work on interesting techs like K8S, Terraform, Gitlab CI etc. With fast paced nature of innovations, tooling and announcements in these domains, sometimes it becomes really difficult to keep in sync with everything going around and at the same time incorporating all such learnings in our daily job roles/work

Faced with similar situations, I tried out something different this week — a quick POC on enabling logging for applications deployed on a K8S cluster. I narrowed down on this use-case because the learnings I would gather here would definitely help me at some point with real application deployments on K8S.

So, the idea was to spin up a Kubernetes cluster (on Minikube) on my local machine or AWS and setup ElasticStack (ElasticSearch, Kibana and Beats) on top of it to enable all application containers to log data. Finally, I wanted to setup a CI/CD pipeline to test/rollout new changes to the ElasticStack Cluster itself.

It would be even better to have this infrastructure/configuration defined as code so that the experiment is repeatable for myself or someone else who wants to try out a similar setup on AWS in future :)

I have had multiple issues trying out Minikube on my Macbook in the past (primarily related to networking, lags and bugs with Minikube tooling itself). Hence, this time I wanted to set it up all on an AWS EC2 instance backed by Linux instead. However, due to limited/costly virtualisation capabilities in EC2 (metal instance types), I was limited to setting up the K8S components as Docker containers (which Minikube does support).

Spoiler : All scripts/code are available in a Github repository linked at the end of the post :)

Summary of things that I could achieve:

Spin up an EC2 instance on AWS with Terraform

Automated provisioning of Gitlab Community Edition Server on the EC2 instance

Automated provisioning of test K8S cluster (Backed by Minikube) on the EC2 instance

Setup a new repository (with pipelines) on the provisioned Gitlab Server to test/rollout the ElasticStack Platform

Execute the pipeline to setup ElasticStack (if does not exist), and then perform some basic tests if all works fine. The same suite can then run every time we make some changes to the ElasticStack configurations.

Access Kibana dashboard on your local machine with traffic tunnelled to the EC2 instance

The entire workflow in the end looked something like-

Now on to the actual implementation steps:

Spin up an EC2 instance on AWS with Terraform

First things first — We need a publicly accessible EC2 instance which allows traffic only from your machine (as a security measure), an access key-pair which you can use to login into the instance and basic VPC/Subnet configurations

data "template_file" "init" {

template = "${file("${path.module}/setup.sh")}"

} resource "aws_instance" "esk8s_instance" {

ami = "${var.instance_ami_id}"

instance_type = "${var.instance_type}"

associate_public_ip_address = true

subnet_id = "${var.instance_subnet_id}"

user_data = "${data.template_file.init.rendered}"

key_name = "${var.instance_key_name}" lifecycle {

create_before_destroy = true

} root_block_device {

volume_size = 200

} tags = {

Name = "esk8s instance"

Application = "minikube"

Environment = "test"

Description = "Test instance"

}

} resource "aws_security_group" "esk8s_instance_sg" {

vpc_id = "${var.instance_vpc_id}"

name_prefix = "esk8s_instance_sg"

} resource "aws_security_group_rule" "esk8s_sg_ingress_1" {

from_port = 0

protocol = "All"

security_group_id = "${aws_security_group.esk8s_instance_sg.id}"

to_port = 65535

type = "ingress"

cidr_blocks = "${var.ec2_ingress_cidr}"

} resource "aws_security_group_rule" "esk8s_sg_ingress_2" {

from_port = 80

protocol = "tcp"

security_group_id = "${aws_security_group.esk8s_instance_sg.id}"

to_port = 80

type = "ingress"

cidr_blocks = ["0.0.0.0/0"]

} resource "aws_security_group_rule" "sg_egress" {

from_port = -1

protocol = "All"

security_group_id = "${aws_security_group.esk8s_instance_sg.id}"

cidr_blocks = ["0.0.0.0/0"]

to_port = -1

type = "egress"

}

2. Automated provisioning of Gitlab CE Server, Docker and Minikube on the EC2 instance

In order to achieve automatic setup of these on EC2 instance, I used a cloud-init script which triggers once the instance boots up

After the execution of the script is complete, Gitlab CE would be listening at the public IP/DNS (port 80) associated with the EC2 instance and can be accessed in the browser. To keep the entire setup simple, I configured the Gitlab CE server to be run as a Dockercontainer as well.

At this point, we can setup a new git repository which would then be used to host the pipeline configuration as well as the Kubernetes manifests to test/roll-out new changes of ElasticStack Cluster on K8S.

Alongwith Gitlab CE, we would also have Minikube, Docker and Gitlab-Runner installed on EC2 to support the next steps.

3. Setting up Gitlab pipeline to enable CI / CD workflow

stages:

- test

- deploy

variables:

NAMESPACE: "default"



test_stack:

stage: test

variables:

NAMESPACE: "test"

script:

- kubectl create namespace ${NAMESPACE} || true

- kubectl apply -f elasticstack/elasticsearch.yml --namespace=${NAMESPACE}

- kubectl apply -f elasticstack/kibana.yml --namespace=${NAMESPACE}

- kubectl apply -f elasticstack/filebeat-configmap.yml --namespace=${NAMESPACE}

- kubectl apply -f elasticstack/filebeat-role.yml --namespace=${NAMESPACE}

- kubectl apply -f elasticstack/filebeat-role-binding.yml --namespace=${NAMESPACE}

- kubectl apply -f elasticstack/filebeat-service-account.yml --namespace=${NAMESPACE}

- kubectl apply -f elasticstack/filebeat-daemonset.yml --namespace=${NAMESPACE}

- elasticstack/test/check_elasticsearch_data_flow



deploy_stack:

when: manual

variables:

NAMESPACE: "live"

stage: deploy

script: "echo 'This is the deploy stage'"

Gitlab pipeline definitions follow a simple yaml definition structure. We can define various stages like test and deploy for example. Kubernetes supports the concept of namespaceswhich are logical isolations/clusters of the same physical cluster. So we can also map the pipeline stages with respective namespaces to try out cluster changes in isolation before rolling them out to real clusters.

4. Setting up ElasticStack on the cluster

As a final step, we can just run the pipeline in the gitlab repository to trigger some commands like:

kubectl apply -f elasticstack/elasticsearch.yml --namespace=${NAMESPACE}

Kubernetes supports concepts of different types of resources like DaemonSets, ReplicaSets, NodePorts etc. which solve a specific problem and can be utilised to provision ElasticStack components like ElasticSearch, Kibana and FileBeat etc. For example, we don’t need to expose the ElasticSearch/Filebeat services outside the cluster but Kibana. Accordingly, there were basic set of manifest definitions which I used from Elastic’s github repository to run these components.

5. Accessing Kibana from remote EC2 instance locally

Once the entire setup is operational, we can access Kibana on the local environment. This requires tunnelling traffic over SSH to the EC2 instance. This is needed because Kibana is only reachable over a private IP within the VM.

Setting up the tunnel was as simple as:

ssh -N -L 5601:<kibana_cluster_ip>:5601 -i <your_aws_key_pair>.pem ec2-user@ec2-xx-xxx-xxx-xxx.eu-central-1.compute.amazonaws.com

All the above scripts and commands are available in the Github Repository with more detail.

I hope this gives you some idea on getting an Elastic Stack up and running on Minikube in AWS in an automated way. If you have more ideas on how to extend such an experiment even further, please do share :)

Until next time, tschüss !