The Elastic stack which was formerly known as ELK stack (Elasticsearch, Logstash, Kibana) is a popular and opensource log analytical tool and has use cases in different functional areas where huge amounts of data are generated everyday. One can read these heterogeneous data, extract the relevant portion and visualize them in real time using elastic stack. The fourth element of the stack is beat(data shippers) which was added to ELK stack and renamed it to elastic stack.

You may face hurdles while installing/configuring elastic stack if you are naive at it and may even consume a little extra time from your estimate. The automation of installation procedure using Docker will reduce the time and complexities of installation procedure to a great extent. So lets start the procedure right from installing Docker to visualizing Apache logs in Kibana Dashboard.

Install Docker

Install Docker with the following set of commands. Skip the step if you have already setup environment for Docker.

$ yum install wget

$ wget -qO- https://get.docker.com/ | sh

$ systemctl enable docker.service

$ systemctl start docker.service

$ systemctl status docker.service

Install Docker Compose

With Docker compose it is easy to define/configure services for container and setup environment for an application in a YAML file and we also need it since we are going to use docker compose to define multi-container elastic stack.

$ yum install -y epel-release

$ yum install -y python-pip

$ pip install --upgrade pip

$ pip install docker-compose

Elasticsearch container

We will begin containerizing elastic stack starting with elasticsearch. First of all, create a root folder where each component of elastic stack will be clubbed together.

$ mkdir ~/docker-elk

Navigate to the root folder of elastic stack and create folders for elasticsearch and associated configurations/storage for elasticsearch.

$ cd ~/docker-elk

$ mkdir -p elasticsearch/{config,storage}

$ chown -R 1000:1000 elasticsearch/storage/

The config and storage folders will be used to define docker volumes in the docker compose file at later stage. The docker volume will enable a folder in the host machine remain attached with the folders in the containers and will always remain in sync with each other.

Create a Dockerfile for elasticsearch which is way to place all the commands needed to assemble an image.

$ cd ~/docker-elk/elasticsearch

$ vi Dockerfile

ARG ELK_VERSION

FROM docker.elastic.co/elasticsearch/elasticsearch:${ELK_VERSION}

Define the version of elasticsearch that you want to containerize in the environment file.

$ vi ~/docker-elk/.env

ELK_VERSION=7.5.1

Now proceed with creating elasticsearch configuration file in the config folder. Having ES configuration file in the host machine will enable us to easily tweak the settings and mount it in the container using docker volume.

$ vi ~/docker-elk/elasticsearch/config/elasticsearch.yml

cluster.name: "docker-cluster"

network.host: 0.0.0.0

discovery.zen.minimum_master_nodes: 1

discovery.type: single-node

logger.level: DEBUG

All the files and folder needed for defining elasticsearch container through docker compose is now ready. Proceed with creating docker compose file in the root of the project which is ~/docker-elk .

$ vi ~/docker-elk/docker-compose.yml

version: '3'

services:

elasticsearch:

container_name: elasticsearch

build:

context: elasticsearch

args:

ELK_VERSION: $ELK_VERSION

volumes:

- ./elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml

- ./elasticsearch/storage:/usr/share/elasticsearch/data:rw



ports:

- "9200:9200"

- "9300:9300"



environment:

- ELASTIC_PASSWORD="changeme"

- ES_JAVA_OPTS=-Xmx256m -Xms256m

- discovery.type=single-node

- bootstrap.memory_lock=true

- http.cors.allow-origin=*



ulimits:

memlock:

soft: -1

hard: -1



networks:

elk:

driver: bridge

Build the container:

$ cd ~/docker-elk

$ docker-compose build elasticsearch

Start the container in detached mode:

$ docker-compose up -d elasticsearch

List the container:

$ docker ps -a

Ping the ES container from your host machine:

$ curl http://127.0.0.1:9200/

{

"name" : "a85bd40e10de",

"cluster_name" : "docker-cluster",

"cluster_uuid" : "cnGc-4uLSIWS-bFwr8ywug",

"version" : {

"number" : "7.5.1",

"build_flavor" : "default",

"build_type" : "docker",

"build_hash" : "3ae9ac9a93c95bd0cdc054951cf95d88e1e18d96",

"build_date" : "2019-12-16T22:57:37.835892Z",

"build_snapshot" : false,

"lucene_version" : "8.3.0",

"minimum_wire_compatibility_version" : "6.8.0",

"minimum_index_compatibility_version" : "6.0.0-beta1"

},

"tagline" : "You Know, for Search"

}

To find any useful information, you can navigate to the shell of elasticsearch container:

$ docker exec -it elasticsearch /bin/bash

Logstash container

Like before, create following folders for logstash and its configuration settings in the root of the project.

config : The config folder will hold logstash system wide configuration settings.

: The folder will hold logstash system wide configuration settings. pipeline : The pipeline folder will hold logstash configuration settings for each log file that you want to process.

: The folder will hold logstash configuration settings for each log file that you want to process. logfile : The logfile folder will contain the log file (network logs, Apache logs and so on)

$ cd ~/docker-elk

$ mkdir -p logstash/{config,pipeline,logfile}

Create Dockerfile for Logstash.

$ vi ~/docker-elk/logstash/Dockerfile

ARG ELK_VERSION

FROM docker.elastic.co/logstash/logstash-oss:${ELK_VERSION}

RUN logstash-plugin install logstash-input-beats

USER root

RUN mkdir -p /home/logstash/logfile

RUN chown -R logstash:logstash /home/logstash/logfile/

Now create Logstash configuration file

$ vi ~/docker-elk/logstash/config/logstash.yml

http.host: "0.0.0.0"

path.config: /usr/share/logstash/pipeline

We are going to use Apache access and error log to test the Logstash pipeline configuration. So how to stream apache log events to logstash ? One way to stream apache logs in real time is by using filebeat.

Create a pipeline configuration for logstash that will accept apache log events in port number 5000 and after applying appropriate filters push them to elasticsearch container.

$ vi ~/docker-elk/logstash/pipeline/01.apache.conf input {

beats {

port => 5000

type => apache

}

} filter {

if [type] == "apache" {

grok {

match => { "message" => "%{COMBINEDAPACHELOG}" }

}

}

} output {

if [type] == "apache" {

elasticsearch {

hosts => ["http://elasticsearch:9200"]

index => "apache-combined-%{+YYYY.MM.dd}"

}

stdout { codec => rubydebug }

}

}

All the necessary configuration settings for logstash are in place to define services for it in the docker compose. Edit the docker compose file and add the following content in it.

$ vi ~/docker-elk/docker-compose.yml

...

...

logstash:

container_name: logstash

build:

context: logstash

args:

ELK_VERSION: $ELK_VERSION

volumes:

- ./logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml

- ./logstash/pipeline:/usr/share/logstash/pipeline ports:

- "5000:5000" networks:

- elk depends_on:

- elasticsearch

...

...

Build Logstash Container

$ docker-compose build logstash

Run the Logstash container without detached mode to view the logstash startup logs in the terminal.

$ docker-compose up logstash

If everything goes correctly, Press CTRL+C, and run the logstash container again in detached mode.

$ docker-compose up -d logstash

List the container

$ docker ps -a

Kibana container

To start with containerizing Kibana, create folders meant for it by the name config that will hold Kibana configurations.

$ cd ~/docker-elk

$ mkdir -p kibana/config

Create a Dockerfile to assemble image for latest kibana.

$ vi ~/docker-elk/kibana/Dockerfile

ARG ELK_VERSION

FROM docker.elastic.co/kibana/kibana:${ELK_VERSION}

Edit and create Kibana Configuration file.

$ vi ~/docker-elk/kibana/config/kibana.yml

server.name: kibana

server.host: "0"

server.basePath: "/kibana"

elasticsearch.hosts: http://elasticsearch:9200

apm_oss.enabled: true

xpack.apm.enabled: true

xpack.apm.ui.enabled: true

logging.dest: stdout

Finally append the Kibana service in the docker file.

$ vi docker-compose.yml

…

…

kibana:

container_name: kibana

build:

context: kibana/

args:

ELK_VERSION: $ELK_VERSION

volumes:

- ./kibana/config/:/usr/share/kibana/config

ports:

- "5601:5601"

environment:

- ELASTICSEARCH_PASSWORD="changeme"

networks:

- elk

depends_on:

- elasticsearch

Build and run kibana

$ docker-compose build kibana

$ docker-compose up -d kibana

List the Kibana container:

$ docker ps -a

NGINX container

The primary motivation for adding a NGINX container is to provide password-protected access to Kibana interface through reverse proxy.

Create folders for NGINX container in the root of the docker project and subsequently create two subfolders by the name public and data

$ cd ~/docker-elk

$ mkdir -p nginx/{public,data,etc}

Create a simple index file for NGINX

$ vi nginx/public/index.html

<html>

It Works

</html>

Create NGINX configuration file.

$ vi ~/docker-elk/nginx/etc/nginx.conf

worker_processes 4; events {

worker_connections 1024;

} http { server {

listen 80;

server_name IP_OR_DOMAIN ; location / {

root /usr/share/nginx/html;

index index.html;

} location /elastic/ {

proxy_pass http://elasticsearch:9200/;

proxy_set_header X-Real-IP $remote_addr;

proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

proxy_set_header Host $http_host; auth_basic "Restricted Content";

auth_basic_user_file /etc/nginx/.htpasswd.user;

} location /kibana/ {

proxy_pass http://kibana:5601/;

proxy_set_header X-Real-IP $remote_addr;

proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

proxy_set_header Host $http_host;

rewrite ^/kibana/(.*)$ /$1 break;

auth_basic "Restricted Content";

auth_basic_user_file /etc/nginx/.htpasswd.user;

}

}

}

For password protected access to kibana, you need to install httpd-tools and create the user/password using htpasswd.

$ yum install httpd-tools

$ cd ~/docker-elk/nginx/etc

$ htpasswd -c .htpasswd.user admin

Define docker service forNGINX

$ vi ~/docker-elk/docker-compose.yml

…

…

nginx:

image: nginx:alpine

container_name: nginx

volumes:

- './nginx/etc/nginx.conf:/etc/nginx/nginx.conf:ro'

- './nginx/public:/usr/share/nginx/html:ro'

- './nginx/etc/.htpasswd.user:/etc/nginx/.htpasswd.user:ro'

links:

- elasticsearch

- kibana

depends_on:

- elasticsearch

- kibana

ports:

- '80:80'

networks:

- elk

Run NGINX container

$ docker-compose up -d nginx

List the NGINX container:

$ docker ps -a

Once the NGINX container is up and running, access the kibana interface using http://SERVER_IP/kibana

Check the logs of any container using

$ docker logs container_name_or_id

e.g. docker logs elasticsearch

List all the running containers