Nginx is an extremely popular open-source web server serving millions of applications around the world. Second only to Apache, Nginx’s owes its popularity as a web server (it can also serve as a reverse proxy, HTTP cache and load balancer) to the way it efficiently serves static content and overall performance.

From an operational and security perspective, Nginx sits at a critical juncture within an application’s architecture and requires close monitoring at all times. The ELK Stack (Elasticsearch, Logstash, Kibana and Beats) is the world’s most popular open-source log management and log analysis platform, and offers engineers with an extremely easy and effective way of monitoring Nginx.

In this article, we’ll provide the steps for setting up a pipeline for Nginx logs and beginning the monitoring work. To complete the steps here, you’ll need a running Apache web server and your own ELK Stack or Logz.io account.

Nginx logging basics

Nginx provides users with various logging options, including logging to file, conditional logging and syslog logging. Nginx will generate two log types that can be used for operational monitoring and troubleshooting: error logs and access logs. Both logs are typically located, by default, under /var/log/nginx but this location might differ from system to system.

Nginx error logs

Error logs contain diagnostic information that can be used for troubleshooting operational issues. The Nginx error_log directive can be used to specify the log file path and severity and can be used in the main, http, mail, stream, server, location context (in that order).

Example log:

2019/07/30 06:41:46 [emerg ] 12233#12233: directive “http” has no opening “{” in /etc/nginx/nginx.conf:17

This example emerg (the most severe logging level) error log is informing us that a directive is misconfigured.

Nginx access logs

Access logs contain information on all the requests being sent to, and served by, Nginx. As such, they are a valuable resource to use for performance monitoring but also security. The default format for Nginx access logs is the combined format but this may change from distribution to distribution. As with error logs, you can use the access_log directive to set the log file path and log format.

Example log:

199.203.204.57 - - [30/Jul/2019:06:35:54 +0000] "GET /hello.html HTTP/1.1" 200 63 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.142 Safari/537.36"

Shipping to ELK

The simplest way of shipping Nginx logs into the ELK Stack (or Logz.io) is with Filebeat.

In previous versions of the ELK Stack, Logstash played a critical part in Nginx logging pipelines — processing the logs and geo enhancing them. With the advent of Filebeat modules, this can be done without Logstash, making setting up an Nginx logging pipeline much simpler. The same goes if you’re shipping to Logz.io — parsing is handled automatically. More about this later.

Installing Filebeat

First, add Elastic’s signing key so that the downloaded package can be verified (skip this step if you’ve already installed packages from Elastic):

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

Next, add the repository definition to your system:

echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list

Update and install Filebeat with:

sudo apt-get update && sudo apt-get install filebeat

Enabling the Nginx Module

Our next step is to enable the Filebeat’s Nginx module. To do this, first enter:

sudo filebeat modules enable nginx

Next, use the following setup command to load a recommended index template and deploy sample dashboards for visualizing the data in Kibana:

sudo filebeat setup -e

And last but not least, start Filebeat with:

sudo service filebeat start

It’s time to verify our pipeline is working as expected. First, cURL Elasticsearch to verify a “filebeat-*” index has indeed been created:

curl -X GET "localhost:9200/_cat/indices?v"

health status index uuid pri rep docs.count docs.deleted store.size pri.store.size green open .kibana_1 RjVOETuqTHOMTQZ8GiSsEA 1 0 705 153 900.1kb 900.1kb green open .kibana_task_manager L78aE69YQQeZNLgu9q_7eA 1 0 2 0 30.4kb 30.4kb yellow open filebeat-7.2.0 xVZdngF6TX-EiRm2e-HuCQ 1 1 5 0 92.9kb 92.9kb

Next, open Kibana at: http://localhsot:5601 — the index will be defined and loaded automatically and the data visible on the Discover page:

Shipping to Logz.io

As mentioned above, since Logz.io automatically parses Nginx logs, there’s no need to use Logstash or Filebeat’s Nginx module. All we have to do is make some minor tweaks to the Filebeat configuration file.

Downloading the SSL certificate

For secure shipping to Logz.io, we’ll start with downloading the public SSL certificate:

wget https://raw.githubusercontent.com/logzio/public-certificates/master/COMODORSADomainValidationSecureServerCA.crt && sudo mkdir -p /etc/pki/tls/certs && sudo mv COMODORSADomainValidationSecureServerCA.crt /etc/pki/tls/certs/

Editing Filebeat

Next, let’s open the Filebeat configuration file:

sudo vim /etc/filebeat/filebeat.yml

Paste the following configuration:

filebeat.inputs: - type: log paths: - /var/log/nginx/access.log fields: logzio_codec: plain token: type: nginx_access fields_under_root: true encoding: utf-8 ignore_older: 3h - type: log paths: - /var/log/nginx/error.log fields: logzio_codec: plain token: type: nginx_error fields_under_root: true encoding: utf-8 ignore_older: 3h #For version 6.x and lower uncomment the line below and remove the line after it #filebeat.registry_file: /var/lib/filebeat/registry filebeat.registry.path: /var/lib/filebeat #The following processors are to ensure compatibility with version 7 processors: - rename: fields: - from: 'agent' to: 'beat_agent' ignore_missing: true - rename: fields: - from: 'log.file.path' to: 'source' ignore_missing: true output.logstash: hosts: ['listener.logz.io:5015'] ssl: certificate_authorities: ['/etc/pki/tls/certs/COMODORSADomainValidationSecureServerCA.crt']

A few things about this configuration are worth pointing out:

The configuration defines two file inputs, one for the Nginx access log and the other for the error log. If you need to change the path to these files, do so now.

Be sure to enter your Logz.io account token in the relevant placeholders. You can find this token in the Logz.io UI .

The processors defined here are used to comply with the new ECS (Elastic Common Scheme) and are required for consistent and easier analysis/visualization across different data sources.

The output section defines the Logz.io listener as the destination for the logs. Be sure to comment out the Elasticsearch destination.

Save the file and restart Filebeat with:

sudo service filebeat restart

Within a minute or two, you will begin to see your Nginx logs in Logz.io:

Analyzing Nginx logs

Kibana is a pretty powerful analysis tool that provides users with rich querying options to slice and dice data. The auto-suggest and auto-complete features added in recent versions turn the experience of sifting through the logs much simpler and easier.

Let’s take a look at some examples.

The simplest search method, of course, is free text. Just enter a search query in the search field as follows:

sydney

Field-level searches allow us to be a bit more specific. For example, we can search for any Nginx access log with an error code using this search query:

type : "nginx_access" and response > 400

There are plenty of other querying options to choose from. You can search for specific fields, use logical statements, or perform proximity searches — Kibana’s search options are extremely varied and are covered more extensively in this Kibana tutorial.

Visualizing Nginx logs

Things get more interesting when we start to visualize Nginx logs in Kibana. Kibana is infamous for its beautiful dashboards and visualizations that help users depict their data in many different ways. I’ll provide four simple examples of how one can visualize Nginx logs using different Kibana visualizations.

Request map

For Nginx access logs, and any other type of logs recording traffic for that matter, the usual place to start is a geographic map of the different locations submitting requests. This helps us monitor regular behavior and identify suspicious traffic. Logz.io will automatically geo enrich the IP fields within the Nginx access logs so you can use a Kibana Coordinate Map visualization to map the requests as shown below:

If you’re using your own ELK Stack and shipped the logs using Filebeat’s Nginx module, the fields will also be geo enriched.

Average bytes

Monitoring the average number of bytes being sent by nginx is a great way to identify when something in your environment is not performing as expected. There are two extremes that may indicate something is wrong — when the number bytes is either drastically lower or drastically higher than average. In both cases, you will first need to be acquainted with the average value during regular circumstances.

The configuration for the line chart visualization includes a Y axis displaying an average value for the bytes field and an X axis using the Date Histogram aggregation type:

Top Requests

Which URLs are visited the most times is probably one of the most important metrics to monitor as it gives you an idea of which request to nginx occurs more often than not. When something goes wrong, this can be a good place to start because it might indicate which service crashed your entire environment.

The configuration for this bar chart visualization includes a Y count axis and an X axis using the Terms aggregation type for the request field (the example is configured to show the top 20 results):

Data table visualizations are a great way of breaking up your logs into ordered lists, sorted in the way you want them to be using aggregations. In the example here, we’re taking a look at the same list of top requests as above:

Responses over time

Another common visualization used for Nginx access logs monitors response codes over time. Again, this give us you a good picture of normal behavior and can help us detect a sudden spike in error response codes. You can use Bar Chart, Line Chart or Area Chart visualizations for this:

Notice the use of the Count aggregation for the Y-Axis, and the use of a Date Histogram aggregation and Terms sub aggregation got the X-Axis.

Errors over time

Remember — we’re also shipping Nginx error logs. We can use another Bar Chart visualization to give us a simple indication for the number of errors reported by our web server:

Note, I’m using a search filter for type:nginx_error to make sure the visualization is showing only depicting the number of Nginx errors.

Traffic by OS/Country/Browser

Last but not least, I’ve grouped together these four visualizations since their configurations are almost identical. They all provide the top twenty values for metrics on the client sending out the request to the server — the top twenty operating systems, countries, and browsers.

The configuration for these pie chart visualizations consists of a metric count aggregation and a bucket configured to aggregate the geoip.country_name field (that is set to display the top 20 results).

To build the other visualizations, we simply switched the field to: name (for the top 20 browsers) and os (for the top 20 operating systems):

These were just some examples of what can be done with Kibana but the sky’s the limit. Once you have your visualizations lined up, add them up into one comprehensive dashboard that provides you with a nice operational overview of your web server.

Endnotes

Analyzing your Nginx access logs provides you with not only good visibility into the health and activity patterns of your environment but also the operational and business intelligence needed to improve and enhance your product. Adding error logs into the picture (by easily shipping the error log into ELK as well) does run the risk of adding another layer to your monitoring but also enables you the option of identifying correlations between the two logs and thus also enables faster troubleshooting.

Logz.io users can install the dashboard above, and many other Nginx visualizations and dashboards, using ELK Apps — a free library of pre-made dashboards for various log types, including Nginx of course. If you don’t want to build your own dashboard from scratch, simply search for “nginx” in ELK Apps and install whichever dashboard you fancy.

To stay on top of errors and other performance-related issues, a more proactive approach requires alerting, a functionality which is not available in vanilla ELK deployments. Logz.io provides a powerful alerting mechanism that will enable you to stay on top of live events, as they take place in real-time. Learn more about this here.