AWS CloudWatch is a set of services offered by Amazon Web Services (AWS). AWS CloudWatch allows you to store and display metrics and logs as well as generate alarms to alert you when things go wrong. It obviously integrates very well with other AWS services such as EC2, and you can also use it for workloads located outside of AWS (on-premises servers, for example).

The AWS CloudWatch Agent can be used to export logs from a running server to AWS CloudWatch Logs. AWS CloudWatch can also receive logs from other AWS services.

For example:

VPC flow logs (when enabled).

Lambda output.

ECS (Elastic Container Service) can be configured to send logs generated by containers.

AWS CloudWatch Logs Insight is a tool offered by AWS to search, analyze, and visualize log data. It uses a custom query language to easily allow you to filter through the log data and extract the information you want. You can then analyze the results and display them in a graphical way.

How Does AWS CloudWatch Work?

A log message is simply a piece of text with a timestamp (additionally, CloudWatch Logs adds its own timestamp called “ingestion time,” which is probably useless for you in most cases). The text can be anything, but it is usually useful when it is formatted, such as in a Combined Log Format or JSON.

Logs are organized in log groups and log streams. A log group is akin to a directory on a filesystem, however it’s only to a depth of one. That is, a log group cannot contain another log group. A log stream is similar to a file, where the log data will be stored.

The AWS CloudWatch Logs console offers a basic way to retrieve and inspect logs. The screenshot below shows what this console looks like. You can access the console by selecting the “CloudWatch” service and clicking the log group and then the log stream that you want to inspect:

In the top-right corner, you can find a box where you can filter the logs based on a date and range of time. The “Filter events” box allows you to further filter logs that contain the text you entered. This view is quite limited and useful only for a quick glance at simple logs.

A very useful feature offered by AWS CloudWatch Logs is forwarding log data to other AWS services (essentially Lambda and Kinesis) and even to external systems such as Splunk or an ELK cluster.

How to Feed Log Data?

In AWS, your code can run in different ways. Let’s explore the various AWS services for this and how to get your logs ingested into AWS CloudWatch.

Collect Logs from Your Code

The first and most obvious way that your code is run is on an EC2 instance. In that case, you will need to install and configure the CloudWatch Agent to monitor local log files and to send the logs to AWS CloudWatch Logs. This is obviously not limited to your code, and you can also monitor log files from, say, a web server or even the syslog if you’re running Linux. Don’t forget to attach a role to your EC2 instance that allows you to send the logs to CloudWatch Logs. You can also use the CloudWatch Agent to send logs to CloudWatch Logs from an on-premises server (for this you will need to use an IAM user account with the appropriate permissions, as it’s not possible to attach a role to anything but an EC2 instance).

The next case is to run your code on Docker containers, whether using ECS (Elastic Container Service) or EKS (Elastic Kubernetes Service). In both cases, you can redirect logs from your containers to AWS CloudWatch Logs. In the case of Kubernetes, you can also configure the control plane to send logs to CloudWatch Logs.

Finally, you can run your code using Lambda functions, in which case the logs are automatically sent to AWS CloudWatch Logs. In fact, Lambda doesn’t offer you any choice on this, which is a bit regrettable. But you can still stream the logs to another service for custom processing if necessary.

Collect Logs from Other AWS Services

Many AWS services either send their logs to CloudWatch Logs by default or allow you to do so through some configuration. CloudTrail (a service provided by AWS to monitor all API calls made on your account) can be configured to send certain events to AWS CloudWatch Logs. This can then be expanded to generate alarms from AWS CloudWatch Logs based on those CloudTrail events.

Another example is VPC flow logs. VPC (Virtual Private Cloud) allows you to build a virtual networking environment for your apps and databases and replicates a real-world networking setup with subnets, firewalls, and routers. Getting a VPC working correctly can sometimes be tricky, and that’s where enabling flow logs can help. The VPC will then stream to AWS CloudWatch Logs information about the networking traffic occurring inside the VPC, along with information about whether the traffic has been allowed or blocked.

Enabling flow logs can generate a lot of data depending on how much of it you enable and the volume of your VPC networking traffic. So you should turn off VPC flow logs as soon as the problem has been solved. Or, if you use flow logs for auditing or analytics purposes, make sure AWS CloudWatch forwards this data somewhere else and doesn’t retain it for too long.

A final example is RDS (Relational Database Service), which is the database-as-a-service offering from AWS. Database engines can be configured to send their logs to AWS CloudWatch Logs; here is an example for MySQL. In the same vein, there are many AWS services that either send their logs directly to AWS CloudWatch Logs or can be configured to do so.

The Generic Way to Feed AWS CloudWatch Logs

It is entirely possible for you to use a custom or in-house solution to ingest log data into AWS CloudWatch. You only need to use the PutLogEvents API call, either directly (not recommended as you’ll have to set up authentication and other AWS headers yourself), through an AWS SDK, or through the command line if you just have a few logs to ingest infrequently.

AWS CloudWatch Logs Insight

AWS CloudWatch Logs Insight allows you to search through your log data, analyze it, and visualize it. In order to filter the data, CloudWatch Logs Insight uses a custom query language, which is fairly intuitive (especially for those familiar with SQL) since AWS CloudWatch Logs Insight automatically detects fields in formatted logs and provides auto-completion features.

To go to AWS CloudWatch Logs Insight, log in to the AWS console, select the CloudWatch service, and click on the “Insights” link in the left pane. Please note that AWS CloudWatch Logs Insight is not provided in all AWS regions, so if you can’t see the link, it’s probably because you’re in such an area.

Field Detection

AWS CloudWatch Logs Insight recognizes certain log formats generated by other AWS services, as well as all JSON-formatted logs, and will make available the fields in its query language. In any case, you have three fields that are always available to you:

@message contains the log text.

@timestamp contains the log timestamp.

@logStream contains the log stream to which the log entry belongs.

For other types of logs (e.g., the Combined Log Format, which is very common for web servers), you will need to make use of the parse command.

A Short Description of the Query Language

The AWS CloudWatch Logs Insight query language is essentially a string of commands linked together through pipes. The output of one command is “piped” as the input of the next command through the use of the pipe symbol: |. Up to six commands can be stringed in this way.

For example, the toy query that AWS CloudWatch Logs Insight shows you by default is:

fields @timestamp, @message

| sort @timestamp desc

| limit 20

This selects the @timestamp and @message fields from log messages, sorts them with the most recent on top, and displays only the first 20 results.

Before being able to run a query, you’ll first need to select one or more log groups in the top drop-down menu. To the right of that drop-down menu, you will find a box for narrowing down your filtering by date and time (the default option might not suit you, so make sure that you enter a date/time range adapted to your query).

On the top-right corner of the screen, you can also find very helpful reminders of what commands are available and their syntax. Below this, you’ll find the fields discovered by AWS CloudWatch Logs Insight (whether through auto-discovery or the parse command), which can thus be used in the commands.

How to Visualize Log Data

AWS CloudWatch Logs Insight allows you to visualize log data, provided that:

The query contains one or more aggregation functions .

The query uses the bin() function to group data into bins.

The following screenshot shows what such a graph looks like:

Conclusion

AWS CloudWatch is a very capable and scalable solution to ingest, store, and process log data. And it integrates very well with other AWS services. For very large and complex projects, third-party solutions such as Epsagon might be more appropriate. But AWS CloudWatch Logs should be the first port of call for small and medium projects. Also, its ability to stream data to Lambda and Kinesis allows literally limitless possibilities in terms of custom processing.

You might be also interested in:

5 Ways to Understand Distributed System Logging and Monitoring

Instrumentation for Better Monitoring and Troubleshooting

Why You Can’t Ignore Changes to Monitoring and Logging for Serverless

Distributed Tracing: the Right Framework and Getting Started