#1 Be aware of the levels

Probably you already know, but there are different levels that you should choose, for each information you need to log.

Does this make sense?

Yes, because all the information that you want to log has a different meaning.

Of course, there are different explanations to each level definition, but one of the most consensuses is the RFC-5424, which proposes these levels and meanings that are used in Syslog.

0 Emergency: system is unusable

1 Alert: action must be taken immediately

2 Critical: critical conditions

3 Error: error conditions

4 Warning: warning conditions

5 Notice: normal but significant condition

6 Informational: informational messages

7 Debug: debug-level messages

This can be one of the approaches to follow.

Now, we should choose wisely what level we want to address for each message to log.

This way, it would be easier for you to prioritize what information you will need to address.

If you don’t, it would be almost impossible, and you will be flooded with information you can not handle.

#2 What is your default log level?

Photo by Jonas Zürcher on Unsplash

You should define one of the mentioned on tip #1, as the default one. And why?

Because this default level defines what should be shown, and not shown, outside the development environment.

Typically is chosen the info level.

This means that only logs with level info and above are going to be saved, and this defines the lowest level of interesting messages to be handled.

And why?

Because the debug level (and sometimes trace) only make sense on development.

One last tip here is that you should make this default level, a configuration that’s dependent on the chosen environment.

This way, we can save different log levels for different environments (staging, production, etc).

#3 For each level address channels

Photo by Volodymyr Hryshchenko on Unsplash

One idea that could be useful is, for each log level, to define specific communication channels, to address certain message levels to groups of employees.

This could make more sense in large enterprises, that have a lot of structure and teams dedicated to infrastructure, and others to the development, with different levels of seniority.

But it could be useful also to small companies, having this way, for example, slack channels addressed to each type of priority that it needed to be addressed.

Even also, types of messages transportation, like being sent by SMS if they are a very urgent topic.

#4 Log retention policy

Photo by Vincent Botta on Unsplash

What is log retention? Is what logs you save, and for how long.

And you should have a log retention policy in place who will define the details.

This policy will identify what type of information is stored, for how long, if is going to be online or offline, and whether, if the data is confidential or not.

One possible policy could be to delete:

Local logs older than 15 to 30 days.

Remote centralized logs older than 6 to 12 months.

But, if you want to retain logs beyond a maximum retention period, you can archive them.

Another strategy could be define first how much disk resources you want to dedicate to storing log information.

Once you have this number you accommodate the logs to the space you have.

#5 Logging personal information or secrets

Photo by Kristina Flour on Unsplash

This is very important.

We shouldn’t log anything personal, or secrets, or tokens. And why?

Because they are being saved in files, and maybe being transferred to external services.

Of course, these services do their best to protect our information, but it’s best to be safe and compliant.

But, if you need to, you can have scripts in place that first anonymize the data that is inside of your logs.

You can use regular expressions to parse your logs, replacing sensitive information.

In this case, it can be useful to use a package that fakes data, like Faker.js.

With this you can replace all the data with a fake username, or email, preserving all the logs formats.

Another option is to use the package Data anonymizer, which allows the use of seed when sending logs for external services.

See below.

const DataAnonymizer = require('data-anonymizer');

const a = new DataAnonymizer({ seed: 'my secret seed' }); // Create a logger instance with Winston.

const winston = require('winston');

const logger = winston.createLogger({

level: 'info',

format:winston.format.json(),

defaultMeta: { service: 'user-service' },

transports: [

// - Write all logs with level `error` and below to `error.log`

// - Write all logs with level `info` and below to `combined.log`

new winston.transports.File({ filename:'error.log', level:'error'}),

new winston.transports.File({ filename:'combined.log' })

]

}); // Log anonymized date.

logger.info('This is an info log! First name:'+a.anonymize('Luis')); // It will output:

// info: This is an info log! First name: Vwbh {"service":"user-service"}

#6 Use the appropriate tools

Photo by Todd Quackenbush on Unsplash

Depending on your language you should choose the best library, the most tested, used, and useful for you to use on your application.

Some good examples by language:

After that, how should we see our logs? It is time to choose the best tool for the job.

Because nobody can’t come to a conclusion only on looking to several running lines appearing on the terminal.

We need a tool to help us see what is most important of what we log. What are the patterns? What logs by level we have?

A tool to save and analyze the logs.

Here we can choose for the several add-ons that exist in Heroku, specifically for logging.

#7 Same log format in every environment

Photo by 浮萍 闪电 on Unsplash

This can be useful because if we have the same log format, in development, and the environments hosted by Heroku, we can get oust to the format of logs of Heroku.

And how we can do this?

I will be giving the following example, of how we can achieve this, with Node.js, using the Heroku-log package.

In the first step, just install the package.

# npm

npm install heroku-log --save # yarn

yarn add heroku-log

After this, we can import it, and use it in your application.

import herokuLog, { error } from 'heroku-log' // # On Heroku

herokuLog.info('Server is starting')

// 2017-05-24T08:58:42.857401+00:00 app[web.1] level=info message="Server is starting" error(new Error('I am an error '))

// 2017-05-24T08:58:42.857401+00:00 app[web.1] level=error error="I am an error"

Notice that we can send also the native Error object as an argument. This will show, instead of message, error.

Heroku Logging Overview

Photo by jesse orrico on Unsplash

To see the logs of an application hosted in Heroku is simple.

This is because Heroku handles logs as streams, and aggregates all application, system, and API log messages that are being sent to STDOUT or STDERR.

And what kind of logs typically an application generates?

Runtime Logs

App Logs

System Logs

API Logs

Add-on Logs

Build Logs

All of these logs will go into a single place, and like you can see below, all the logging that happens is centralized through a channel, and then sent to log drains.

Next, how can we see what is happening in our application? How can we see what is being outputted to STDOUT or STDERR?

We can see the logs that are happening with the following command. Notice that this command shows the last 100 lines of logs, by default.

heroku logs

If we need more lines, we need to use the -n argument, defining the number of lines we want, (to a maximum of 1500) like the example below:

heroku logs -n 1500

We can also live tail the logs to see what’s happening in real-time:

heroku logs -t

Finally, we can use filters, by dyno and/or source:

heroku logs --source app --dyno api

You can find a more extensive description of this, on Heroku documentation.

Logplex, sources, and drains

Photo by George Pagan III on Unsplash

Let’s start with Logplex.

Logplex is a router, responsible for joining and distributing all log lines that your application generates, on Heroku.

Logplex can not read logs from file-based application logs, it only reads from STDOUT and STDERR.

These logs are then available through the Logplex public API and Heroku command-line tool.

This way Logplex, makes it easy to collect logs from our applications and forward them to where we want to use them.

Logplex is available as an open-source project, and basically, routes messages from sources to drains.

Logplex is necessary for an add-on logging solution to work. Logplex will feed that add-on, that is a drain.

So, the drains are any network services that will consume the app’s logs, and sources, any processes that might create relevant log entries.

Heroku and file-based logging

And if we want to write all our logs to a file-based solution?

For this approach, ideally, we should write them to a remote server.

This way, we are not going to face any low space issues, and any problems with your application server will have no impact in your logs.

These logs could also be helpful for you to know after, what happened.

But this kind of approach, of saving remote logs at each request will have an impact on the performance of your application.

To accommodate this, we can have several options, one, implementing a buffer, or other, writing to a local file.

For the latest, you can regularly transfer that local file to a remote server.

A big detail here is that Heroku has an ephemeral file system, and cycle their dynos, so we need to catch the SIGTERM and upload everything before the SIGKILL.

Also, think carefully if you want to go down this road. Because this brings several challenges, like the archiving and analyzing of all your logs.

Heroku and Syslog or Database drains

Photo by Jordan Harrison on Unsplash

Do you want to store your logs locally?

Remember, Heroku only saves your latest 1500 logs, so if you need older logs, you will need to have other solutions.

And, of course, if you’d like to have your own solution, one possibility is to have a Syslog Drain.

Syslog, how?

You can configure a Syslog instance to receive packages from Heroku, defining his URL, a domain, and port, as Syslog drain.

This way we redirect all the Heroku logs to a Syslog server, that will work as a permanent archive.

Of course, if it is sufficient for you to see all the logs in real-time, then the tail is more than enough.

In a PostgreSQL database

This package heroku-log-store allows you to have a service in Ruby that receives, via HTTPS, and save, messages, in a PostgreSQL database.

You can push this package to Heroku dyno, and use it as your own log data store.

After that, you can query the logs that were saved in the database.

Troubleshooting with Heroku logs

Photo by Markus Spiske on Unsplash

As applications are becoming more complex, and scaled across distributed infrastructure, the need to track and know what is happening, is becoming more difficult.

Numerous components and services each produce their own log streams.

All logs must be centralized, and have useful information for humans to use them.

Which Application?

And how to see from where these logs come? You can use the source flag.

For example, if you see “Application error”, then it should be caused by our own application code.

To see this kind of issues, execute this command:

heroku logs --tail --app your_app_name

Notice, if you, for example, don’t provide your app name, you will have an error “Realtime tail failed: Missing required flag”. This is telling you, that is your tail that failed, not your application.

Add-ons

You can also choose any add-on, related to logging.

With them, you will have more options when it comes to filtering, configure alerts, and, for example, have quality levels perversion.

These normally will store all the logs you have and will allow you to have more insight into your application.

Without add-ons

Of course, if you want, you can choose any of the solutions I mentioned above, having all the logs in files, in a PostgreSQL database, or in a Syslog server.

This can work also, but this way, you will need to have all the work by yourself.

With Splunk

You can also use Splunk Queues. Splunk is an application to manage data, that can receive Heroku logs in his queues.

It can be configured as a log drain, and with this SPL Recipe, having dashboards with locations of the users, loading times, and much more data.

What do you think?

Photo by Mitchell Griest on Unsplash

Logging is a serious matter.

We should make sure that we are logging what is important, and send it to appropriate channels, depending on its importance.

Logging should also be a priority of your project. Small, medium, or big. It should push you to go a step further when it comes to your application insight.

In this article, I shared with you some best practices related to logging in general, and how, some of them can be applied to a Heroku application.

But I also want to know your opinion. Share it with a comment.