With my first post, I thought I would post about something that I didn’t have much luck in finding an existing solution. I wanted to get emails from my running Python program when there were errors. The logging library offers the SMTPHandler as a simple solution (assuming you have an SMTP server to send the message to). But what if you are running a very early beta with lots of errors or you pushed a new change that’s breaking?

Your inbox will get flooded.

If your program is single threaded, it is simple to create an expanded SMTP handler that will limit repeated emails:

import datetime import logging.handlers class SMTPPlusHandler(logging.handlers.SMTPHandler): def __init__(self, mailhost, fromaddr, toaddrs, subject, credentials=None, secure=None, timeout=5.0): super(SMTPPlusHandler, self).__init__(mailhost, fromaddr, toaddrs, subject, credentials, secure, timeout) self.state = {} def uniq_key(self, record): key = record.levelname + record.pathname if record.exc_info: key += str(record.exc_info[0]) return key def block_ts(self, count): throttle_secs = 60 * count ^ 3 return datetime.datetime.now().timestamp() + throttle_secs def update_status(self, key, status): status['count'] += 1 status['block_ts'] = self.block_ts(status['count']) def is_blocked(self, status): return datetime.datetime.now().timestamp() < status['block_ts'] def is_emittable(self, record): key = self.uniq_key(record) status = self.state.get(key, {'count': 0, 'block_ts': 0}) if self.is_blocked(status): return False else: self.update_status(key, status) self.state[key] = status return True def emit(self, record): if self.is_emittable(record): super(SMTPPlusHandler, self).emit(record)

Let’s break this down:

class SMTPPlusHandler(logging.handlers.SMTPHandler): def __init__(self, mailhost, fromaddr, toaddrs, subject, credentials=None, secure=None, timeout=5.0): super(SMTPPlusHandler, self).__init__(mailhost, fromaddr, toaddrs, subject, credentials, secure, timeout) self.state = {}

We create a derived class of the logging library’s SMTPHandler and initialize it via super(). state will be used to track emailed log records.

def emit(self, record): if self.is_emittable(record): super(SMTPPlusHandler, self).emit(record)

Working from the top down, we make a wrapper around the base class emit() function. The base emit() is what takes the record and sends an email. Here we add in a check to see if the record is emittable before calling the base emit().

def is_emittable(self, record): key = self.uniq_key(record) status = self.state.get(key, {'count': 0, 'block_ts': 0}) if self.is_blocked(status): return False else: self.update_status(key, status) self.state[key] = status return True

is_emittable() will take care of tracking the state and checking if an email should be sent. First, we create a key that will be used to store this record type in our state. Next, we get what I call status. status is a dictionary that contains a count of the number of emails sent and a timestamp, in seconds, for what we will block until. If this is the first time this record type has been emitted, status will default with zero values.

def uniq_key(self, record): key = record.levelname + record.pathname if record.exc_info: key += str(record.exc_info[0]) return key

uniq_key() takes the log record and makes a key out of it. This is what we’ll use to determine if something is a duplicate record. This is subjective and what I’ve done here is just an example. You can find the different fields here for LogRecord.

def is_blocked(self, status): return datetime.datetime.now().timestamp() < status['block_ts']

is_blocked() compares the current timestamp to the blocking timestamp. Note, datetime.timestamp() is only available from Python 3.3 and up.

def update_status(self, key, status): status['count'] += 1 status['block_ts'] = self.block_ts(status['count'])

If the record is not blocked and we send an email, update_status() increments the count and sets a new blocking timestamp.

def block_ts(self, count): throttle_secs = 60 * count ^ 3 return datetime.datetime.now().timestamp() + throttle_secs

block_ts() takes the current timestamp and adds seconds to it depending on the count. I chose 60count^3 because it does a good job of spacing out the intervals. After the first email, it will block for 1 minute, second it will block for 8 minutes, third for 27 minutes.

Now that our handler is complete we can add it to our logging config:

[handlers] keys=smtpPlusHandler [handler_smtpPlusHandler] class=my.python.path.smtpPlusHandler.SMTPPlusHandler level=WARN formatter=simpleFormatter args=('mailhost', 'ALERT@mydomain.com', ['devops@mydomain.com', 'bossman@mydomain.com'], 'EMAIL ALERT PROGRAM1')

Note, you have to change the class path to your path. You can set the from address, who will receive the email as a list, and the subject of the email.

From there, any time you log something with the level you set for the smtp handler, an email will be sent. I will discuss how this can be modified for multiprocessed programs in a subsequent post. Feel free to leave ask questions or leave feedback.