tl;dr - It’s pretty easy to use Graylog as a System.Logger backend, check out the code at the end, also if you’re interested in just regular crash-level logging with Servant, there’s some code you might like at the bottom too.

On a recent contract I was introduced to Graylog – it’s a pretty awesome log aggregation tool, with a great rontend and I was drawn to the simplicity of use. I encountered some difficulty using it at work, but I found that all the answers made sense after I found the solutions, which is to say that the tool was very internally self-consistent. I value that self-consistent kind of quality in tools I use very much. in my tools for example kubernetes is very much the same way, it’s concepts fit together so well that when something goes wrong, you can often think in terms of kubernetes concepts and arrive at the answer long before you start digging.

It’s also kind of difficult to find a tool smaller than Graylog, since it seems just about everyone else uses the ELK stack otherwise (ElasticSearch + Logstash + Kibana). Graylog does require two data services, ElasticSearch and Mongo – you can find more information on running it locally at their docker hub page. As far as the L and K in ELK, they’re both contained in Graylog, and gray log handles whatever is necessary to view, search (with the help of ES), and house the logs (with the help of mongo).

After getting Graylog set up on the server, it came time to figure out how to integrate it with my haskell app. Luckily for me,Andrew Rademacher has written an awesome library for haskell already haskell-graylog, which contains a very easy-to-setup connector for Graylog’s logs-over-UDP functionality. While lots of things around Graylog have changed, the library still works great, so that saved me some work.

My app uses hslogger, which produces the System.Log.Logger module amongst other things. I’m pretty happy with this library, as it’s pretty robust, and I started looking to see how easy it would be to add a logger to my setup that would log to graylog. Turns out it’s rpetty simple.

The Process

The process broke down into roughly these steps:

Create a data structure that will hold a possibly-connected Graylog UDP connection Add a typeclass that allows the data structure to send messages over the Graylog UDP connection, if present Define a LogHandler (as defined by the hslogger module) to enable that LogHandler to be inserted smoothly Do the setup for Graylog UDP, build the log handler, and insert it into your app for use

Note #4 is pretty application-specific so I’m not including it here.

The Code

The code to do steps 1-3 is pretty succint:

import Graylog.UDP import System.Log.Logger (Logger, Priority(..)) import System.Log.Formatter (LogFormatter, simpleLogFormatter) import System.Log.Handler (setFormatter) import Control.Exception (handle) -- A few imports might be left out here, so you might have to add some more -- Here's the type that will hold the graylog connection and some other stuff data GraylogHandler = GraylogHandler { glhMaybeConn :: Maybe Graylog , glhLevel :: Priority , glhFormatter :: LogFormatter GraylogHandler , parentLogger :: Logger } -- Note that making the graylog handler can fail, so I've had it require a logger from somewhere else -- so that in case things go bad, we can at least log about it makeGraylogHandler :: Logger -> Priority -> LogFormatter GraylogHandler -> String -> String -> IO GraylogHandler makeGraylogHandler logger p f ip port = Control.Exception.handle handleGraylogFailure getAndReturnGraylog where h = GraylogHandler Nothing p f logger logMsg = logL logger INFO handleGraylogFailure = (\_ -> logMsg "Failed to connect to graylog" >> return h) :: SomeException -> IO GraylogHandler getAndReturnGraylog = openGraylog ip port defaultChunkSize >>= \g -> logMsg ("Successfully connected to Graylog @ [" ++ ip ++ "], port [" ++ port ++ "]") >> either (\s -> putStrLn s >> return h) (\conn -> return h { glhMaybeConn=Just conn }) g -- Instance that makes the GraylogHandler fit into System.Log.Logger's worldview instance LogHandler GraylogHandler where setLevel glh l = glh { glhLevel=l } getLevel = glhLevel setFormatter glh f = glh { glhFormatter=f } getFormatter = glhFormatter emit glh record _ = case (glhMaybeConn glh) of Nothing -> logL (parentLogger glh) INFO $ "SENT TO GRAYLOG: " ++ (show record) Just g -> sendLog g

Here’s what the code looks like that attaches this log handler, near the rest of the app setup code.

Whew, that was easy!

Just kidding – wouldn’t you know not long after I got this working, I actually ended up dropping Graylog, mostly because it takes a larger amount of RAM to run comfortably than I was OK using up. A combination of docker compose (which I was using to start it) leaving child processes hanging (when managed with systemd), and some other issues made the startup and running of Graylog kind of inconsistent and error prone. Note that this wasn’t a problem with Graylog itself per-say, but more my fault, as the person operating the servers and determining the deployment procedure.

After starting a very small VPS (this was back when I was using VPSes, that’s changed a bit lately), Graylog itself was thrashing and having all sorts of errors because things like elastic search didn’t have enough space to start up comfortably. After getting a look at what I could find regarding the base amount of memory I realized that I just didn’t have enough (and didn’t want to upgrade more to get enough). luckily future (now) me is introduced to the wonders that are dedicated hosting, and the robot auction. I bought and use a machine from there and have found it fantastic – I’ve got tons of space still left on it so I might actually look into running Graylog in the future.

Some might ask “But what happened to uhhh, logging the messages?” The answer to that, is that I actually took this format, wrote a very similar EmailLogHandler , that now sends all the errors WARNING and above straight to my email. While not idea, I find it to be a pretty decent solution, I get some more visibility into app failures.

The Email Code

Here’s the email code (actually what I’m using at present):

-- Going to skip the import statements this time, you likely shouldn't be copying this code as it's pretty specific to my usecase/how I chose to set my app up. -- Chunks of it are probably reusable though so knock yourself out -- ... near some utility functions at the top of the file ... defaultLoggerFormat :: LogFormatter a defaultLoggerFormat = simpleLogFormatter "[$time : $loggername : $prio] $msg" removeRootLogger :: IO () removeRootLogger = updateGlobalLogger rootLoggerName removeHandler logLevelsWithHandles :: [(Priority, Handle)] logLevelsWithHandles = [(DEBUG, stdout)] -- This function just generates a bunch of default log handlers that have the default logger format) based on the `logLevelsWithHandles` value defaultLogHandlers :: IO [GenericHandler Handle] defaultLogHandlers = mapM (fmap (`setFormatter`defaultLoggerFormat) . makeHandler) logLevelsWithHandles where makeHandler (l, h) = fmap (\handler -> handler { formatter=defaultLoggerFormat }) (streamHandler h l) -- This function builds a logger given a logger name and priority to operate at buildLogger :: String -> Priority -> IO Logger buildLogger loggerName p = defaultLogHandlers >>= \hs -> (`fmap`getLogger loggerName) (setHandlers hs . setLevel p) -- Given a MailerBackend (an large piece of code that handles emails whenever they need to be sent), and the overall AppConfig (needed to get the logging configuration, `lc`), and a logger to start with -- this function creates the email log handler (remember, EmailLogHandlers require a logger so that if the EmailLogHandler can't start, it can tell *something* about the failure) addEmailLogHandler :: MailerBackend -> AppConfig -> Logger -> Logger addEmailLogHandler mailer c = addHandler (EmailLogHandler mailer p defaultLoggerFormat lc) where lc = appLoggerConfig c p = loggerEmailPriority lc setupMailerLogger :: MailerConfig -> IO Logger setupMailerLogger = buildLogger "App.Mailer" . mailLogLevel -- mailLogLevel gets the log level out of a `MailerConfig` -- ... further down in the code, in the area of the code that sets up the app ... -- Set up the mailer mailerLogger <- setupMailerLogger mailCfg mailer <- makeConnectedMailerBackend mailCfg (Just mailerLogger) -- ... in another file dedicated to the `MailerBackend` (a big app component that handles emails whenever they need to happen) ... data MailerCfg = MailerCfg MailerConfig (Maybe SMTPConnection) data MailerBackend = LocalMailer (Maybe Logger) MailerCfg | SMTPMailer (Maybe Logger) MailerCfg makeConnectedMailerBackend :: MailerConfig -> Maybe Logger -> IO MailerBackend makeConnectedMailerBackend c l = connectMailer $ (if host == "" then LocalMailer else SMTPMailer) l (MailerCfg c Nothing) where host = mailHostname c port = mailPortNumber c -- ... lots more code: the instances for MailerBackend, functions like `connectMailer`, on and on ... ------------------------ -- Email log handling -- ------------------------ data EmailLogHandler = EmailLogHandler { mlhBackend :: MailerBackend , mlhLevel :: Priority , mlhFormatter :: LogFormatter EmailLogHandler , loggerConfig :: LoggerConfig } instance SLH.LogHandler EmailLogHandler where setLevel mlh l = mlh { mlhLevel=l } getLevel = mlhLevel setFormatter mlh f = mlh { mlhFormatter=f } getFormatter = mlhFormatter emit mlh (priority, msg) logName = simpleMail from to subject body body [] >>= sendMail (mlhBackend mlh) where to = Nothing `Address` (loggerToEmailAddr . loggerConfig) mlh from = Nothing `Address` (loggerFromEmailAddr . loggerConfig) mlh subject = DT.pack $ logName ++ " - " ++ show priority body = DTL.pack $ "Log record:

" ++ msg

Some pretty obvious similarities to the Graylog-based code – going to prove the age-old addage(s) of success being built upon repeated failure. Despite not ultimately using the Graylog based log handling I set up earlier (it was working, however, so that’s good), I was able to parlay that pattern/structure into something I DID end up using.

Next Steps

This experience made me want to write a smaller, simpler, and less memory intensive open source log monitoring solution that competes with gryalog, however, maybe something meant to be ridiculously simple, like storking the last 500 or 1k (or configurable) log messages, and rotating them out automatically. I think I could get a long way with SQLite + SQLite FTS and a very Graylog-like simple frontend to watch things with. SQLite even has some support for LISTEN/NOTIFY type features – maybe I could make it a rust project? I’d finally get a chance to do something interesting in that language which would be awesome.

I also considered making a small library out of this functionality (the Graylog LogHandler), but ultimately decided against it because of laziness and the likelyhood that if I could think of and implement this solution within minutes/hours, someone else could easily do the same thing so it’s not that impressive. Maybe I’ll revisit this some day, but it’s more likely I won’t – hopefully people will at least find this page and get a good starting point.

BONUS: Want to catch unexpected errors with Servant? Set some settings on Warp

Hot tip: Don’t be a dummy like me, if you’re wondering about error handling for critical failures while using Servant, You should be handling it by adding some settings to the underlying Warp server. Here’s an issue I filed about it and felt kinda dumb right after other project contributors alerted me to the obvious solution. Here’s what the code looks like:

-- ... inside an IO action that sets up the application (I included some lines for context) ... -- Start the app let appGlobals = ApplicationGlobals c backend mailer userContentStore appLogger let app = makeApp appGlobals cookieMiddleware sessionKey logL appLogger WARNING "Application starting up...." -- Set the default application-wide on-exception handler let appSettings = (setTimeout appTimeoutSeconds . setOnException (appOnExceptionHandler appLogger) . setPort port) defaultSettings runSettings appSettings app -- ... more code in between ... appOnExceptionHandler :: Logger -> Maybe Request -> SomeException -> IO () appOnExceptionHandler logger _ e = logL logger ERROR (show e) >> when (defaultShouldDisplayException e) (TIO.hPutStrLn stderr packedErrStr) where packedErrStr = T.pack $ show e