tl;dr: Paul’s done some fun things with event sourcing, we released a site.

We recently deployed our first functional site. It’s not fancy— it’s currently just some information about who we are and an email sign up form if you want to keep up-to-date with our engineering adventures. However, those of you who know us will know that we tend to like standing up in a hammock. So, we wrote it in Haskell. And decided to go fully event-centric.

Paul explaining our email sign-up state diagram to our photographer friend Liz, whilst Dave pretends to listen but is really playing bass guitar.

Seriously though, we wanted to try out some of these new (to us) ideas to see how they would gel together and whether we would want to use any of them in production. What follows is my account of using David Reaver’s event-sourcing/CQRS library, Eventful, to build our email sign-up backend. There’s quite a sprinkling of Haskell source code in this article. I’ve explained each snippet as we go, so hopefully it shouldn’t be too daunting even if you’re not familiar with Haskell.

Hello Eventful

Somewhat unusually for a Haskell library, Eventful came with a great little tutorial and a bunch of examples. So once you’ve got the high-level gist of what Eventful is trying to help you do, there’s quite a gentle learning curve from reading the examples to adapting them for your own needs.

At the core of the library are five concepts: events, event stores, states, projections and commands. You may recall event-sourced applications revolve around events as the primary source of information, rather than storing the final target state. So, the first four concepts are intimately related: Events are things that happened to our system, which are our primary artefacts and therefore what we persist in event stores; States are recovered from events by projecting a stream of events from some initial start state.

Command/query responsibility segregation (CQRS) is primarily about separating your update/writing behaviour from your querying/reading behaviour. That’s where commands fit in. In general, this area felt the least mature in Eventful. That could be due to there being no Right Way™ to break up your read and write patterns in general.

Let’s define some events

So far, so abstract. Let’s start looking at our email implementation by looking at what events we defined for our system:

Users can only do three things in our system. They can submit their email address, they can verify that they own the email address they submitted, and they can unsubscribe from our mailing list. These events come to us via our web server. In addition, we need to track whether we’ve actually emailed our users, so our system can send itself Emailed events.

Note that it’s a good idea to name events in the past tense — events describe things that have happened. This will help keep the code understandable when we define commands later on.

Defining state

Deciding what to do at any given point requires us knowing our current state. So let’s first define what the state for any particular user is:

Our UserState is made up of three components. The first, VerificationState , is a type that describes what phase of verification the user is currently in: Unverified , Pending or Verified . We also keep track of what emails we need to send and the user’s email address.

initialUserState is the UserState value from which all other UserState values are derived. More on that in a moment. We also define a helper function withinValidationPeriod to help us know if a particular UserState is allowed to transition away from Pending .

Tying events and state together

Now we need to tie together UserEvent and UserState , which we do via a Projection type. This is the first time we see Eventful getting involved in our code. There’s a reasonable chunk of stuff in the next block, but don’t panic! The majority of it is just logic inside one function and is conceptually straightforward.

The first thing to notice is that we define a type alias TimeStamped to wrap up our UserEvent values. We need to record the time at which we receive events because we need to only allow verification in a specific time window. The verificationTimeout constant is used to calculate when that time window expires.

The meat of this snippet is the UserProjection alias and initialUserProjection value. At the type level, the UserProjection type ties our UserEvent type and our UserState type together. At the value level, initialUserProjection ties together the initial state value we defined earlier and a function updateUserState for updating it in response to events. That’s everything we need to convert a stream of events into a state.

Note that updateUserState has no way to report failure. All it can do is take the current UserState , apply a UserEvent to it and give back a new UserState . This has implications about how we handle the user trying to verify their email address when the time window to do so has expired.

The definition of updateUserState might look a little complicated at first sight. Let’s break it down. First, notice that there is a definition of updateUserState for each of the four event types it can receive.

Taking the simplest first: If we receive a UserUnsubscribed event, we simply reset their state back to initialUserState , as if they’d never submitted any information. [That’s the data protection check-box ticked, kinda. We’ll come back to that towards the end of this article.]

If we receive a UserVerified event we just record that the user is now Verified . The update function is pure, so has no access to the system clock to check if the verification has arrived in time. This ties in nicely with not being able to report failure, and is also crucial because we need to be able to guarantee from a given event stream that we can always recover the same state.

If the system recorded that it Emailed a user, we filter that email out of the user’s usPendingEmails list.

Finally, if we receive a UserSubmitted event, we have to check if they’ve already verified their address. If they have, we just queue a ConfirmationEmail . If not, we note down their verification as Pending with an expiry time derived from the time we received the UserSubmitted event, and we queue up a VerificationEmail .

Command and conquer

In Eventful, commands are present-tense actions that might result in events. I personally found this abstraction slightly less intuitive and perhaps less useful than events and projections.

Our system defines three commands that it can be given for any particular user: Submit , Verify and Unsubscribe . These mirror the three past-tense user events we can encounter, but omit a corresponding command for the system-generated Emailed event.

Similarly to how we bound events and state together with a Projection , we can bind UserCommand to both UserEvent and UserState via a CommandHandler type alias. We also define a userCommandHandler value that wraps them all together at the value level.

There is, however, one important difference from our Projection here. Notice how userCommandHandler receives a DateTime that it passes to handleUserCommand . This means we can make a decision when handling a command as to whether the user can be verified or whether they’ve delayed too long.

This distinction is subtle, but important. Commands are handled when they are actually given to the system, and they generate events which are stored. When handling events themselves we don’t pass the additional context of time. If we did, we’d generate different states depending on when we rebuilt the state from the log!

Persistence

Handling commands, generating events and producing states is no good if we can’t have some kind of record of what happened to our system. For production, that’s likely to be in a database somewhere, but for testing or prototyping that could also be a simple in memory store.

One of the useful abstractions that Eventful provides is that of an event store — specifically the EventStoreReader and EventStoreWriter types. This means that you can write code to store and retrieve events without worrying about the details of how those events are actually going to be persisted. Eventful also provides guarantees about the order in which events will appear in the log.

So far, as you may have noticed, all the events, state and commands we’ve talked about have been from the viewpoint of a single user. When Eventful stores events, it does so in a single log so that the global ordering is preserved. Therefore, each user’s stream of events has to be identified with a key, for which Eventful uses a UUID.

How to use the in memory store is covered nicely in the counter-cli.hs example bundled with Eventful. That example uses a CommandHandler to process commands from the CLI and update the in-memory store all in a loop in main . When writing the email registration flow, my main function coordinates several different components (web server, mailer thread, and read view update thread), which need to handle updates asynchronously. Therefore, I ended up adding a little bit of structure for handling stores on top of that shown in the example.

Home-grown helpers

This section comes with a minor health warning — it’s just how my application works at the moment and I include it only for completeness. I’m not advocating this as the Right Way™ of doing things, or that it’s how David Reaver intended Eventful to be used! Also, I’m eliding some extra stuff to do with how my components communicate, just so that the code is a bit easier to follow.

We start by defining an Action as a function that takes a UUID and a UserState , and can perform some action that returns a list of events. For now, we just wrap up userCommandHandler as such an action. Later we can use this abstraction to look at the current UserState , send any outstanding emails and then emit the corresponding Emailed events.

timeStampedAction is a little wrapper that brings together working out what the current time is and injecting that into the UserEvents resulting from an Action . If you recall, we need the time on the events to know whether a verification is allowed or whether the user is too late. However, we want to inject it only in one place — when we’re handling writing new events to the store.

Next we have the Store abstraction, which is a way of gluing together functions on an EventStoreReader and an EventStoreWriter , with an interface that handles our newly-defined Actions :

In the snippet above, you can see that we have a function for making a newInMemoryStore and that calls an abstraction that handles (line 42) getting the latest state, applying the action to generate events and storing the new events. This is useful, because it means we can separate out creating and initialising the Store from running actions and applying the results.

And now — the moment you’ve all been waiting for — an example of how this all hangs together:

Here, we create an empty in-memory store, and run a bunch of actions that illustrate what a typical flow through our email sign-up process looks like. In production we can simply swap the store for one backed by a database like PostgreSQL, which is supported by Eventful.

hashEmail is simply a function that can (securely) turn an email address into a UUID. (In our case we use a salted SHA256 hash.) This means that we can always locate the event stream for a particular email address, but it’s extremely unlikely that someone else could guess the identifier. This is a Good Thing™, because we don’t want other people using verification links for email addresses they don’t own!

sendPaulAnEmail would typically be triggered by something else watching the event store and noticing that it needed to act. This component is called the Mailer in our production system, and is actually prompted to act based on a message queue that the Store writes to when it has updates. In the example we just knew to “send” an email.

In the production system, the commandActions correspond to side-effects triggered in our HTTP handlers for client requests. The email address actually just comes in via and old-school URL-encoded form POST!

Read views

We’ve covered a lot in this post already, so I don’t want to add too much more. However, it would be remiss of me not mention how can make global queries of the data we’ve collected. After all, what good is an email registration service that can’t report registered emails?

The problem is that so far in this article we’ve only been dealing with updates to individual streams of events. To report all the email addresses, we need to traverse the entire event log to capture them all. Whilst any one event stream may only contain a handful of events, we could potentially have a very large number of sign-ups. Traversing the whole log every time we need the list of emails is expensive. So, we need a way to process each event written to the log exactly once to update a cache of the current state.

In our production system we have a database table that contains all the email addresses users have submitted. We also store metadata about how how much of the log has been consumed to generate that table. We have a simple ReadView abstraction that runs in a thread and is triggered on log updates to refresh the view only from events that it has not yet processed.

If we ever want to make a different view onto our data, say one that shows us just the email addresses of people who have unsubscribed from our list, we can easily do that because our log contains that information. That’s why earlier I said that we only “kinda” handle our data protection obligations. It is certainly true that our existing read view doesn’t give us the ability to spam unsubscribed users, but it’s not true that we never could.

If we truly want to forget our users, we will need to compact the event log. That is, to remove information from the log that is no longer required. As far as I know there’s no utility for doing this in Eventful, and our service doesn’t yet truly forget its users.

Summing up

So, that’s a pretty accurate walk through of how our email sign up flow works under the hood. Leave us a response if you want more deep-dives like this. I hope it’s been enlightening, and has given you a better understanding of how event sourcing can work in practice. If nothing else, it’s been — sorry in advance — Eventful.