The part that really standout for me - not only as a bedroom musician but also as a software engineer was the idea of using the simplest tools possible and not getting overwhelmed by the technology.

It’s something that keeps getting repeated even though we’re very much in the age of the resume driven development.

To add to that, I’m guilty of making bad technical decisions in the past. They weren’t small either - for example picking RethinkDB as one of the data stores at EnjoyHQ, took a considerable amount of time to undo and migrate out of.

When our team was faced with solving a particular architectural challenge that was impacting us for a long time - we chose to take a different approach than just building stuff: step back, evaluate potential solutions and pick the simplest possible approach.

Repeating yourself in the era of distributed systems

Here’s the problem: our backend is composed of 10+ Clojure services. Around 70% of them have to run work on a fixed schedule (send account digest emails, fetch new data from a 3rd party service, do data cleanups etc). Given that not everything can be made idempotent, and we’re strong believers of having more than one of everything we had to come up with a way of ensuring that only one scheduler runs across all instances of a given service but without having a single point of failure.

For a long time we have worked around this problem by introducing a configuration setting which would instruct one of the instance of a service that it’s the designated scheduler runner and then provision an extra instance of that service to be the designated scheduler. This is not ideal because:

it ties service operation to how it’s deployed

it’s a single point of failure

if a deploy happens at the same time when a scheduler is supposed to run, we might miss the “tick”

But we put up with it despite these pitfalls, because this setup just worked. But after one-to-many issues we had to go back to the drawing board.