Challenges We’ve Faced

If this all sounds like music to your ears, that is great! But like any framework, it’s not all rainbows and unicorns. There certainly are challenges that we’ve faced and continue to encounter.

Bugs will slip through — be prepared

By not fretting as much about manually testing everything all the time, bugs WILL slip through. Anyone who tells you different is probably selling some kind of snake oil. This is a calculated risk though. Think the 80/20 rule, only in this case it’s probably more like 95/5. That is, automated testing (5% of the cost, compared to an army of manual Testers) will probably catch 95% of the problems.

We have a couple systems in place to combat those cases where something DOES slip through.

We find out about it quickly. Our support team is constantly dog fooding our app (as mentioned earlier), and we have enough monitoring in place to usually find out about a bug before a customer comes across it. We’ve invested heavily in building a quick deployment pipeline. From the time a fix is identified and committed, an engineer has the tools and autonomy to get their fix rolled out to production on their own within 60 minutes.

Infrastructure and Flaky Tests

“Flaky Tests” will happen. Things will fail for no obvious reason. As I alluded to, our Tech Leads keep tabs on this, but there are times when the pipeline will grind to a halt.

We’ve taken a two pronged approach to this. Firstly, each team has a rotation of ‘build masters’ who are expected to do any babysitting that is required to get code deployed.

And we also have a small dev tools team who is focused on making it easy for anyone to see how things are going, and to see what happened when things go wrong. Among other things, engineers can get an overview of build health, and quickly drill down into a failing test.

The EventMobi Builds & Deploys dashboard

An Engineer spots a failing test — and quickly accesses a video to see what happened!

Would this approach work for you?

At our current size and level of product maturity — it’s been a very successful approach for us. Would it work for you? It’s certainly worth considering! As we scale, and as our product continues to mature, our inspect and adapt cycles will tell us if the process is breaking down. If we feel that the bar is starting to slip, we will change.

[If you liked this article, click the little ❤️ on the left so others in your network know. If you want more posts like this, follow EventMobi publication with the (Follow) button on the right!

Finally if solving problems like this seems up your alley, we’re hiring! Checkout our current open positions at http://www.eventmobi.com/careers/]