In this article we will give an overview of the circuit breaker pattern and explore one Golang package providing an implementation.

Circuit Breaker Definition

Context

Resiliency is one of the architectural qualities in systems design. It is the ability of a given system to adapt and recover from failures. The circuit breaker is a pattern intended to increase the resiliency of a system.

Use cases

In a system it is possible that a key service becomes unresponsive.

The risk when one service is unresponsive is that the bottleneck created can bring other components of the system down.

Example scenario

Pizzeroo the on-demand pizza company has launched the 1 million free pizzas marketing campaign at 12am on their website. At 12.15 the traffic is 10 times higher than usual and users are experiencing longer and longer response times on the website because the Order service has reached its maximum capacity. Auto-scaling is in place but it takes 10 minutes for new instances to be created. Users are getting frustrated and keep hammering the submit button to not miss out on the offer, which creates even more requests. At 12:20 system administrators start noticing that services consuming the Order service start running out of memory because requests pile up in memory. At 12:22, the website is still up but all requests are now timing out, users are furious and start posting on social media how upset they are at Pizzeroo, investors take notice and the company’s shares start to plummet…

This scenario is completely fictional of course however it is realistic enough to give a good use case for a circuit breaker.

Solution

Administrators and developers at Pizzeroo have studied this production incident and decided that there needs to be a circuit breaker around the Order service, they also realised that scaling up the service would not have helped because Order is dependent on legacy third party APIs that cannot handle a very high load.

A circuit breaker can be used to detect when a downstream dependency starts failing and returns an error immediately instead of waiting until the requests time out. Stopping requests going through prevents the problem from spreading and notifies clients that the system is temporarily down instead of slowly collapsing.

Next time there is a high load on Pizzeroo’s website and the Order service reached its maximum capacity, requests will fail immediately and the frontend can even disable the submit button and display a friendly message while the circuit breaker is activated. The Order service is given some time to process requests and has some capacity again and users can submit their forms.

A good analogy : the Order service getting a break while it processes current requests…

The 3 States of a Circuit Breaker

As we have seen the circuit breaker essentially decides whether to let a request go through or not. This decision is conditioned by 3 different states which can be thought of as a state machine.