Writing backend systems

Backend contains data models -- which is how your data looks. This can be your Django models or database tables. More often than not, we run into a problem where the model behavior changes faster than the actual data model. Changing requirements or additional features make us change our data models to account for new features that we introduce into the backend. Over a period of time, we also remove some of the old features and fix some bugs but the data models don’t change very often with these changes.

We then run into problems where the additional features built upon old features break existing functionalities in certain edge cases because a certain flag was not reset. Sounds familiar?

Writing stable backend systems

For example, take the following scenario: a payment is “initiated” by the user, we then “capture” the payment from the payment gateway and then “complete” the payment process. Any payment now goes through the three phases: started, captured, completed.

Solution 1 -- Database flags

This can very easily be captured via few flags: is_started, is_captured, is_completed

This solution is really straightforward, but it has a few obvious drawbacks:

As your number of states increases in the system, it becomes increasingly difficult to handle edge cases and conditions like you see in can_capture, we have to check if the proper flags are set.

The complexity to maintain the model increases exponentially as the number of flags increase, as we have to check all our methods and see how they behave when each of the flags is set or reset.

Solution 2 -- using a separate column for state

Since in this example all the flags are used to indicate a single state of the system, we can just use a state variable: state which can have a value of either started, captured or completed and remove all the different flags.

This seems like a much cleaner way of implementing the functionality, and we have gotten out of the situation where we need to check multiple flags. Since the object can only be in one state at a time, this approach greatly simplifies the system for us. This is awesome!

But now we are losing money because few of our transactions are failing while contacting the payment gateway. Luckily, our payment gateway allows us to retry an incomplete transaction, so now we can mark certain transactions as incomplete and then later retry them.

We can make the following changes to our model:

Wow! Not very complicated. This solution allows for much more flexibility over the previous solution because now you don’t need to alter any database tables if your states change in the future. But at the same time, this solution does have a drawback similar to the previous solution. Based on any new states added we have to update some our existing methods to take into account the new incomplete state and make sure that we are considering all the edge cases. Can we minimize refactor required to the existing logic and still implement the functionality?

Every time we update an existing method, we have to update their tests and also make sure that there are no regressions and the system is working as expected.

Also, note that this solution will only work in cases where the data cannot be in more than one state at the same time. In which case, it is better to have separate columns for a separate group of logical states like in the previous solution implementation.

Data has different states

Now it must be clear to you that data and its behavior changes very frequently as business requirements or system status changes with time. It is very easy to demonstrate the data state changes in a diagram.

State diagram

If we add retry feature into the system, this is how it will look as:

Is there a way where we can model our database model (no pun intended) in a similar way and still perform all of your business logic in the same way?

State Machines Crash Course

For those not familiar with state machines (commonly known as “finite state automata” in computer science textbooks) have 2 major components:

States

Every state machine consists of a finite number of states, which is the state in which your data can be. There are some special kinds of states, like: start/initial state: This is the initial state for your data, also the state from which your machine starts from. end/final state: This is the state which is considered to be the end goal or final state of the machine, after which no further transitions are possible. There can be multiple final states for a state machine. Transitions

Transitions indicate how a state machine moves from one state to another. Every transition has a source and an end state.

There may be transitions which don’t change the state of a machine, these transitions have the same source and end state.

When to use state machines

The complexity of the program increases when the number of flags in your model starts increasing, if you have more than 4 flags, which are being used for similar states, then modeling it into a state machine is worth considering.

If you are using an attribute to store the current state of the model, then this model can be modeled as a state machine.

If your business logic has a lot of flags based on some of your model attributes, you can try and find ways to store data in your model and then re-model it as a state machine to simplify your business logic.

When there are complicated transactions in your codebase, we can run into issues because of distributed processing. If you are storing an intermediate state of a single or distributed transaction, you should definitely try to use state machines in this case.

There are a lot of libraries in Python that implements a finite state machine. I will present an example using, pytransitions which is fairly easy to setup and extend.

Example using pytransitions

$ pip install transitions

Implementing the machine

As you can notice, there are no additional code and edge case checks over here, you can use the capture(), complete() methods in the same way as we were using them in the above examples.

The real beauty of state machines is evident when we have to make changes to them. For example, if we have to implement the retry mechanism in the above state machine. It will look like this:

As you can see now, our model logic is now exactly same as the state diagram above, and we have not done any major changes to any of our existing functionalities. The new states and transitions are mostly independent of the existing ones

Conclusion

We have used the power of state machines to implement logic that is modular, robust, testable self-documenting and easy to maintain. It will require lesser amount of refactor for future updates as long as the data has mutually exclusive states.

We should bear in mind that even though state machines are powerful tools to solve certain kinds of problems, it is not a panacea for all your database modelling problems and not all problems can be modeled using state machines, but nonetheless, it is a good tool to have in your toolbox.

Best practices

Never use set_state or transition to a state directly in your logic, always use state transitions. Make changes to your existing state machine and data models based on new requirements and features.

Extra reading