Testing race conditions in your Rails app

From time to time, there comes a requirement in your application where you have to guarantee that something can be done at most X number of times. For example, only 15 people can subscribe to a course, only limited number of people can buy this physical or virtual product, only 200 people can go to a concert, etc. How do you test that this limitation actually work? And not only work but that it works under heavy load, when there are multiple customers trying to buy the last item because it is a hot event or the product is offered with a big discount?

You might think that testing it is hard, and maybe not worthy? But I found out that sometimes it is not so hard at all, and it might be simpler than you imagine. Sometimes, all you need is 4 threads to reproduce and fix the problem.

specify do begin expect ( ActiveRecord :: Base . connection . pool . size ). to eq ( 5 ) concurrency_level = 4 merchant = TestMerchant . new product = merchant . create_product ( quantity: concurrency_level - 1 ) customers = concurrency_level . times . map { TestCustomer . new } fail_occurred = false wait_for_it = true threads = concurrency_level . times . map do | i | Thread . new do true while wait_for_it begin customers [ i ]. buy_products ([{ quantity: 1 , product: product }]) rescue Orders :: CreateOrderService :: Invalid fail_occurred = true end end end wait_for_it = false threads . each ( & :join ) expect_fail_for_one_customer ( fail_occurred ) expect_success_for_rest_of_customers ( customers , product ) expect_product_not_oversold ( product , concurrency_level ) ensure ActiveRecord :: Base . connection_pool . disconnect! end end

Let’s go step by step through this code.

expect(ActiveRecord::Base.connection.pool.size).to eq(5)

By default Active Record connection pool can keep up to 5 DB connections. This can be changed in database.yml . This just checks if the preconditions for the test are what I imagined. That no developer, and no CI server has these values different.

concurrency_level = 4

One DB connection is used by the main thread (the one running the test itself). This leaves us with 4 threads that can use the 4 remaining DB connections to simulate customers buying in our shop.

merchant = TestMerchant . new product = merchant . create_product ( quantity: concurrency_level - 1 )

We instantiate a new TestMerchant actor which creates a new product with a limited quantity available. There are only 3 items in the inventory so when 4 customers try to buy at the same time, it should fail for one of them. Actors are implemented with plain Ruby classes. They just call the same Service Objects that our Controllers do. This code is specific to your application.

customers = concurrency_level . times . map { TestCustomer . new }

We create 4 customers in the main thread. Depending on what being a customer means in your system, and how many Service Objects it involves, you might want to do this in your main thread, instead of in the per-customer threads. Because, we strive to achieve the highest contention possible around buying the product. If you create your customer in the per-customer threads it might mean that one customer is already buying while another is still registering.

fail_occurred = false

In one of the threads, we want to catch an exception that placing an Order is impossible because it is no longer in stock and remember that it occurred. We need to define the variable outside of the Thread.new do end block, otherwise it would not be accessible in the main scope.

wait_for_it = true

I couldn’t find a way in Ruby to create a thread with a block, without starting it immediately. So I am using this boolean flag instead. All the threads are executing a nop-loop while waiting for this variable to be switched to false, which happens after initializing all threads.

threads = concurrency_level . times . map do | i | Thread . new do true while wait_for_it begin customers [ i ]. buy_products ([{ quantity: 1 , product: product }]) rescue Orders :: CreateOrderService :: Invalid fail_occured = true end end end wait_for_it = false

We create 4 threads in which 4 customers try to buy the product which can be purchased max 3 times. One of the customers should fail in which case we will set fail_occured to true. The first thread is created faster than the rest of them and remember that we want high contention. So we use true while wait_for_it to wait until all threads are created. Then main thread sets wait_for_it to false . That starts the buying process for those customers.

threads . each ( & :join )

We don’t know how long it will take for the threads to finish so we gladly wait for all of them.

expect_fail_for_one_customer ( fail_occured ) expect_success_for_rest_of_customers ( customers , product ) expect_product_not_oversold ( product , concurrency_level )

Then we can check all our expectations. One failure, three successes, and product not over-sold. These, of course, are very specific to your domain as well.

When building such test case it is very important to start with red phase when you go through red, green, refactor cycle. Without the race condition preventing method that you choose to implement the test should fail. If it doesn’t fail it means the contention you created is not big enough.