CasaOne is a technology powered contemporary furniture rental company with a mission to deliver the best rental experience to our customers. Our software services run lot of scheduled tasks periodically in the background to ensure the inventory data is accurate, send email and text notifications to keep the customers informed about the furniture subscription, delivery, pickup etc.

During early days of the startup, we had started with simple in-process cron schedulers to get up and running fast. It looked like this

As we started serving more customers and cities, this approach didn’t scale well. We couldn’t run multiple instances of service, as it would have led to undesirable duplicate executions of the tasks. This hampered availability of our services, during peak load and service upgrades

We had to find a solution to run multiple instances which co-ordinated to schedule these tasks only once as desired. We couldn’t find any out-of-the-box library for distributed scheduling across multiple instances. Building and managing our own solution for distributed scheduling didn’t seem like the best investment of our time

We took a step back and started breaking the problem into 2 parts

Scheduler for triggering the tasks at periodic intervals Distributed workers executing tasks from a task queue

This decomposition simplified it to 2 smaller problems with well known architecture patterns and solutions. Part 2 of the problem, got reduced to the way we run other background tasks in the system using AWS Simple Queue Service(SQS). For part 1 of the problem, we used AWS CloudWatch rules to publish messages to SQS task queues at periodic intervals.

Final solution looks like this

We wanted to ensure this solution is easier to work with during development and deployment. Using localstack during development and terraform for AWS infrastructure automation helped in achieving this goal. As an added benefit, force triggering any of these tasks was as easy as publishing message to the task queue

This engineering effort to compose existing solutions saved us from a lot of development effort and helped us provide highly available service to our customers!