Like many technology organizations, Customer Care Technology, my team at The New York Times, uses scrum as our main process. To plan our sprints, we would go through the backlog, pick stories and assign a story to each of our six engineers, who would then work hard to get their story done. Because of unforeseen roadblocks, the stories would often get pushed to the next sprint or we would work together on a story to finish it before the end of the sprint. The problem with this process was that by the time we realized we needed to work together, it was often too late in the sprint.

Recently, we started working together, or swarming, for our entire sprints and only taking on a small number of stories at a time. Swarming is not a new concept; Many agile teams often swarm on a story to finish it before moving on to a new story. We took the definition of swarming and made it or own: We don’t just swarm to finish what has already been started, we swarm as soon as we start something new. And we’ve become more productive and collaborative as a result.

How we got here

A few months ago, the head of our department had a meeting with our team to encourage us to try a new way of working. As an engineering organization, it is important to focus on the right metrics, so we often focus on utilization or efficiency. However the more important metric is throughput: How much value can we deliver to the business and how quickly can we achieve this?

Translate this to scrum. Focusing on reducing the amount of inventory (work in progress) we have at any given time means we can execute faster. Working on fewer stories at one time means we can finish more, faster.

To understand how this works in practice, imagine this sprint scenario: We start with one story (let’s call this Story One), but Story One gets blocked by an external dependency, so we move on to the next story (we’ll call this Story Two). We have more work in progress. Because we started on Story Two late in the sprint, it is not done by the end of the sprint. During the next sprint planning we identify three new high priority stories (Stories Three, Four and Five). Now there is even more work in progress.

At this point, the dependency that blocked Story One is resolved, so we switch gears to work on Story One again. But when our product manager asks us when we can get to Story Six, we have to finish stories one through five first. Increased work in progress hinders throughput. It takes longer and longer to get anything done.

The head of our department suggested that instead of assigning one story to each engineer, multiple engineers should work simultaneously on one story. Thus we would have less inventory, less waste and increased throughput.

At our first sprint planning meeting where we were going to swarm for an entire sprint, we divided the team into subteams and took on only two or three stories. When my subteam met for the first time, we looked at each other and scratched our heads; Trying to figure out how to divide the work was awkward.

The idea behind swarming is to divide a story into smaller tasks that can be done in parallel, but sometimes it can be hard to divide the work because the tasks can end up being very interdependent and even overlap.

Different stories will require different solutions. Here are some of the ways we tackled this.

Techniques for Swarming

Divide by functionality

This may be the easiest and most logical one. If the story is, “Display a list of transactions, and for each transaction, there should be a ‘delete’ button to delete that transaction,” one engineer could work on displaying the list, while another could work on the ‘delete’ aspect.

Another example, if the story is, “build new functionality that should only show up under certain circumstances,” one engineer could start working on the conditional display of the feature, while another engineer works on the feature itself. It is probably the most natural way of dividing up work, but everyone would still need to work closely together to coordinate the touchpoints between the two tasks.

Divide by front-end and back-end work

This is pretty obvious as well and is an easy way to divide work. Very often an engineer can start working on the UI, while another can work on the back-end. Again, working closely together is key. Agree on the contract that will bind the front-end and back-end first before you start working.

Pair program

Two engineers work together on one machine: one person is the driver and one person is the navigator. The driver writes the code and the navigator sets the direction, observes and reviews. The benefits of pair programming are well documented: fewer bugs, shared knowledge, more efficient problem solving. As part of our effort to swarm, we have done our share of pair programming as well. Especially for more thorny issues or complicated features, it has proven quite valuable.

Test-driven development

If you practice true test driven development, you’re supposed to write the test first, see it fail and subsequently write the feature. In our version of swarming, one engineer starts writing component tests and/or integration tests, while the other one works on the underlying functionality.

Integrating QA in the development process

We have gotten much better about integrating QA in our development process. We used to loop QA in after our code was pushed to the master branch, which was clunky. Now our QA engineers will identify and create a test plan as soon as we begin working on a new story. As initial development nears completion, an engineer and a QA engineer will run through test scenarios together on our local machines.

The QA engineer may notice something that we had not thought of, or something that was missed in development, and at that point it is easy to remedy and fix. When the QA engineer likes what they see, we merge the branch into master and QA can verify it again on the dev server.

The way we are doing QA now has been a huge revelation to me and has had a major impact on our team. Because QA is more involved with the coding earlier on in the process, we have been able to develop features much faster. Bugs and unforeseen features are discovered earlier in the process and can be handled immediately.

Illustration by Tim Peacock

Intense planning and coordination

In order to make sprint-length swarming work, you need planning and more planning. We typically have a story lead who meets with the members of the team and together, they break the story into subtasks in the way that makes the most sense. It is a collective planning effort. At that point we also agree on the contracts between the various subtasks. We also rely on the story lead to perform any needed coordination as we move along with the development.

It is often believed that scrum involves less (project) planning, but we all know that this is not true. When you swarm, you need even more planning.

Takeaways

After “swarming” for a few sprints, we started to get the hang of it. But ultimately, we are increasing throughput, which means we are completing features faster and delivering more.

There are some other benefits as well.

Few problems in this world don’t benefit from increased communication and software development is no exception. We talk about how to divide up the work, and we stay in close contact while we’re working because our projects often overlap. This means we can identify issues or roadblocks early in the sprint and correct accordingly.

I would also argue that increased communication leads to better software design decisions. We often have a healthy debate about the best way to tackle problems, and we probably wouldn’t have these conversations if we worked independently

By working closely together and discussing new features, we inevitably learn from each other: we learn about our specific domains, about the application we’re working on and general software building best practices.

All in all, this has been a very positive experience for our team and I encourage everyone to try it. If you have any questions, feel free to reach out.