All that junk that is not inside your trunk

Trunk based development has been used widely in the world of backend and service development for some time now. A study of many thousands of development teams across the globe has concluded that teams which rely heavily on TBD with a very strong focus on DevOps perform much better, produce less errors in production, can fix errors if they happen quicker than others and are overall happier employees and developers (https://itrevolution.com/book/accelerate/).

In all of my professional career, I have not once come across a team that was using TBD. The majority of iOS Teams I encountered utilised some interpretation of gitflow, with a CI/CD system that was hardly receiving any major focus, the teams were constantly stuck in a cycle of death. Business would put pressure on pure feature related output rather than investing in robustness. The velocity of tickets was more important, regardless of the amount of bugs or technical debt that was created as a result of it. I am not trying to point a finger here, this is just how things are based on the responsibilities of a role, rather than any one person.

It is the responsibility of the tech leads to clarify and communicate the importance of infrastructure. The quality of a city’s infrastructure impacts how effective the city runs. Similarly, the infrastructure of a software development team effects the quality of its product.

With TBD, a powerful CI/CD environment is key to driving successful teams, as mentioned in the scientific studies done by Nicole Forsgren PhD and Jez Humble. The numbers shown in the findings were promising enough to give TBD a go. However, without much resource on how TBD would work in an iOS specific case, my team and I plunged in and tried to figure it out as we went along. I wanted to share some of our findings, successes and the things that need to be improved still.

25 captains and no master push

At peak, I was working with a team of 25 iOS developers. Over time, 20 developers on average which were split in cross functional teams across 3 timezones. Each team would operate with about 2–5 iOS devs. At the pace that the project was going, a person would commit around 3 to 9 times per day, making it an average of 150 commits on trunk every single day.

Don’t get me wrong, in my opinion, no application should warrant for a team size of 20 developers. The entirety of the App was split up into multiple bite-sized projects, but following the monorepo approach and to ensure the application would work as an entirety, they were all located within a single xcworkspace . Since we decided not to engage with Interface Builder at all (neither nibs nor storyboards), our UI code was created purely in code. This was actually very helpful as it would allow for an atomic design of reusability. Each and every XFT (cross functional team) should be relying on the same custom button styles, the same AlertControllers , the same primary and secondary colours, and so on. On top of that, making changes to designs, even in situations where people across XFTs would make changes to the same or similar components, merge conflicts would be clearer to understand in the realm of git.

Experiencing this speed of development and the frequency of commits, working with the trusty 3 steps of 1. Create PR 2. Review PR 3. Merge PR would surface issues when trying to merge. If the review did not take place immediately, even if it was just a couple of hours, there was a high risk of having to make changes before a PR could be merged. Most of the times it was simple enough, sometimes whole PRs had to be abandoned and redeveloped, costing a lot of time and effort spent on the given task.

It might be unsafe to leave the deck unmanned

But why? I believe part of it has to do with some manifestation of a control mechanism. Some of the greatest developers that I have met in my career are the most difficult to work with, because they are tough on your code, they will scrutinise your output. Not because they do not care about you, its actually quite the opposite. They do care, immensely. People sometimes can take it personally which they absolutely should not.

These developers, through series of positive outputs and leaning towards control, tend to get the latter in the end. Flipping on the switch to protect branches in Github is easy. And in a normal sized team, being forced to review code before it is merged means that ‘controlling’ people can become difficult to bypass. Multiple feedback loops take time and this is a killer for productivity, morale, motivation and thus has an effect on output and happiness. Developers are rare as is, if we could encourage people to stop being the ‘developer police’, would it have a positive impact overall? Would the quality of output sink? These questions are hard to answer.

What can be defined as ‘quality of an application’? Velocity of tickets moving into development done? No, tickets can be sized differently and we do not estimate tickets by default. Number of errors found in QA Testing? Average review rating? How happy the client or the product owner is? I will try to focus on two things. How easy is it for developers to contribute to the application and how many issues do we have to solve continuously before we can release.

Releasing the kraken

We removed the restrictions on trunk on one of the small Xcode projects which contained all of our reusable UI components first. I advised people to push to trunk first, making sure that the code was either adding to the base and maintaining clear guidelines on how to increment versioning and to deprecate APIs that are no longer to be used in cases where components changed or disappeared. The CI would, with every commit to remote trunk, check out the code, build, test, archive and release to our distribution platform for all developers, QA and business to download.

In this controllable environment, the outcome was overwhelmingly positive. We pushed our code to trunk, CI would spit out a new App every 30 minutes. Merge issues were minimal because of higher commit frequencies. Fewer files and less code was modified with every commit. At the same time, people did not have to wait for reviews to finish. Idle time in-between tasks and stories was practically obliterated because they did not have to wait for code to be reviewed for them to be able to pick up another piece of work. Some might say, you would still be able to do some work despite having to wait for code reviews, but our reality is that teams are normally confined within a project, which sits within a number of files. Xcode project files, not being the best friend of git merges, would make things more difficult than it had to be.

Shipping and messages in bottles

In order to do TBD properly, feature flagging is essential. The application has to be in a production releasable state at any given point in time. This is to reduce the stress of creating a release build. A feature that is being built would be hidden in release and only visible in Debug. We would use a combination of plist files, macros and build configuration to enable releases to a set of environments and local build. Newer features would be available on dev, as they matured we would enable them on staging and once fully enabled, they would also be enabled on production. After which the feature flag would be removed completely if no longer required.

Crashing into an iceberg

Of course, humans make mistakes. People create breaking changes, miss running their tests before pushing and sometimes not even take the time to hit ‘run’ even once for minor quick changes. That has happened a number of times. Here is where a strong CI/CD pipeline comes into play. The speed at which the CI needs to verify a stable build has to be swift. 5 minutes was our ultimate goal. The CI would notify everyone on Slack that the build broke, what the commit message was and the responsible person that put out the commit.

With this approach, automatic policing and blame became the safeguarding element. Someone who would appear as irresponsible in front of other developers, would take more care and invest more time in verifying his or her builds. Even though they were the same people, even though we removed reviewing code before merge, we managed to create higher quality code. Relinquishing control had the exact effect which we were trying to get. Broadly speaking, the developers that experienced more responsibilities, also behaved more responsible.

Breaking the ice

Whilst code reviews did not happen as frequently as with PRs, it was still being reviewed regularly. From this, if technical debt was identified, new tickets would be created and scheduled for development from detailed feedback given by reviewers.