You Aren’t Ready For Scrum

As startups grow, most engineering teams hit a point where they find it necessary to add some amount of process to keep things in order. Often, they jump from no process directly into a relatively-heavy process such as Scrum. While Scrum works very well for projects with mature architectures, it doesn’t fit into fast-moving environments where the foundation has not yet stabilized. This is due to a number of causes:

• Variance in estimation is often so large as to be of no value

• The cost of accounting overhead (meetings) is not justified by the ROI

• Sprint cycles combined with rigid contracts can create long feedback/correction loops

Fortunately, process doesn’t have to be all-or-nothing. Lean alternatives can bridge the gap between a complete lack of process (which works in early stages) and Scrum (which works for mature projects). Three concepts in particular fit together to provide the right amount of scaffolding for maturing projects: steel threading, continuous delivery, and full transparency.

The Steel Thread

Many software development projects still take the waterfall approach: lay out specifications, break them up into components, create isolated teams, then finally integrate! It’s like building a bridge starting with individual segments and trusting that they will be aligned when they meet.

That process works for bridges, because it has been honed for millennia. Software development is immature by comparison, and it is more likely to succeed when built like an Inca rope bridge: start by shooting an arrow tied to a small thread across a canyon.

Then pull over a string, then a rope, then more ropes, and the next thing you know you have a bridge! This way there is something usable from the very beginning. Sure, it might only support small objects, but at least it’s performing its function of being a bridge. Getting people across is now just a matter of scale. Repairing or adding to the bridge is just a matter of repeating the process. Meanwhile the stakeholders have constant visibility into how the project is going.

With the steel thread approach, the goal is much the same: start with one feature and get it working all the way to deployment:

1. Start with an import from the old system, even if just one database table

2. Get the app connected to the database

3. Select a web framework

4. Get the content into the browser

5. Make sure it will all persist back again!

The particular steps will vary, but they should all be completed in the first sprint! The new platform doesn’t need to have every whiz bang feature, but it has to work right away. It also has to be fully deployed, and the product owner has to log in and use it. After that, everything is just a feature. Security is a feature, scalability is a feature, even functional correctness might be a feature. The product isn’t complete, but your Continuous Delivery pipeline is in place, and that’s what is most important.

Continuous Delivery

To fully realize the benefits of steel threading, you also need Continuous Delivery. You can think of Continuous Delivery as an incremental improvement on Continuous Integration: in addition to compiling and testing every commit, an automated system also deploys the app to a staging environment to which the organization has access. This has 2 key benefits. The first is that the process of delivering software to production/staging has been automated, eliminating human error and ensuring rapid delivery. The second benefit is that it code is quickly exposed to the “sunlight” of user testing. This ensures rapid feedback and eliminates waste.

To achieve this, it’s important to remember that no feature branch should ever be too far diverged from master (or develop). Set a time limit (e.g. <= 1 sprint), and if the divergence lasts longer than that, delete the branch! It might look like you’re losing value, but the real value is the knowledge you gained. Use that knowledge to break it into smaller tasks and try again. This ensures that features are small, and that they are getting merged and deployed quickly. This rapid deployment to a common environment creates full transparency.

Full Transparency through Hard Metrics

The definition of a Story is a simplified version of reality. Without conscious communication, everyone develops a different image in their head when they read a story. Stories aren’t reality. Products are reality. When you have a product right from day one, you have something more valuable than a collection of stories. You have a tangible thing that stakeholders can try out. Everyone can see the exact status of the development team as they progress. Anyone from QA to to the CEO can contribute, and feedback is flowing through the organization right from the start. This feedback is rapid and continuous, not a multi-sprint feedback/correction cycle.

Instead of stories and points, progress should be measured by selecting a fixed and tangible aspect of the project to track. For example, on one server rewrite we had 100 REST end-points that had to be compatible with the legacy server. This was a hard metric: how many end-points had we completed? On another project, we were moving 100 web pages from a server-rendered architecture to a single-page-app, another hard metric. In a different project we were writing a database engine that had to implement all 60 keywords defined in Spark SQL. These are all examples of hard metrics with concrete acceptance tests. When all of them were converted, the project was complete! This gave us a consistent progress indicator, and it saved developers the task of point-sizing tickets on an unfamiliar and evolving architecture. Only at the end of these projects, when the architecture had stabilized did developers have the familiarity to confidently estimate.

Conclusion

Keep it real! Admit you don’t know enough to dwell on long term plans, and focus instead on rapid iteration combined with constant progress tracking. Favor the verifiable products over abstract stories, and observable metrics over the guesswork of estimates. An observation-based forecast is better than a speculative plan! Keep all roles in sync by focusing on a common view of the world, the product itself, and make sure everyone is communicating. Think of this as TDD for your development process: don’t act without evidence! If it works for your code, why not for your process?

Only now are the team and the product ready to graduate into Scrum.

Give us a Shout Below

Have you jumped into Scrum too soon and got burnt? Got a story to tell? Your comments are valuable to us!

Need Help?

Contact us if you need help choosing or implementing the best solution.