How to apply Agile Principles to large scale transformations

I have been asked if Agile methodologies can be scaled successfully. The obvious answer is yes, for if they can not be scaled the methodologies have no agility. In reality the scaling process does involve methods that are additional to those identified as Agile Software Development.

These methods work well with and are closely aligned to agile principles. They are closely associated with the origins of Agile and Lean methods.

The roots

Agile and Lean principles derive from, among other things: Kaizen “Continuous improvement.” which “means that all team members throughout the organisation are continuously looking for ways to improve operations” Jidoka — “Making problems visible so that they can be immediately addressed” and Genchi Genbutsu which is “Going to the source to find the facts to make correct decisions, build consensus and achieve goals.”

The roots of Agile and Lean methods are founded the Toyota Production System of the 50’s. Around 1990 a detailed outline of the entire Lean system was explained in The Machine That Changed the World. At this time a problem existed in software development. Known as “the application development crisis” it referred to the gap between specifying requirements and delivery. At the time the gap averaged 3 years, but in some instances the gap was as big as 20 years. Against this background:

In 1999 Kent Beck published Extreme Programming Explained: Embrace Change

In 2001 the ‘Snowbird Meeting’ was held in Utah, where the authors of the Agile Manifesto met.

In 2003 Lean principles were presented, as an Agile tool, in Lean Software Development.

Although it is common to view refer to Agile Practice and Lean Principles simply as Agile and Lean (which I will do in subsequent parts of this post) both words are adjectives and should be followed by the noun (or the noun phrase) that they are qualifying.

Principles

Lean is built on the fundamentals of Purpose, Process and People. It recognises the need to do something as stemming from a problem. It asks if the methods of overcoming the problem are: valuable, capable, available, adequate and flexible. It requires that everyone is engaged in a process of continuous improvement.

The principles of Lean are:

Eliminate waste.

Amplify learning.

Decide as late as possible.

Deliver as fast as possible.

Empower the team.

Build integrity in.

See the whole.

Agile is built on:

Individuals and interactions over processes and tools

Working software over comprehensive documentation

Customer collaboration over contract negotiation

Responding to change over following a plan

The principles of agile are:

Deliver early and often to satisfy the customer.

Welcome changing requirements.

Deliver working software frequently.

Business people and developers must work together.

Trust motivated people to do their jobs.

Face-to-face conversation is the most efficient and effective method of conveying information.

Working software is the primary measure of progress.

Maintain a sustainable pace.

Continuous attention to technical excellence and good design enhances agility.

Simplicity — the art of maximizing the amount of work not done — is essential.

The best architectures, requirements, and designs emerge from self-organizing teams.

Reflect and adjust at regular intervals.

A reduced version of ways of doing things in an agile way were later offered by one of the manifesto authors as;

Find out where you are

Take a small step towards your goal

Adjust your understanding based on what you learned

Repeat

How to do it:

When faced with two or more alternatives that deliver roughly the same value, take the path that makes future change easier.

Face-to-face conversation as the most efficient and effective method of conveying information does not have the same relevance now that collaboration tools and net comms are so widely developed. I have sometimes found chat to encourage more communication than waiting to be in front of a colleague. This is splitting hairs however as the Agile Manifesto is not prescriptive, it is a set of general principles.

Lean proposes a holistic system as does Agile. Lean is not confined to manufacturing but can apply to many business activities, it is well established in service and software industries. Agile is equally not confined to the production of software. Despite being initially defined by programmers and intended to overcome problems in creating software, Agile has been successfully adopted by other practices.

The benefits of employing Lean and Agile principles both provide benefits to wider and more general technology projects. They can also be applied to other business processes. Lean methodology still breaks a large program into smaller tasks that can be completed individually. It also sets workflows for these tasks such as; discovery, planning, design, build, testing, and deployment.

The workflows of Lean sit well with modern practices such as devOps where

Continuous Integration

Continuous Delivery

Continuous Improvement

and a culture spread across, the tooling and technologies, and, the architectural are the cornerstone of the process to deliver fast and expose risk.

Teams operating within a Lean framework can each use their own Agile management tools, some teams may be XP, some Kanban, some Scrum, some 6 Sigma. It is quite feasible to have a polymath program where different methodologies sit side by side. If the program covers structurally distinct structures (e.g. hardware, software, network, business process) it is perhaps inevitable. It is certainly desirable to have a program where a set of methodologies share core elements that are based on agility.

While the difference between Lean and Agile is not that wide Lean has a more general set of principles they are easier to draw upon when looking to scale.

Scaling

As an example of how Agile/Lean methodologies can scale let’s consider a largish project. A dynamic and successful business finds that it has grown out of the technology it uses. It employs 500 people, operates globally and is located across continents. It has a core platform for delivering its product but this can no longer cope with the demand caused by growth. It integrates with number of other platforms for different subject areas.

The core platform needs replacing and at the same time many of the fundamental subject areas require new systems that match the growth and ambition within the business. All functional areas can be radically improved with new technology but the driver are the real pains and problems that are holding the whole company back.

Conventional Agile methodology as it applies to software development is not relevant to the scenario above. The example involves a program of change and implementation across workstreams and the business as a whole. It involves hardware, infrastructure and new ways of doing things.

The fix is not to scale Agile teams but to scale the projects by adding teams and including them within the wider Lean program. This does not mean that Agile principles get thrown out. Instead they sit in alongside the Lean toolbox to reduce disruption, waste, cost uncertainty and delays.

A business, or even a business product will consist of a number of technologies and applications. These will form a system which will be made up of interrelated and dependant sub-systems. A typical business will have a number of functional areas, for example: Finance, Sales, Operations, Marketing, HR, each will need to adapt to the changes the business is subject to.

Each functional area may have its own subsystems, finance for example may have: Invoicing, Purchases, Accounting, Payroll. These may be subdivided, invoicing may have a ledger that links to the accounting functionality, it may also have an application that provides invoice breakdowns derived from the operational sales application.

Each subsystem could have its own agile teams delivering internal or vendor supplied applications and functionality. It may have started with a need for one team delivering the features to support invoicing, this team may begin to grow in size to add additional functionality. At this point the opportunity occurs to split the team into two. Another agile team may then be assigned to provide payroll functionality. The teams need to be connected while remaining separate.

There now exists a number of feature based teams. The teams are agile. The business needs a way of connecting these, and other, teams to deliver a coherent system. The business/customer needs a method whereby these teams can be managed efficiently. Empowered teams that deliver quickly, reduce waste and build integrity into their products. Lean management principles allow this.

The arena where: the visions is shared, feedback is communicated, insights are revealed, value recognises and waste avoided is able scale accordingly.

The diagram at the opening of this sections shows a hierarchal view of a program structure. The one directly above is a far better way of visualising relationships and the structure as it suggests a program centric view.

Size

In the example transformation above the main sets of tasks are born from the need to replace the core platform. It is too big a task for a single team, agile only works well with small teams (5–12 members). One definition of the ideal size for agile teams is if it is too big to share two pizzas it is too large.

Any team that gets bigger than 11 members becomes inefficient. This does not mean program is confined to projects of less than 11 workers. It just means that teams need to be split before they become inefficient. While a team of ten can work it is also an opportunity to form two teams.

We do not want the teams going in opposite directions or duplicating work effort so someone needs to coordinate them. This is the role of project management. The role assumes the link between the team and the client (or for internal teams, the team and the business). The agile practices of:

Deliver early and often to satisfy the customer

Trusting motivated people to do their jobs

Face-to-face conversation being the most efficient and effective method of conveying information

Working product being the primary measure of progress

still apply. Alongside them sits the Lean principles of:

Deliver as fast as possible

Empower the team

Build integrity in

See the whole

The difference is that they are now reside within a wider implementation framework rather than a pure production framework.

Big projects will spawn many teams. If it is too many for a single project manager to handle it will require someone to sit over the project managers and pull them together. At this point the project has in fact become a program and requires a program manager. As the number of program managers increases another role is needed to coordinate their efforts and provide a conduit for communications and feedback.

The program manager will possibly be also involved in a number of work streams within the transformation. For example alongside the technological change they could be responsible for opening a new regional centre. These parallel work streams will also be able to benefit from the principles of;

Eliminate waste.

Amplify learning.

Decide as late as possible.

Deliver as fast as possible.

Empower the team.

Build integrity in.

See the whole

Early delivery

Delivering value early through iterative and incremental processes provides opportunities for feedback. Feedback is critical to allowing design options to be made later in the project lifecycle. Keeping options alive and open is a way to continuously reduce uncertainty. All of these help to eliminate waste.

Options will be influenced by the output of the iterations and the result of testing releases. Aside from providing incremental improvements they give ways to reduce risk and cost. Faults detected early save waste as they can be fixed before they affect too many parts of the system.

As complex systems are made up of subsystems and components. Frequent product releases, prototypes or demos of the subsystems and components gives flexibly and agility. This is achieved by specifying performance and functional requirements at a high level. An early release allows the product design and implementation to respond to feedback.

Parts of the system may have functionality that is replicated in other parts. It is useful to find duplication of features early. As an example; issue tracking, project tracking and task tracking use similar functionality and could be broken out into a single service rather than being built into separate sub-systems.

The Agile practices implicit in devOps enable and encourages early delivery. devOps is driven by a number of motivations. To regulate, reproduce and roll back. It comprises:

A desire to rationalise dependency management, to provision and configure all the constituents of the stack.

The requirement for reproducible deployments so that the whole application can be erased and then accurately reproduced.

The ability to create multiple instances of the same application and significantly to recover from failure at any point during provisioning.

The effect of taking a devOps approach, which has been described as “a cross-disciplinary community of practice dedicated to the study of building, evolving and operating rapidly-changing resilient systems at scale.” is that is allows test rigs to be thrown up and torn down very easily. This enables instances of the full stack to be delivered very quickly. As every change to a system should trigger feedback which should be obtained as soon as possible, having a test rig is a key component for this strategy.

Agile principles are not just for software

The advantage of automating and getting a full stack test rig up is that the hardware, network and configuration can all be tested so when it comes to releasing to production, no new errors can creep in. Small errors (a typo of 120 instead of 210) in configuration settings, made during manually entry, are capable of causing disaster. The error may not show until a critical time (when the system is in high demand) so the ability to use automating settings determined by load testing the installation could avoid the catastrophe.

Delivering working software frequently aids eliminating waste by delivering defects early and providing the opportunity to fix them. This allows fault recovery to be built into the process. Having a prototype for soft configurations of hardware allows settings to be fine tuned. Providing early access to hardware installations achieves the same result.

Applying a ‘test early’ approach to hardware could involve deploying a demo version of the hardware. This would allow it to be load tested and run as a prototype of the installation. In this incarnation it may suggest the design for a ‘failure recovery’ plan. As hardware is more prone to breaking more than software building in a strategy for recovery gives value.

For example if the hardware was a mobile phone. Having taken delivery of a particular model as a demo would allow the assessment to:

Configure it.

Test its performance in the field.

Evaluate the options for backing up data.

Time the sequence required to mirror the configuration.

Run a test of reinstating the backup.

This gives procedures and information that will save future waste. The performance may suggest an additional infrastructure or a different model. Having a demo is similar to the agile software practice of delivering early. It also allows a window to respond to change.

In the above example the capability of the hardware demo may reveal other capabilities that hold value such as having: a built in voice recorder, voice recognition, the capability for conferencing or navigation. Discovering this would prevent duplication within other program work streams. Capabilities that without a demo would be invisible to other work stream project managers.

Using the Agile principle of ‘Deliver early’ when adopting hardware or provisioning network also provides the opportunity for sharing demos across the business. This could lead to an awareness of other options early enough to prevent waste. Again as an example; waste could be reduced if a network solution allowed distributed data storage; as the need for an internal backup would be superfluous.

Decide as late as possible.

The advantages of delivering early can be negated if deliveries are not shared. We have all heard “If I knew it did that — I would not have bought this as well”. I have seen complex ERP systems put in place with overlaps of functionality to parallel systems. Yet the overlaps were not apparent until the end of a very long specification, configuration and implementation chain.

The number of times I have seen a business where the same thing is done in different ways by different people is amazing. Subject areas will frequently commit to a technology and not implement it for some time. This can lead to instances of functional areas having solutions that never get out of the box due to some other area already having a package with the same functionality.

It is not unknown for two projects to overlap, deciding late does not prevent this but it can help by giving more time to share the knowledge, achievements and pitfalls. Procuring a system that is dependant on a second system relies on both systems being coordinated to arrive at the same time and as expected. One system may change quite a bit between specification and delivery.

The decision to procure a system should only be made once the system it is dependant on is in a stable form. The decision to implement should only be made once the resources are available and able to benefit. Many packages stay unused because no one is has the time to familiarise themselves with them.

Dependencies are not confined to technology. If a work stream is likely to go in more than one direction before settling on a final place, the technology to support that work stream should not be decided until there is certainty. There is no point buying a help desk system if support is going to be outsourced.

Agility is also required here to prevent the tail wagging the dog and a willingness to embrace change is required. Deciding as late as possible should not lead to, or be confused with, indecision. The extent of discovery will point out when is the correct time to decide. This is usually when dependencies are in place and a good domain knowledge has been acquired.

It would be foolish to decide and then research but again this is a common sequence of events. A technology is acquired, installed and then the business starts to learn about it. Technology should be trialed before decisions are made about deploying it. There are many sales department that have gone through many CRM applications because they decided on which CRM they wanted before they had decided how they were going to use it. All subject areas are prone to this syndrome.

Managing the project managers

As projects get subsumed into programs this makes yet another Lean principle ‘seeing the whole’ ever more important. The role of the Program Manager may cover production alone but more often than not it will cross production and implementation. Project Managers on the other hand tend cover just one type of activity; possible production or implementation. On large programs a Program Director will cover whole transformation at a high level.

The success of projects is not dependent on micro management but on;

‘Empowering the teams.’

‘Incorporating integrity.’

‘Seeing the whole.’

Attempts to run plan based programs are on most occasions doomed and will lead to a lot of wasted effort. Requirements will change and they will change often as more insight into the new platforms are discovered. This does not mean there should be no planning, just that planning should be a continual process carried out throughout the life of the program.

As new possibilities emerge.

‘Deciding as late as possible’

‘Welcoming change.’

are the Agile and Lean principles that will encourage success and reduce wasted effort.

Trying to create a static plan for a large change to the technological ecosystem of a business is neither possible nor desirable. On the other hand planning processes to:

Implement change.

Mitigate the risks of change.

Form strategies to simplify into small manageable steps.

is very possible. Enabling individuals and teams to become completely engaged in very desirable.

A very good book on Lean Enterprise suggests that Program Management being seen as “Instead of creating detailed, upfront plans on the work to be done and then breaking that work down into tiny little bits distributed to individual teams, we specify at the program level only the measurable objectives for each iteration. The teams then work out how to achieve those objectives, including collaborating with other teams and continuously integrating and testing their work to ensure they will meet the program-level objectives.”

The idea of rolling plans, continuous planning and keeping plans small to give faster throughput and faster feedback is gaining acceptance. It gives the planning process a chance to be informed by the progress of the program. It allows flexibility and allows a plan to be responsive to newly discovered requirements and changes.

Headlights

Being agile is not about being blind and not planning it is about looking down the road for potential disasters and planning the steps to avoid them. Being Lean is recognising values that can be measured and evaluated. Planning paths that can be adjusted and re-routed as the impact of implementation is assessed. Looking ahead is very valuable but the view should be as far as the path is clear.

Above all these are strategies that are in place to deal with the unknown. Every implementation of enterprise level systems is different, the outcomes are unknown but this does not mean they should be leaps into the dark. Plan the first iteration, test it, measure outcomes, learn from the test.

On the subsequent iteration apply the knowledge gleaned from the first iteration, change and improve. The first iteration may or may not need to be refactored. Either way, once completed, released and evaluated, it will cast some light on the next step. It will show if variations from the path are needed, it will illuminate potential collisions.

Instead of swallowing the elephant whole, the program should be seen as natural divisions that are interdependent and can each be controlled. Each step should independently provide value and avoid waste. If it does not provide value it can be abandoned. There is less waste in abandoning a small step than a large step or a lot of steps.

Small iterations require that less effort is needed if the route needs to be redirected. They also illuminate where the program is going and allow a redirection. Being able to see where the road map is actualy taking the program will stop it going down blind alleys or worst still going off the road.

Self-organising

Some of the tools used to manage large programs may be different to those used on smaller projects. The main aims are for simplicity, visibility and that to encourage collaboration. It is impossible for a single person to know what 100 developers are doing and what progress they are making towards the desired and result.

It is possible to know that there is interaction between 20 agile or lean teams. It is possible that communication channels exist to share knowledge and enable best practice. It is possible to see milestones being met, the road map being traversed and to view achievements via hard data and metrics. It is possible to encourage, obtain and allow responses to feedback.

I have written on the tools for agile methodology in other places but in the main they are ones that encourage responsibility, engagement, communication. The most important aspect of an agile program is to enable interaction between individuals. This interaction can be scaled to almost any size but it is not achieved by taking a small team and making it bigger.

The military have discovered optimal team sizes to be between around five t0 11 and recommend breaking large teams (7) into two.

Customer collaboration

The Agile/Lean methodology begins by scaling communication.

The instigation of a program is from the businesses managers and executives. It is they who plan change and who understand the need for improvement. They will have the clearest view of the difficulties that their existing systems are causing. They will have a vision of where the business can succeed most once the constraints of the old systems removed. They have the measures of success.

It crucial that the vision of the business and the program are aligned. This is where whoever is leading the program will establish the benefits of scaling Agile/Lean methods. The initial process involves lines of communication, in the first instance these should come from the users of the system. Finding their problems and discovering where improvements can make is the root of all specification.

Sharing the users problems with the project teams gives them an opportunity to propose solutions. This is the point when the business as a whole becomes involved, the executive can now be shown a problem, the source of the problem, a solution and an estimate of the effort that is needed to provide the solution. The business is then in possession of the information required to judge the value and form decisions based in information.

The sub projects will begin to fit together as number of simple functional features. The business may decide that a feature set would give distinct commercial value if it was changed to include different or additional features. This is a point where many technologists and engineers on the teams will groan.

Despite having subscribed to ‘welcoming changing requirements’ many project teams dislike having their work to date altered, they may see it as waste. Iterative methods allow value to be clearly communicated back to the teams. The advantages of fast delivery and quick feedback are; reduced waste and a visibility of the reasons for changing requirements.

Teams thrive on seeing improvements and delivering value. As they were involved in designing the solution and delivering an realease they are motivated. Early feedback will show them how a change in requirements improves the product and increases value. It also gives the business an early chance to evaluate the success of the program as it progresses rather than waiting for it to end.

Agile and danger

Are there dangers in using an Agile and a Lean approach on large scale programs? Any large transformation brings risks and danger but using an Agile/Lean methodology to approach big changes allows risks to become evident early on, giving opportunities to mitigate and reduce them as the projects develop.

The big danger in having a large plan is that it takes a long time to get to a point where errors and mistakes are evident. It takes a lot more effort to change the direction and it is harder to respond. The more people involved the harder it becomes. On the other hand flexible and adaptive teams who have devised their own plans can respond to both need and change.

The program manager leading the big change is not there to instruct those delivering the change. They are not there to tell the business what direction it should take. They are there to assist the teams to deliver value to the business. They are there to communicate to the business the ways in which problems can be removed and where new solutions lead to.

They discovered that the role of a leader is to serve those they lead. This is reflected in the motto of Sandhurst — ‘serve those they lead’.

This is an update of a post originally published August 10, 2014