Avoid Local Optimization in Agile Transformation

The benefits of agile transformation can be elusive if you don’t take the big picture view and understand how value is created in your organization. Organizations that start by applying Scrum in the application development area may be fooling themselves into thinking they’ve really improved. What they’ve actually done is local optimization. Let’s explore this further.

[Check out this related article – How Do You Measure the Effectiveness of Agile Approaches]

Water-Scrummer-Fall

I had heard the terms Scrum-Fall and Water-Scrum before, but it was Jez Humble that introduced me to Water-Scrum-Fall in the video of his talk at GOTO 2015, Why Scaling Agile Doesn’t Work.  It is a great video, go ahead and watch, we will wait here. (BTW, Jez Humble was one of 4 authors to the Dev Ops Handbook, a must read for serious agilists!)

What Jez explained in the video was that many organizations apply agile and Scrum to the application development portion of the work. Sure they improve application development. But they ignore the months of work before application development when the business need or project is being analyzed, dissected, “committeed”, justified, budgeted, estimated, beat around with other projects, approved, designed, architected, and a bunch of other non-value add activities. People are doing work, but the work is not work that a customer would appreciate or value. In lean terms it is non-value add work, or waste.

Similarly, many agile transformations ignore what happens after application development to actually get their valuable solutions into production. It can take months to push finished features through long test cycles or release processes. This often extend the benefits realization by 3 to 6 months.

Similarly, Jonathan Smart of Barclays describes a remarkably similar process experienced at Barclays in his video called The PMO is Dead, Long Live the PMO. In addition to the things that Jez mentioned, Smart and co-presenter Morag McCall called out the impact of annual budget cycles that worsened the problem.

The point that both Jez and Jonathan make is that no matter how fast the middle part (app dev) goes with agile methods, the real delay and waste is elsewhere. You can only optimize your so much by focusing on the app development portion of the value stream. Implementing Scrum in the middle of a waterfall process will only take you so far. Implementing Agile within one part of their value stream won’t make much impact on the overall delivery time. Or business agility for that matter. The promise of agile transformation fails to deliver.

Understanding Your Value Stream

Smart posted the notes from his presentation online. Those notes showed a hypothetical map of the value stream, with all the various steps and the various wait times.

As you can see in the diagram, there are a lot of steps in the process. And time. While our attention might naturally go to the application development step, this approach ignores the biggest opportunities for improvement.

The best way to illustrate this for yourself is to create a value stream map for your process. Take a specific example of an idea that is hatched or a business need identified, and then follow that idea all the way through to production.

I spent some time last year with a client to map their value stream from the recognition of a great idea to delivery of that idea. This particular client had not introduced Agile yet so it was a good baseline for the current state. For this particular client, it took on average from 17 to 24 months to go from identification of need to delivery of solution.

The Agile Transformation Needs to Consider the Entire Value Stream

So many of the Agile Transformations I have observed made the mistake of focusing only on the middle part and ignoring upfront and downstream time and effort. The “benefits” gained in that middle part are partially undermined by long lead times and wasteful work before or after. This represents local optimization, something most agilists are trained to avoid.

A true agile transformation needs to look at that entire value stream. But this represents a huge challenge. Organizational change is extremely difficult and disruptive. Just implementing agile into the App Dev area is risky enough. At least the App Dev team is organized under one leader. Expanding the focus to the entire value stream is going to require the involvement, buy in and leadership of multiple leaders across different silos, which can be difficult.

So Where do you Start Your Agile Transformation?

Except in very small organizations, it is daunting to imagine changing the entire value stream. So where do you start? It seems to me that there are three places to start based on chunking the value stream into intake, development and deployment:

1. Start with intake including Project Initiation, Prioritization and Requirements
2. Start with development including Application Development and Testing
3. Start with deployment including Systems/Integration Testing, Regression Test, Release to Production

Let’s take a look at the merits of each of these approaches:

#1 Start Agile Transformation with Intake

I have a client that started upstream of IT. And at first that really concerned me.

If you fix the left, you will only optimize the front end and begin pushing more work to the right. This was one of the key anti-patterns identified by Jonathan at Barclays. They created a bottleneck much like Lucy and Ethel faced in the classic Chocolate Factory Episode.

By optimizing the front end or the intake process, you will create a big queue in front of application development. The system will be unbalanced, and work will being pushed rather than pulled. Rather than creating a flow, they will re-create big batches.

The downstream team will be a bottleneck. Like Lucy and Ethel, app dev will be under pressure and scrambling. Quality and sustainable pace will suffer.

In addition, your teams upstream teams will begin to see that nothing is happening. They will likely get discouraged and disband their efforts.

#2 Start with Development

As noted previously, most Agile Transformations start with app development, either because of the money spent there or the perception of it being a bottleneck.

The befit of starting with app dev is that it is usually within the same org snd leadership structure. Also, quick wins can be used to create momentum for changes elsewhere. For example, it may be easy to justify automation to the test and release processes or a complete dev ops implementation.

The main downside of starting with application development, as noted above, is local optimization and little improvement in overall business agility.

#3 Start with Deployment

The third approach would be to start on the right with those processes that test and deploy. By automating and optimizing, and removing batches, you can begin “pulling” work from the left.

As you address the issues with deployment, you begin moving left to application development. Like a bucket brigade, they all hand work directly to the next person in line with no queue. By implementing feedback loops, everyone in the chain recognizes how their work fits in to the overall value chain. This is the essence of single piece flow.

Maybe It is Ideal to Start Your Agile Transformation Here

The downside of all three of the previous approaches is that they don’t necessarily re-engineer the value stream. We could simply be automating processes as the exist and not thinking about whether they are the right processes.

Additionally, the underlying organization structure that supports (and enforces) that value Stream remains in place, fostering silos and local optimization.

There is a fourth approach, to design an ideal value stream and setup an org structure and processes to support it. Rather than propagating all the existing steps which are typically a function of the existing departments, create the flow first. Then design the organization around it.

For example, the silos and handoffs from intake to development to deployment may not make sense or be necessary. Can we create one organization, one team even, that would handle things all the way from need to delivery? Small organizations and startups do this all the time. They don’t have the luxury to create silos with limited spans of control and responsibility.

The challenge is that this type of change is extremely disruptive and would need to be driven from the top of the organization.

I am interested in your thoughts on this topic. What have you seen?