There are a variety of hoops that need to be jumped through in order to get software from someone's head to someone's screen. Hoops such as "build the thing" and "test the thing" and "sign off that the thing can be released". Historically we've fallen into the trap of making these separate activities happen at separate times - one after another. After all, how can you test what hasn't been built? From these seemingly discrete activities, organisations have built silos around each 'discipline' leading to a waterfall of work cascading between these silos, but not much getting delivered.

At the turn of the Millennium, the Agile Manifesto was published. Sure, it wasn't the first document to espouse the ideas within. It drew upon decades of practice, from the Toyota Lean Manufacturing ideals, Kent Beck's Extreme Programming and even further back, to the Scientific Method of empiricism. To me, this was akin to a modern-day Martin Luther pinning his 95 Theses to a church door. Just like those demands begun the Protestant Reformation, so did the manifesto fuel the Agility Revolution.

Old Man Yells at Cloud (from The Simpsons)

Out with waterfall, in with Agile. Capital A 'Agile', that is. One Agile please. We treat Agile like a solved problem, but it isn't). There's frameworks like Scrum and SAFe and even this Kanban thing which tell us how to build software, so in they came. Unfortunately, this doesn't solve anything by itself. When it doesn't turn out as expected it's written off as not having enough buy-in, requiring too large of a culture / mindset change. Does this ring true? Do you feel seen?

Truth be told, culture changes are a lagging, not a leading indicator. If we wait for the culture to change, we're allowing perfect to be the enemy of good. We need to be the change we want to see in the world - as only by doing this can we extend our influence outwards and concern ourselves with larger cultural shifts.

How do we go about this then? How do we do the agiles so that we can get the desired results? The sage advice is "you need to find out for yourself", because it's in the journey where the learning takes place. This of course, is why we can't simply off-the-shelf an Agile solution. It doesn't mean you can't be pointed in a starting direction though, so I shall do that.

Focus on that flow of value from idea to delivery. Make it lean.

Mapping Value Streams

Agile processes advocate the delivery of working software to users frequently in small increments. Compared to the 'big bang' of waterfall, frequent delivery reduces risk. It provides the ability for rapid experimentation. For many organisations, getting to a position of frequent delivery will require taking a holistic look at their software delivery processes and successively refining them to be lean by removing waste. Value Stream Mapping describes this process. By plotting out the organisational and developmental activities that take place to build software, we can begin to understand where waste accumulates in the process. Removing this waste makes the value stream more efficient.

More efficient value streams enable agility. Fast feedback throughout the lifecycle provides a competitive advantage by removing the time to market and creating a culture focused on its customers, rather than its own internal functional silos.

Reduce Handoffs, to Reduce Waiting

When one silo completes its piece of the puzzle - whether that's front-end development, back-end development, testing, security etc - the work needs to be handed off between these silos for it to move along the path to delivery. No individual silo is responsible for delivering value, they're each responsible for their bit. Then we hire managers to oversee this process, championing work under their stewardship over other priorities within each silo.

Let's put this in practice with an example. Consider an organisation whose work is arranged into sprints of two weeks. However, their developers and testers work in different, staggered sprints. The devs complete a sprint, build their integrated work and hand it over to the testers for a sprint of testing. Although the work is superficially taking part within an 'agile' framework - it's clearly still a waterfall process oriented around functional silos. While the testers are looking at the previous (dev) sprint's worth of features, the devs are off doing something new entirely.

This methodology seems nice if you're interested in having the people in your silo working to capacity. However, what value have the developers produced at the end of their sprint? Nothing. None of their features will be delivered to production. It doesn't matter if they're planning their sprints using story point velocity. Only functionality delivered to production can be considered 'done', and nothing's getting done if you're not aligned to that goal.

You may think it seems unfair, and it is. It's unfair to teams further to the right of the process who inherit all of this work, who operate closer to deadline day, who've lost contextual information because of the handoff between silos. They suffer the cascading snowball effect and receive all the blame when things go wrong. The handoff has introduced waste into the process, which slows us down and affects our ability to get stuff done. To see why, we need to additionally look at waiting times.

When a tester finds an issue with a given feature, it needs to be raised back on the developer queue of work to be looked at. Even if it jumps right to the front of the queue and is worked on immediately, that overall feedback took a minimum of one sprint - 2 weeks - of waiting to be received. That's because when a developer completes a feature during a sprint, it isn't looked at by a tester until the start of the next sprint. Between these two dates the work is sitting idle, making no progress towards providing value to end users.

It's clear when viewed holistically that handoffs between teams introduces waiting, which indicates process inefficiency. For instance, each silo has their own queue of work. They're not aligned on the same queue and seeing it through to production!

So an improvement to this would be to align the teams to the one value stream, working within the same sprint on the same stuff - all aligned towards a common sprint goal. By doing this, when a developer completes a feature they can get a tester to look at it immediately. Even better, they could work together! (I know, imagine that! People speaking to each other!) The handoff is reduced, the waiting times eradicated.

But what about my capacity? Won't this mean devs sitting idle towards the end of a sprint and testers sitting idle at the beginning? It might. Or it might not. It probably won't. Remember that we're not in the game of stacking each individual to capacity. We're in the game of delivering value to our users. Telling team members to stay in their lane doesn't happen when the team is aligned to the same goal. They all chip in, doing what's necessary to see the work get done. I'm sorry this doesn't align to your expectations of each person must be working on their own unique task.

A Safe Framework for Experimentation

When we achieve this continuous flow of value to users we open up a competitive advantage. We can get features to market quicker, gaining feedback sooner. We don't suffer the sunk cost of having to build the whole thing before we ship it and find out it's not what's wanted. Rather, we can perform small experiments and trim what doesn't work early, before it's broken our budgets. The flow from left-to-right then becomes a mechanism for learning. We've built ourselves a laboratory, where we can develop a hypothesis, put it to the test and evaluate the results. We're utilising the most powerful algorithm known to humanity - the Scientific Method. When we have full control of our ability to deploy to production, we can even target these experiments on a smaller user base. A/B testing lets us experiment with varying implementations. Feature toggling and gradual rollouts let us beta and battle-test functionality on real users before releasing en-masse. If this sounds exciting, it's because it is!

That's the Power of Flow

The ideals behind agility, devops, bizdevsecops (whatever we're calling it now) all point towards the same ideal. That is, to speak in Geordie, not "hoying stuff over the wall", but instead being aligned to delivering value to customers. We do this by breaking barriers, by talking to each other and by learning from each other. Maximising flow should be the top aspiration for all organisations. No longer should the 'IT Department' be following a contractor-based model, providing a fixed-scope service to the business in exchange for a fixed sum of money. Instead, the development, operations and business teams should all working together, in full alignment.