"It is a requirement that the system..."

How many times have you heard this in your career? And of those, how many times has it been used to actually mean "shut up and do as you're told"?

Software has a very interesting property - it's soft. Flexible. Malleable. It can be dynamically adjusted to meet changing user needs, as they become apparent. Unlike a manufacturing production line for a physical product (i.e a car) that undergoes 'production' before 'delivery', software runs these concurrently, continuously.

Software will transform radically throughout its operational lifetime. Features once considered foundational may be jettisoned entirely, should things move in a different direction. So given this, to what extent can something actually be a requirement?

Definition of Done

Requirements come in at the start of the process, so let's start by looking at the end, where we consider work to be done. Using this let's tease out the tensions that exist between short-term projects and a long-term product.

Within the agile community, we talk lots about the 'definition of done'. What do you describe as 'done'? In a previous role, the team commonly referred to 'done' (devs had finished), done done (scheduled for release), and done done done (running in production). I laugh at this now, but getting teams to define done is hard. That's because done means different things to different people.

With a project-based approach of delivering requirements according to a schedule, 'done' ultimately means the project is delivered. Should the project have separate waterfall-like phases of dev > test > production, then within a given phase, done signals progression to the next phase. Realistically, this means "done... probably" because there's always a chance the work can come back for rework.

Teams going with the flow instead look to reduce these handoffs and rework exercises, which gets us to 'done' in the 'live in production' sense sooner and safer. But even with a super-agile Kanban workflow, there's still a chance work comes back - even if it's via an incident ticket from the production system!

Truthfully, we want 'done' to mean "we can work on something else". Even though a project might be done when the work is delivered, even though the team move onto the next project, the business still carries the maintenance costs of that delivered work, right up until the point it's decommissioned. Is a business truly done with work while it's providing value in a live system? Only when we die are we done with life.

Okay, this is a very broad definition of done. You're splitting hairs, Sam. This isn't useful for planning our Sprint Velocities, or for moving work to the 'done' column of the Kanban board. All true and fair enough - you need a definition of done that you can use as a nearly-always-reliable indicator that something else can be worked on now. On a separate note, when that indicator becomes unreliable, that's your signal you have a quality problem.

There's a distinction that exists between projects, with an end date, and the continued usage, maintenance and costs incurred beyond the initial scope of that project. I'm making this point because it's within this space that the usage of the word 'requirement' sits.

Requirements

We've talked about the dynamics of software, its always-changing, adapting nature. We've talked about fixed periods of 'projects' that exist within this continuum of a piece of software's lifetime. How does this all impact on software requirements?

Projects have requirements. A business expectation is set. Analysts analyse, transmorgifying these expectations into a series of requirements in a change request document. The work is estimated. The estimate is treated as a quote. Funding is sought. A code is allocated in the timesheet for an individual, named developer, to work on ticket WTF-123: blue button background MUST be changed to green button background. Very often the business context - the why - isn't included in the work ticket. Sometimes it is expressed in the Given / When / Then or As A / I Want / So That format - but even then, it's considered a given that this ticket WILL produce the desired outcome. And you as a developer will never find that out.

Is it actually a requirement of the system that this button needs to be changed from blue to green? Surely requirement means required. Are we saying that without this change the system isn't functional? Of course, this is a rather rigid and literal interpretation of the word requirement.

The honest answer is that for most of our purported requirements, not many of them are required in that sense of the word. Although it can be the case in organisations working in projects, that the requirement is actually on you, dear Developer, to do what is being asked of you.

Of course, there are regulatory requirements (and security requirements from the back of this, that incur fines and public outrage from non-compliance). But what is seen as requirements for work that delivers business value? Well, similar to a practical definition of done above, actually, these aren't requirements but guesses. The guess is "does this work meet the identified needs and produce business value"? Is the button background colour change a requirement, or a guess that it will result in more users clicking this button, thus bringing in more revenue? More radically, is the need to have a button a requirement at all, or could this be achieved more effectively through some other means? Fancy taking a punt?

Projects fix this uncertainty upfront by expressing a solution in the form of requirements. They remove all the context of why we're doing something. They remove all the creativity of how something will be done. And then the work is driven through a delivery framework. Only then can the output be inspected. If it's discovered it didn't work, start a new project I guess?

Requirements -> Hypotheses

Prof J Hackman's book "Leading Teams - Setting the Stage for Great Performances" outlines what happens when both the "ends" (the outcomes we want to achieve) and the "means" (the requirements - the way we're going to achieve those outcomes) are fixed:

We talk often about how software (both the development, and the testing) is a knowledge-based, creative activity. But in environments where both the ends and the means are specified we don't have a huge opportunity for creativity. Traditional project thinking and project management plans to turn uncertainties into certainties, then works the plan. By doing so, the work is expressed through tickets that can be completed almost by rote, through precise itemisation. The desired outcomes of the change has been almost forgotten, over the outputs considered "required" to be delivered.

What if, instead, we embraced the idea of uncertainty? What if we harnessed the creative powers of our team? What if we used time-boxed iterations to express a goal (a desired outcome) and experiment towards what delivers us value? From a costing/budget perspective this is simple - your cost is the headcount of the team. This is the power of focusing on outcomes over the outputs.

The 2020 Scrum Guide talks about a Product Goal - the intended direction, strategy or vision of your product. The aim is to give clarity to the team the purpose of the work undertaken. To achieve this goal, there's a Product Backlog - a series of steps to reaching the goal, or perhaps more powerfully, a series of hypotheses about meeting parts of the product goal, and future value that can be attained by doing so. Scrum also distinguishes between product backlog items and sprint backlog items. During Sprint Planning the developers plan the work. By expressing product backlog items in the form of hypotheses, the developers are given the freedom to innovate. Likewise, through product backlog refinement, large-scale hypotheses are broken down into sprint-sized hypotheses that can be worked on. This naturally forms the objective of a sprint - the sprint goal.

Interestingly, agile-based frameworks such as Scrum do not eliminate the planning that is present in projects. However, it moves it from an "up front" activity to a "just-in-time" activity. By deferring planning until the most-responsible moment, when the team possesses as much information as possible, they are able to better experiment with options available to work out what they should be doing. Note that at this point, requirements virtually fade into existence. Rather than dealing with requirements, the team is considering user stories, the outcomes of those user stories, why the business wants to meet them, how they will be met and, for the very mature teams on this journey, begin to understand when it will be known that these user needs are being met.

By creating a cross-functional product-aligned team, rather than ephemeral project-aligned teams, the inconsistency of formally expressing uncertainties through requirements is removed. Instead, teams focus on the true purpose of meeting user needs. Teams are empowered and trusted by respecting their autonomy, unleashing their creativity and having them own what they build.