Software, Skyscrapers, and Scrum

Imagine I’ve just handed you $200 million and asked you to build a skyscraper. How would you do it? You’d probably come up with a plan that looked something like this:

1. Initiate. Discuss the concept with an architect.
2. Plan. Select a site and spend a year drafting and refining the plans.
3. Execute & Control. Spend a year and a half building the skyscraper according to specifications.
4. Close. Hand off the property to a management company.
5. Profit!

Generally speaking, this would be a good strategy — skyscrapers are successfully built like this all the time. So how would the process change if I asked you instead to build an iPhone app that aggregates news based on your friend’s recommendations?

A common-sense approach would be to apply the same methodology: Initiate, plan your requirements in advance, execute, and close. But it turns out, despite being the prevailing methodology of the first fifty years of software development, that process would run a huge risk of failure. We know this because the history of large I.T. projects conducted in this way is littered with failures. To take just one example, in 1995 the U.S. Department of Defense reported just 2% of its $35 billion I.T. budget delivered working software. Two percent!

So why does the common-sense approach fail? To answer that, we need to understand why building software is different than building skyscrapers.

Feedback Sensitivity

Suppose you’re halfway through building your skyscraper when I come to you and say “You know, I’ve had a change of heart — I’d like a bridge instead. Can you make a bridge out of this half-skyscraper?”

As absurd as that sounds in the brick-and-mortar world, it happens all the time in software. The reason is that for skyscrapers, the requirements for a viable, marketable product are known in advance and remain relatively stable throughout the project: spaces to live and work, plumbing, structural integrity, fire protection, and so on. But for a viable, marketable software product, the requirements change frequently as we learn new information. The original requirements are often just guesses — radically wrong guesses.

Consider our news app. We’ve decided to aggregate news user’s friends recommend to generate a more interesting news feed than, say, browsing the New York Times. While that’s a neatly packaged idea, after launching the product we could find out, for example, that customers find drastically more value in news recommended by thought leaders. To accommodate this change, we would need to do the software equivalent of making a bridge out of a skyscraper, meaning we’d throw away months of development and tens of thousands of dollars.

As bad as that is, there’s even worse news. Software projects aren’t just awash in faulty assumptions about the product’s viability — they’re often plagued by internal misalignment. Our designers, for example, might craft an elegant interface based around the idea of a real-time news feed. But when that design is handed off to our developers, they may reject it on the grounds that the system architecture won’t have the capacity to support it. Or even worse, the developers might implement it and only during testing does QA discover the architecture can’t handle the load. Either way, we’ve burned thousands yet again.

Building software, unlike building skyscrapers, is highly feedback-sensitive work. Development teams must be able to turn on a dime and yet deliver complex, interlocking modules containing thousands of lines of code. To be successful, digital project managers have to take an approach fundamentally different from the traditional project management scheme, and Agile is one such approach.

Agile

At its core, Agile is about increasing transparency in order to expose faulty assumptions and misalignment more quickly. To mitigate faulty product assumptions, agile teams gather regular stakeholder feedback on little working bits of a product rather than waiting to launch the entire thing. So instead of spending twelve months to plan and build our full-featured news app, we could spend one month planning and developing a basic version and then hand that to users. We’d discover our faulty assumption about user’s preferences eleven months and thousands of dollars sooner.

To mitigate internal misalignment, Agile demands collaboration sooner rather than later. Traditionally, product work “flows” from one skill area to the next: business analysts suss out and define requirements, designers create a solution, developers implement it, and QA tests it. Agile upends this “waterfall” approach and replaces it with a roughly simultaneous approach, wherein contributors in each skill area perform their work at the same time. In the month we’d spend building a basic version of the app, instead of having one week for design, two weeks for development, and one week for QA, we’d start all three at once. With the increased collaboration required to do that, we’d be forced to shift into better alignment about the impossible design much sooner.

Sounds good, right? So how do you implement it?

Scrum

Scrum is a system of teamwork based on the insights of Agile. It starts with the idea that in order to get internal and external feedback sooner, a product team should collaborate regularly to release working, self-contained chunks of a product. To do that, you need three things: transparency, adaptation, and regularity. I’ll cover each of these briefly.

Transparency. Scrum teams begin their day with a fifteen-minute update meeting, often called a “standup,” in which team members report to each other what they were working on yesterday, what they’re working on that day, and if they have any roadblocks. Tasks have physical representations as sticky notes on a Scrum board, which has three columns: To Do, Doing, and Done. The Scrum board is updated continuously as tasks move from one category to the next.

Taken together, the standup and the Scrum board have two positive effects: first, they keep everyone naturally accountable, and second, they encourage spontaneous collaboration which brings internal planning into better alignment.

Another pillar of transparency in Scrum is regular demonstrations to key stakeholders. When working bits of the software are completed, the development team shows them off so that stakeholders can be abreast of developments and also change directions if necessary. Scrum can handle these changes because the process is designed to be adaptive.

Adaptation. Instead of defining the requirements for a product up-front, Scrum recommends roughly outlining planned features and then fully defining the requirements in person with the team at the time of implementation. This means developers don’t wait months for a comprehensive set of requirements, when in reality many of the requirements may never be executed on due either to faulty product assumptions or internal misalignment, as in the examples above. On this model, even late changes can be accommodated.

Adaptation is important not only for the work of product development itself, but also for the processes a team uses to do that work. Scrum prescribes meetings called “retrospectives,” wherein the entire team honestly evaluates what’s working and what isn’t, and proposes improvements. These process improvements are implemented by the team during the next “sprint,” which brings us to regularity.

Regularity. Scrum recommends the idea of a cadence, or a regular rhythm, to work. The calendar is broken into regular chunks called “sprints,” often two or four weeks. At the beginning of each sprint, the team (not a manager) decides which features it can complete, and commits to finishing them in that time window. The list of features the team chooses from are pre-prioritized by a product manager, called the “Product Owner” in Scrum terminology, so that the most important features get done first.

By allowing team members to select from a pre-prioritized list of features, you grant autonomy and a sense of ownership to your team without sacrificing control of product development. And by breaking up work into sprints with a stated end date, your team becomes progressively better at estimating how much work they can get done, giving management more realistic estimates.

Putting it all Together

When all three of these principles are taken to heart by management and the development team, a high-functioning team takes shape. At Upspring, our Scrum implementation began with the basics in each of these categories: transparency (a scrum board and morning meetings), adaptation (retrospectives), and regularity (breaking work into sprints). With these fundamentals in place, we’ve continued to enhance our processes, because no system of teamwork is ever perfect.

Which brings us to a principle of Scrum I’ve left out: the value of people over processes. While that may sound paradoxical given what I’ve covered above is mostly about processes, Scrum is ultimately about helping people work together well. In the feedback-sensitive discipline of software, Agile and Scrum recognize that the common-sense approach to working together doesn’t deliver. Scrum gives teams the tools to succeed in this environment. If it’s forced onto a team, you can expect massive pushback and failure. But if it’s built from the ground up as a way to work smarter and happier, you might just be able to turn a skyscraper into a bridge.

Originally posted at my company’s blog.