The Power of Technical Excellence in Agile Development

Technical excellence is one of the most important concepts when you’re building products. Scrum talks about this in that you should always have a usable working product at the end of every iteration. Agile talks about this. The idea is that in order to mitigate risk, we need to create a usable working product.

In the more linear model, where we know lots of stuff up front, let’s say we were going to build 300 houses on a construction site. We know a lot up front; we know exactly what designs of the houses we’re going to have. We’ve probably had to get planning permission, right? So a lot of that is locked in for what it is we’re going to do. We know how much materials, how much wood, how much plaster. We know when we’re going to do certain things, or mostly when, right? There’s some variant, but not a lot of variance. So when there’s low variance, we mitigate risk by doing more planning. We plan up front, we create a bunch of documents around that planning. We create a list of risks that might happen. If this risk happens, here’s the mitigating action we’re going to have. Here’s the percentage likelihood that that risk is going to happen. We’ll maintain that data over time.

We create this whole body of material and planning and dates and things around that story, and that’s great when variance is low. That allows us to mitigate the risk because if you’re going to spend, I don’t know, £20 million building this area of houses, then you need to plan for that. But even then, you’re going to take a little bit of an agile approach, right? You’re not going to build all the houses at once and then sell them all at once. You’re going to build a few houses and sell them, and a few houses and sell them because you’re testing the market and you’re seeing. You might then even need to apply for different planning permission to change your mind. So there’s still a lot of variance in that story.

I do get that for those that are in that industry, but in general, old school, you’re kind of making those decisions up front. But when we have high variance, which, as I just noted, can happen in the building industry as well, we can’t mitigate risk by writing everything down and creating a story for every single thing that might maybe happen because every single thing that might maybe happen is going to change every day. Think of the multiverse, right? You’ve got all those decision points that happen, and every time you make a decision and go down a new path, a whole different set of decision points and problem points and issue points occur. So every time you make one of those decisions, every time you make one of those little changes, you would need to rebuild all of that body of asset, and it’s just not reasonable to do that. It puts too much of an overhead on people, time, everything, and you end up with that story of the plan not being able to keep up with the reality of the world. Therefore, we then make bad decisions because we can’t really see what’s going on, and so we’re not mitigating risk at all.

But in that high variance world, we mitigate risk by creating a usable working product on a regular cadence. That’s how we mitigate the risk. We deliver something; we create that thing that the customer is looking for, and we offer it to them for inspection so they can look at it and see, “Yeah, yeah, yeah, you’re doing the right thing,” or, “Oh, I don’t understand why you made those decisions.” Great, that’s a great conversation.

In order to be able to have a usable working product at the end of every iteration, I do want to point out this is not a pipe dream; this is not an ideal world thing. I want you to all go look up the US Department of Defense article on detecting agile. It was an article, a paper that was sent out to every procurement officer in the Air Force to let them know what they need to look for when they’re working with vendors, right? When they’re buying stuff from vendors so that they can make better purchasing decisions. This vendor is telling me that they’re agile. How do I know that they’re agile? I’m a layman; I don’t necessarily understand agile. I don’t necessarily understand that new world. How do I ensure that I’m buying the right thing? This article lays out a whole bunch of questions that enable that person to at least start getting some insight.

One of the key questions, and this is like the first one for leadership, was, “Do you have a working version of your product every iteration, including the first, that gets into the hands of real users?” Real users, not UAT, not fake users, not test environments, real users every iteration, including the first. So if you’re doing two-week iterations, then at the end of the first two weeks, you should have a usable working product. That’s how we mitigate risk in the agile and scrum space. That’s how we ensure that when people give us money, we’re actually delivering something that will be valuable for them.

That’s really hard if our software, our product, the quality of our product is terrible. Can you imagine trying, even if you were building houses, right? And there are broken joists, there are things not attached properly, there are leaky pipes, there’s flooding. We’ve not built the house properly, and we’re trying to hand that house over to the customer and have the customer come in and look at it, and the customer’s like, “Well, this is terrible; I’m not going to buy this.” That’s where engineering excellence comes in; that’s where technical excellence comes in. We want to build things that actually work. We want to deliver less stuff up front, right? In the fictional, you know, that old phrase, “80% of the work takes 20% of the time, and the last 20% takes 80% of the time.” That is because our quality was bad; that’s because we didn’t have technical excellence in what it was that we were doing.

So we got that 80% of the functionality we’ve done; it looks like it works, but actually, behind the scenes, it’s a total mess, spaghetti, absolute garbage. The customer comes along and tries to use it, and they’re like, “Oh, we need to fix this; we need to fix that,” and you end up patching, right? You’re getting out the sticky tape and super glue and trying to figure out how do we patch this together. Then they’re like, “Oh, I don’t like the way this works.” You can’t rebuild it because it’s part of the fundamental architecture, so you have to add another layer of obfuscation on top to make it look like it works the way the customer wanted it to work. But actually, behind the scenes, it just works the way the developers originally built it to work, and now we’ve got extra complexity. We’ve got something that is more difficult to maintain, more difficult to understand, and you can imagine that over time that builds up on top, and you end up with massive amounts of layers that inhibit your ability to deliver value.

That’s something that I would quantify with some data. I do a lot of presentations on how the Azure DevOps team, that’s the team at Microsoft that built the Azure DevOps product, they, in 2012, were technically a waterfall team. They were delivering once every two years to production, and they moved almost overnight towards continuous delivery and three-week iterations, right? So they were delivering usable working product every three weeks. They were shipping to production every three weeks, and they’ve delivered every three weeks for the past, I don’t know, 26 three-week sprints.

Back in 2012, with 650 people, they were delivering around 24 features to production each year. Part of the reason that that number is quite low, 24 features, is that one, the features were really big, but also the difficulty in adding the features and interacting with the codebase in order to get the product to do the things that you wanted it to do was really difficult because you had all those little layers of things that had built on top. This doesn’t quite work the way it was supposed to. I think Azure DevOps was originally Team Foundation Server, if you remember those days, and Team Foundation Server was originally a bunch of other products that were native inside of Microsoft. They had source control in-house, source control products; they had in-house work item tracking products; they had in-house other stuff, and that was kind of cobbled together in order to create the first version of Team Foundation Server.

Now, I’m sure lots of stuff was rewritten, but you’ve got that fundamental architectural mindset, right? Even though you’re rewriting something, you’re taking something else. So actually, the genetics of that product is quite long, quite old. Then you get to that point where you’re switching from needing two years to get a piece of functionality working to needing three weeks to get a piece of functionality working. So they moved to this new model, and in 2012, they were delivering about 24 features to production. They then focused absolutely the majority of their time—maybe going majority is probably too much; their primary focus was on paying back technical debt, building in technical excellence, solving those effectiveness problems of moving to that new model, both in their process and in the product as well. By 2018, they were delivering 280 features to production each year with the same number of people.

So from 24 features to production, 24 experiments, 24 new features to try and see if the customer likes to 280 new features, the customer experiments to try things that might work and might help the customer. Now, yes, those things were smaller, right? But they were also delivered faster. That focus on technical excellence enables us to do that. If we don’t have technical excellence in the way we do things, then it just doesn’t work.

I did a recent interview with Dan Helm, who runs Azure Boards, the work item tracking part of the system. When he was quizzed on the definition of done and how that works, he didn’t really have a solid answer, and that’s because technical excellence is so baked into the way they do everything at that level. All the engineers know what it is they need to do, what level of quality is required. They know all of those things; they don’t need to focus on it, right? It’s just the way they do business. Technical excellence, engineering quality is just the norm; it’s not an exception, which means that they can then focus on delivering value to the customer. They’ve reduced risk, and they can focus on delivering value. They can focus on those other things. That’s what technical excellence brings for product management, for product leadership, is the ability for them to stop focusing on fighting fires, on fixing bugs, on struggling with complexity, on being frustrated that they can’t get certain features done because it’s just too hard to do within the architecture of the product. You start focusing on the art of the possible: what is it we can do? Where are we going? What are we trying to achieve? You can then let go of those things because you’re no longer having to ride that quality.

That’s a terrible analogy, but you’re no longer having to hold on for dear life while this product bucks around. You’re now in the groove; it’s smooth. We’re adding features, we’re building capabilities, we’re building an awesome product. That’s part of that story of mitigating risk, building technical excellence. There’s a myth as well; there’s a huge myth in the software industry that it’s okay to sacrifice some quality in order to go faster and meet the market need. There’s a recent article, I think it was a Harvard Business Review article; it might not have been, but it might have been a Forbes article. There are some studies done around that idea of what happens when we cut quality in order to go faster, right? And what’s the trade-offs involved in that? The ultimate outcome was that it’s not a valid trade-off. The impact of that reduction in quality is much higher than the benefit you get from being able to go fast over the short term.

So that a cruel, let’s call it two things: there’s undone work, stuff that you need to do in order for your product to run in production that hasn’t been done yet, right? That’s that 80/20 thing; that’s where that pit comes from. Then there’s technical debt, which is shortcuts, deliberate shortcuts that we make in order to do things in a suboptimal way within the product. That’s technical debt, right? Both of those things added together don’t scale linearly; it’s an exponential curve. That’s where that Azure DevOps team was at the start. They were at the top of that exponential curve, only delivering 24 features to production each year. But in focusing almost religiously, zealot-like, on paying back that technical debt, getting rid of all of that undone work, and bringing that curve back down to a manageable level, right? And beyond, getting it towards zero. We want to have as high quality, effective a product as possible. They actually, in one year, nearly tripled the number of features delivered to production, even though their main focus was on, “Let’s do it right; let’s pay back the technical debt.” They actually delivered more stuff.

So that goes: reduced quality to go fast is a losing proposition. It always has been a losing proposition. Definitely, what you want to do is focus on technical excellence. If you don’t have a definition of done, create one. If your organisation doesn’t have a definition of done, which it should, your organisation should be thinking about, from a technical excellence perspective, what is the minimum level of quality we as a business require in order to have a product that is fit for purpose? Right? Protects our brand, our business as a whole. This product is representing our brand or however that works, representing your brand. Protecting the people that are using it and protecting the people, your employees and your developers, right? To make sure that everything’s done right. You don’t want, like, think about Boeing, right? The reduction in quality is just now seeping out. What else do you need in order to have a viable product in your market, in your situation, right? So different levels for different markets; that’s the business’s requirement for done.

We’re not talking about features. There’s a big conflation, I think, between features being feature complete and quality. Engineering excellence is about quality; it’s not about having the features, right? So let’s take that, put that aside. What we’re building, we’re going to take that aside, and we’re going to focus on the how we’re building it. Do we have the level of quality required to meet compliance? Do we have the level of quality required to meet the testing level that we want? Do we have the amount of telemetry in there to see what’s going on? All of those things are part of that definition of done, which has nothing to do with the features. We’re only focusing on the quality of the product, and that enshrines your engineering excellence.

So if the organisation doesn’t have that standard, it’s the developers, the scrum team, or the developers that are charged with creating one that is relevant for your product and significant enough that you’re going to be able to create a usable working product every sprint. The reason you have a more usable working product is to mitigate that risk and build out your engineering excellence. So don’t get left behind; don’t get bogged down in the quagmire of engineering inferiority. Focus on engineering excellence. Build the best product you can and maximise the value to your customers.