Most organisations are blissfully unaware of the sheer amount of waste lurking within their systems. It’s staggering to think about the money they’re haemorrhaging simply because they lack a focus on engineering excellence. In the context of agile and business agility, we need to embrace a continuous mindset—continuous integration, continuous delivery, continuous feedback, and continuous testing. This dynamic nature of operations allows us to identify problems sooner, which means they’re smaller and easier to address. By doing so, we not only eliminate waste but also gather valuable data and telemetry that help us understand our processes better. This understanding is crucial for pinpointing problem areas and taking action before costs spiral out of control.
In my work with various customers, I’ve seen an alarming amount of waste. To illustrate this, I often refer to DORA metrics, which provide a clear picture of how organisations can improve their performance. Let’s break down a few key metrics that can help us understand the return on investment (ROI) for our efforts:
Innovation Rate: This metric measures the percentage of time your team spends on innovating versus maintaining existing functionality. On average, organisations see an innovation rate of around 39%. This means that only 29% of the budget is allocated to net new functionality—essentially, new capabilities that can attract new customers or open new markets. This is a dismal figure, highlighting the need for a shift in focus.
Product Index: This metric looks at how much time your engineers spend working directly on the product versus other tasks. Typically, about 80% of their time should be dedicated to product development. If we assume this holds true, we’re left with just 23 cents on the dollar in terms of potential ROI.
Usage Index: This is a critical metric that reveals what percentage of the features you build are actually used by your customers. The industry average sits at around 35%. This means that a significant portion of your development efforts may be wasted on features that users don’t engage with. It’s essential to ask your engineering teams about their usage index—understanding this can lead to more informed decisions about where to invest resources.
Version Adoption: For applications that require installation, it’s vital to know what percentage of your user base is on the latest version. The industry average is about 70%, meaning 30% of users are still on older versions. In many cases, I’ve encountered organisations where this figure is much lower, further diminishing the ROI.
When we piece these metrics together, the picture becomes quite grim. Starting with a dollar, we see that only 29 cents are spent on new functionality, which drops to 23 cents when we factor in the product index. The usage index further reduces this to a mere 8 cents on the dollar. Finally, if we consider version adoption, we might find ourselves with just 6 cents in return for every dollar spent. Is that a good investment? I think not.
So, what can we do about this? The first step is to understand your innovation rate. How much time is your team dedicating to innovation versus maintenance? Next, assess your usage index. Are you collecting the necessary telemetry to understand which features are actually being used? This data is vital for making informed decisions about where to allocate your resources.
Agility plays a crucial role in this narrative. By closing feedback loops and iterating faster, we can improve our cost-to-value ratio. While it may seem like we’re not reducing costs—since the same people are doing the same amount of work—the goal is to deliver more value for the money we’re investing.
In conclusion, it’s imperative that we take a hard look at these metrics and understand the underlying reasons for our low ROI. By focusing on innovation, understanding usage, and embracing agility, we can turn the tide and ensure that our investments yield meaningful returns. Let’s not allow waste to become the norm; instead, let’s strive for excellence in everything we do.
Most organisations don’t realise the amount of waste that they have in their system, the amount of money that they’re haemorrhaging because they don’t have a focus on engineering excellence. This idea, within the context of agile, of business agility, of everything being continuous, right? So you’ve got continuous integration, continuous delivery, continuous feedback, continuous testing. That continuous nature, that things are happening dynamically, enables you to find problems quicker. So the problems are smaller; that eliminates some waste, but it also enables you to collect data and telemetry and understand what’s going on a little bit more easily, which means that you can identify problem areas and do something about them before those costs spiral out of control.
I work with so many customers that have so much waste. I’ve got some DORA metrics here. I think not all of them are DORA; most of them are DORA metrics. We use it to tell this exact point for agile leaders in our agile leadership classes, but we talk about a number of different metrics, right? And if we’re talking about a $1, one Euro, one pound budget and how much of that, how much of that pound, dollar, Euro ends up as actual value delivered in your product, right? And what do you measure to understand your return on investment on that story?
So these are just four metrics that I’m going to use. There are many more metrics that you could be looking at that can impact on this. But one of the biggest ones is innovation rate. What percentage of your people’s time do they spend innovating rather than augmenting existing functionality or support and maintenance, right? So it’s net new capability within your product. This is really important because it’s the reason why Netflix creates a new show rather than doing the second series of that show that you liked. It’s because when you do a second series, right, you augment existing functionality. When you do a second series, your audience for the second series is going to be smaller than the first series unless there’s some kind of information that leads differently. The default is that the second series is going to have less audience, right? So it’s going to make less money than the first series.
So if your second series is going to cost just as much as a first series, wouldn’t it be better to invest that money in a first series to get a higher potential audience? It’s a bet, right? A higher potential audience return than it is to do a second series of an existing. If the existing series has enough of an audience and enough of a demand for a second series, then it’s probably worth it, right? You’ll get your return on investment. But otherwise, it’s not worth it, which is why you see the rise of limited series, right? I think we’ll see the rise of limited features and products. But innovation rate is around 39% on average. That was the last DORA metrics that I kind of looked at. So 29% of the budget goes on net new functionality—that’s new customers, new markets opening out, new capabilities for your product. That’s not a lot; that’s 29 cents on the dollar, 29 cents on the Euro, 29 pence on the pound. That’s terrible, right? And that’s just one metric.
So now we only have 29 cents of big ROI in our product or big potential ROI in our product. What about our product index? That’s what percentage of our engineers’ time do they spend, or our people’s time do they spend working on the product versus doing other stuff, right? The average is about 80% of their time they spend on the product, 20% on other stuff. So we would be down to 23 cents on the dollar. What about usage index? What percentage of the features that you build in your product are actually used by your customer, right? The industry average is about 35%. 35% of the features that you build are used by your customer. That means the rest is waste, right? It’s not really all waste because sometimes you need to build features that people don’t like to find out what they do like. So that wouldn’t be waste; that would be learning, right? So some of it’s learning. It’s difficult to quantify the learning, but certainly a huge piece of it is waste. Lots of companies build lots of features that your users have absolutely zero interest in, right? You’d be surprised. Go ask your engineering teams what their usage index is, what percentage of the features that they’ve built are used and used often by their customers.
So Microsoft has a measure that they use, which is monthly active users for each feature in your product. What are the monthly active users? If you can’t answer that question, you have no idea whatsoever whether your features are being used or not. So why are you building them? How do you know you should be adding new features to this piece of functionality, right? That augmenting existing? If we don’t even know what our user base is, why are we adding? Why are we spending money on it, right? It’s a blind bet, right? That’s a bad bet; it’s completely unhedged. But a 35% usage rate now brings us down to 8 cents on the dollar. So we went from a dollar to 28 cents on the dollar, 23 cents on the dollar, now 8 cents on the dollar, and we’re not even finished yet. Because if we’re building an application that people install, if you look at things like Windows, right, what percentage of the user base is on the latest version of the product? Now the industry average is about 70%, right? So 30% of users are on an older version of the product. I have lots of customers where that’s not even remotely true, that it’s a much smaller percentage on the latest version of your product. So that brings us down to 6 cents in the dollar.
If your return on investment for every dollar you spend, you get 6 cents, that’s 0.06% return on investment. Is that a good investment? I think it’s not a good investment. I think we need to do a whole bunch of work in understanding why that’s so low and dealing with the reasons why it’s so low. So in our example here, the biggest hit at the start was innovation rate, right? Do you even know your innovation rate? Percentage of time on innovation, percentage of time spent on augmenting existing functionality, support, and maintenance, right? What are those three percentages for your business? You should absolutely understand that for every product that you have. The next biggest thing is usage index, right? What percentage of the features are actually used by your customers? Do you understand that? Are you collecting the telemetry that you need to understand that so that you can make decisions on where you’re investing your capital and not letting costs spiral out of control?
Agility is part of this story. Agility and DevOps talk about how you can close the feedback loops, get faster at iterating, and reduce costs. No, you’re not really reducing costs because you’ve got the same people doing the same amount of work, but your cost-to-value ratio should go up, right? You should be delivering more value for the money you’re putting in, not the other way around.
