Measuring and Monitoring Costs for Continuous Improvement

So if you’re building products, you really need to understand the cost of those products. There are a number of factors; it’s really complicated because there’s the actual cost you pay, which is normally due. In general, your two big costs at building software products are hardware—that’s the smaller one—and people. Right? So if you have 50 people working on a product, that’s probably going to be your biggest cost. If you deploy to the cloud, that’s going to be your second biggest cost, right? So your infrastructure and people, those costs should be fairly consistent. You should understand your cost profile. I usually talk about it in terms of what’s the cost per sprint, right? Why should we do another sprint? What’s it going to cost? What do we get for it? And are we getting a clear return on investment for the cost that we’re putting in, right?

So sprints are part of the Scrum framework, and they’re a way of planning and managing work, right? They don’t imply an engineering process; it’s just about the management of the work part and planning for what’s happening next. So if I was looking at a cost per sprint, let’s say I’m doing a two-week sprint, what’s my cost per two weeks of work? What are my teams working on for those two weeks, and is it valuable? It’s also really important in that context for the people doing the work to understand the cost of doing the work because they’ll make better choices, right? If you want to create an environment in which people are spending the money as if it’s their money, and if they understand the cost of the sprint, they’ll think about what it is that we’re doing for this sprint, the cost of the sprint, and try and maximise your return on investment. That needs to be an evolution, an effort that every single person that’s working on your product is involved in; otherwise, you’re not going to have people working on the right thing, and you’re not going to have people making the right decisions.

So one effective way to minimise your cost and maximise your value is to have every member of every team and everybody working on your product understand what the costs are within their context. I’ve actually spoken to some colleagues about this, and I’ve kind of come to the conclusion that every team should be running a P&L, right? Not just are we adding value in the product, because value is a little bit in the eye of the beholder; it can be a little bit difficult to actually measure its capability. But revenue, we can absolutely measure that, and what percentage of the revenue is attributed to a particular feature. This team owns that feature; therefore, that’s the revenue they’re making. This is the cost that they’re having, and are we as a team making a profit working on what we’re working on? If we’re not making a profit working on what we’re working on, should we be working on something else, right?

That dynamic and flow of the team within the context of product management and product leadership with deliberate choices on investment in long-term efforts, right? Just like an infrastructure build. If you were running a country and you decided to invest in roads, that’s a big capital expenditure to invest, and that needs to be understood as a concept. We’re investing a bunch of money just now, so it looks like we’re making a loss in order to get a larger return in the future, and that’s a deliberate thing that we’re doing. But we’re still looking at the P&L; we’re still understanding what it is that we’re doing.

So I think from a team, from a group, from a product perspective, we should be looking at all of that information, and every member of every team should understand it to the extent that they need to, right? Within their context. To do that, you kind of need to understand a little bit more about what your product is, where you intend your product to go, what market you’re in, what’s happening in the market, and how you’re interacting with the market. This is where I tend to fall back a little bit on evidence-based management. So there’s the evidence-based management guide from Scrum.org, and it does give example metrics, right? But it doesn’t really talk about specific metrics in the context of the guide. It basically says there are four key value areas that we need to think about. There are four key questions we need to ask ourselves, and in order to answer those questions, we need to have metrics which help inform but not control those decisions.

So the four key value areas that we need to monitor, right, in order to understand our cost-to-value profile, I think that’s probably a good way to say it, is we need to understand where we sit in the market for our product. That market could be an external B2C or B2B market, or it could be an internal market—whatever you want to call that. You’re the people that are using your product internally in the organisation. That’s why lots of organisations have internal cost structures as well. You know, I worked at Merrill Lynch, and if I wanted a server or to use a piece of software, even if it was an internally built piece of software, there was a cost attributed to it. Yes, it was a little bit of a fictitious internal cost, but it allowed them to do a little bit more P&L and understand what’s going on with the product. They definitely went too far with bean counting, but it is important.

So we have to understand our market value, and that’s made up of two areas. There’s current market value—what we’ve got right now, what we’re delivering right now. So that could be customer happiness; it could be employee happiness; it could be usage, right? Lots of information we can collect around our current value. And then we’ve got hypothetical value, future value, which should be represented in our product backlog, right? What’s the future value of our product? Looking at what market opportunities we can open and how do we measure whether we’re opening enough market opportunities?

Think of the example of Netflix. Why do they choose to create a new show, the first season of a new show, rather than add a second season to an already successful show? That’s because second seasons always have less users; less people care about a second season, whereas the first season of a new show, you can big it up and onboard new people that weren’t customers before, right? So that’s opening out new market opportunities rather than leveraging existing market opportunities, and that’s what your product backlog is about, right? You need to measure and understand those things within that context, and those things are really, really hard to measure. That’s one of the hardest parts, I think, to measure for most organisations. Are we solving customers’ problems? Are there other problems that the customers have that we should be solving instead? All of those kinds of questions.

But that’s your market value. And then the other thing that you need to understand and have metrics for—I’m deliberately not referring to specific metrics because it really depends on your context, your industry, what it is you’re doing, how your product works—you need to find metrics in these areas that suit your particular product in your particular world. I do work with organisations to help them understand those things, but it’s something that they need to understand, right? The people doing the work, the people in the organisation need to understand.

So that’s the market value, and then you’ve also got your capability of your organisation. So the two pieces are time to market—how quickly do you get your product in front of real customers, right? Time to market is about how quickly can you realise an idea to data, right? So that’s time to market. And then the other one is the ability to innovate. How much time do you spend opening those new markets versus maintaining and supporting existing stuff, right? This is where if you had high technical debt, you might spend more time on maintaining and supporting. If you have low product quality, you might have more time on supporting and maintaining.

So those are the types of things you might measure in that world, right? What’s our technical debt? We might do so in our code analysis of our product or whatever else you’re doing. How do we understand bug trends, live site incidents? We might be looking at all of those things within our ability to innovate context and our time to market. We’re looking at cycle time and lead time, right? How quickly can we get stuff out the door? What are other things in your organisation that impact those ideas, right? What are the things that inhibit your organisation’s ability to reduce its time to market? Measure those, right? To get faster at delivering your products, what are the things that are getting in your way? That’s what you should measure. If you’re struggling to innovate, what are the things that are getting in your way? Those are the things you’d measure, right?

And once you’ve got a handle on those things, figure out what the next thing you measure is to get your next biggest bang for your buck. And that’s a constant cycle. So the metrics are changing over time. Some of them will take longer than others to resolve, but measuring and monitoring the value that you’re creating—revenue in P&L and the cost of delivering in your product—these are other metrics that help you get that holistic picture of what’s going on. It is almost the most important thing for a CTO, for a business, to understand and disseminate through the organisation so that we can all, at every level, make the best decision we can.