Monitoring and Measuring for Success

If you want to ensure that your products… I actually don’t like the phrase “stay within budget.” I know everybody uses it, but I don’t think that way anymore. When I’m investing in things, I think about it as I have a pool of money, and I’m going to move that pool of money around where it’s going to provide the most value. Right? So I need to be looking at… I need data, right? I need to understand what’s going on.

For example, I talk a lot about hypothesis-driven engineering practices. If we’re building brand new products or even adding capabilities to existing products, I want to have a hypothesis-driven story. That means that we’re going to have a… you know, it starts with an idea. What do we think we would like to add to our product? This could be applied to a thing on your backlog, this could be applied to the whole product, or it could be applied to a portfolio of products. Right? But what do you think you’re trying to do?

Then you would do some kind of analysis of that and figure out, well, what’s the outcome I’m trying to achieve? Right? This is what I want at the outcome. This is the thing that I’m going to do that I think will help us move towards this outcome. What’s the smallest experiment that I can run? And then I would look at how am I going to measure whether I’ve made progress towards that goal? And I’m going to regularly assess based on my progress towards that on whether I continue to invest in this thing or I don’t. This can be at the small scale or it can be at the grandiose scale—any scale you like.

So a good example is the Azure DevOps team at Microsoft. They were interested at one point in trying to answer a particular question. Right? So the head of the product, Microsoft called it the product unit manager, I think that’s their technical term, has overall budgetary control. Right? So they have all the money. They know that that means people. Right? So they have exactly 600 people—it’s not but exactly 600 people—and I need to allocate those people to the ideas that make sense in order to maximize my return on investment and keep my stakeholders happy, keep my customers happy, keep building products, and have an eye to the future on ideation and what is it that’s coming next.

So they’re allocating people’s time as the biggest expense. Right? So they had this idea brought to them that we’d like to help… this is the idea: we’d like to help customers discover and deal with their technical debt. Right? All they’re building software that helps build software. So we’re all building software, and we have technical debt. So it would be great if we could understand what technical debt we have and then have things we can do to help us—things that help us minimize that technical debt. So that’s a grandiose idea; it’s a big idea that goes across the entire product suite.

You know, I’m going to allocate a bunch of time. So the idea is that I, as the product owner—the Uber product owner—there’s lots of people in charge of different parts of the product, but the Uber product owner, product unit manager, decides we’re going to try this thing. I’d like to see some experiments within the context of this story, and you folks figure out what it is you’re going to do.

So his lieutenants, who run the different parts of the product, come up with ideas on how to collaborate together. What are we going to do? What are the ideas we think we can have? They come up with some ideas, and they bring that, saying, “This is what we’re thinking. This is what we want to try.” Okay, cool, let’s try this one. This looks pretty good, and they try it. This could involve 100 people spending some amount of time within a quarter, within a year, within a fixed time scale to try and create some features within this context. Right?

So when we’re spending time on this experiment, that’s money not being spent on other things that we know are valuable. We’re speculating on something that we think might be valuable in the future, right? That we might be able to do something with. So they spent a bunch of time on it. I think they spent about six months—four to six months on it, if I’m remembering rightly—and they come back and they’re like, “We don’t understand how to solve this problem.” The things that they’d created, they’d managed to create some things, they’d got them in front of real customers—like myself, I’m their customer—got them in front of real customers, and they just didn’t resonate. They just didn’t provide value.

They tried a whole bunch of different ideas. There were five or six different teams, different groups involved in it. They all had different ideas; they all collaborated together to create some of these ideas, and nothing resonated with the customers. So at some point, the product unit manager is like, “Right, let’s not invest any more money in this because we’re not… we’ve not got an idea that’s good enough to move the needle or solve this problem.”

You could say that that amount of money—maybe they invested 10 million in this capability—was wasted money. You could say it was wasted money, but you could also say that there was a lot of learning involved. They learned not to go tackle that problem. They learned to save their money and spend it on something else. So in the traditional world, they would have spent years on that capability to find out that it was a bad idea. Right? Think of Windows 8. No, not Windows XP—Windows 8, sorry, my brain died there. The Windows 8 debacle, where Microsoft spent six years. They had 4,500 software engineers working on Windows. They had 15,000 other people involved in that story. So you’re talking nearly 20,000 people working for six years. Imagine the cost in your budget for 20,000 people working for six years on a product, and the end outcome did not meet the expectations of the customer, did not meet the brand value, right? In fact, it detracted from the brand value, detracted from the customer experience, and you had a lot of unhappy customers that went to other products like Apple. Right? That was a big jump ship moment that people moved to other platforms.

This fundamentally was the catalyst for Microsoft looking at themselves and kind of going, “Yeah, we’re doing this wrong.” This long cycle, long budgetary five-year plan, right? And not getting things very quickly in front of customers, very quickly testing and validating whether we’re doing the right thing. That’s what we need to be doing, and we need to be doing it on every product, everywhere in the business, at every scale possible—from the lowest teams working on products, hypothesis-driven engineering practices, to the board making decisions about which products and capabilities to fund and which products and capabilities to not. Hypothesis-driven engineering practices—you need to be doing it at every level of your business.