Mastering Evidence-Based Management for Agile Success

With evidence-based management, it’s really important to gather the data, the evidence that you’re going to use to help you in your decision-making, to help inform your decision-making. So we need to gather a bunch of data, and then we need to be able to analyze that data. There are lots of different places you can get your data from, depending on what type of data it is that you need.

If you remember, there are four key value areas in evidence-based management that we want to kind of try and make sure that we have at least something in each of those areas. In each of those key value areas, we want to find what it is we want to collect and then figure out how to collect it. Then we have to actually collect it, and then collecting it’s not good enough; we have to actually analyze it.

There are some areas that are easier than others. In the organizational capability, the two key value areas that are part of organizational capability are time to market and ability to innovate. There’s a plethora of data that you can collect within the context of your engineering teams. My focus tends to be around software engineering, so I might collect things like how much technical debt do I have, how many bugs do I have over time, right? So the trend of bugs. I might be collecting, “Oh, there’s loads of data I can collect.”

I could collect code coverage or the trend of code coverage over time; that might be valuable at some point. There are hundreds of different data points that we can collect within that context. A few that might make sense: if you’re in the ability to innovate, you’ll probably find a lot of metrics coming from D. If you look up the DORA metrics, there’s lots of great stuff in there.

There are things like defect trends, but also innovation rate is really important. What percentage of time do you spend innovating versus doing all the other stuff that you need to do? Collecting things like technical debt, some of these things are easy to collect. Innovation rate is fairly easy; you just categorize the work that you’re doing as net new work, augmentations to existing functionality, and then support and maintenance work, whatever you want to call those categories.

Then you graph that over time, right? So you’ll be able to see what’s going on there, and you can then make decisions based on the amount of money that you’re investing in each of those areas. Do you want to change how your investment strategy works? Do you want to spend more time on innovation, or do you need to spend more time on augmenting your existing functionality? That might depend on some of the other metrics from the market value side of the story.

In market value, we had current value, so what our product currently does, and unrealized value, which is the things that our product might do that it doesn’t do yet. This would ultimately result in your product backlog, and this is your actual increment, your actual product. If we were looking at data for your actual product, we might be looking at something like customer satisfaction. How satisfied are the users of your product with your product? How satisfied are they with the features? Do they meet their needs?

There’s a bunch of questions you can ask around that, interviews you can do to try and get some of that data, and that might inform where you want your innovation rate to be. If your existing customers are very unhappy, then perhaps you need to address that. Perhaps you need to invest more in augmenting the existing stuff or figuring out what you’re missing in that space. That could be an expectations gap, which would be more on the unrealized value side—stuff your product doesn’t do yet.

But understanding that data informs the decisions that you make. So there’s a piece of data in our organizational capability, specifically in the ability to innovate, and our innovation rate. Our decision on what should we do with that data depends on this other data, right? Data on how people feel about our current product or the features in our current product. Maybe it’s all the way down to individual features; granularity is quite important there.

What we’re missing in our product, and if we’re missing a lot, we probably want to invest a lot of money in the missing stuff. If we’re not missing much but our customers are unhappy, we probably want to invest a lot in increasing the satisfaction of our customers within our existing product structure. All of these decisions are informed but not controlled by the data, right?

So not only do we need to collect the data, but we need to realize that just because the data says something doesn’t mean we should immediately act upon it. There are some things that you absolutely should immediately act on, like if the trend of defects is suddenly shooting up or customer satisfaction suddenly takes a big drop. Those are things that you want to check the data first, then perhaps have some kind of bigger action on it.

But there are times where you might take the opposite action. One of the things that you would definitely collect in current value is your usage index, as they call it in evidence-based management. But how, what is usage data? It’s telemetry in your product, right? So do you know if you have a button that takes you to a feature, and that feature has options? How many, what percentage of your users use that feature? How often do they use that feature? And when they do use that feature, what parts of that feature do they interact with?

That should be data that you have on hand, right? That’s data that we can reasonably easily collect. It requires engineering work; your developers need to add stuff to your product to collect that data. If you’re building a website, you might get some of that with Google Analytics, but usually, we’re looking for a deeper story than just what pages are hit, depending on the way your website’s constructed.

If you’re doing web, exactly what features, how long features take to load, how long actions take—these are all parts of that telemetry story that give you an absolute wealth of data that you can access. I heard a great story from a car company that I worked with, and they were talking about how easy it was to access that data within their organization. Most car companies collect this type of telemetry, even if they have no over-the-air updates.

When your car goes in for a service and they connect up the computer system at the dealership, it sucks down all of the telemetry data on all of the gear shifts, how the gear shifts performed. There’s a big log in the system; they load that data, and it gets sent off for faults and all those kinds of things. That gets sent off to the parent company, and then the idea is you would analyze that data, right?

They were talking about their inability to access that data, even though it exists. To get access to a slice of that data, to a piece of that data, required a whole bunch of time and approvals. In this particular case, they said something like six to eight months before they could get access to the data once they requested it, and that’s far too long. That’s us then making business decisions based on six to eight-month-old data. Because by the time we get that slice of the data, it’s old, and we need a new slice of the data.

Whereas they told me that one of their colleagues worked at Tesla, and at Tesla, everybody has access to all of that data instantaneously, continuously. You can go look at the real-time stream of that data from the cars to know what people are doing, how people are interacting with the features and capabilities within the car. That’s game-changing data, right? Having it at all is game-changing.

Having access to as close to real-time as possible—like if I’m working on a feature right now and I’m doing continuous delivery, so I’m shipping to production on a regular basis, let’s say it’s on average every couple of days. I’m shipping this feature to production, and it’s shipping out to all the cars like Tesla does, right? They do continuous updates. Then I want data at this point. I want as close to this point as possible.

So I make a change, I ship it in, I’ve got two days until my next change. I’m going to push out, what do I need to fix? What do I need to change? Well, I’m going to be analyzing the data in that segment to update my product backlog and engage with the team and figure out what needs to happen. This is what product management is all about, right? You’re actually managing the product, and you’re managing the product with the data to understand how you need to manage the product.

This is hugely important, and that difference is your customers loving your software or hating your software. I have a Nissan Leaf, and their software is probably some of the worst, least useful. I don’t know how much more to describe it; it’s the most horrible software to use and work with and the least useful from any perspective in the way that it’s constructed, the way it functions, the way I engage with it. It doesn’t provide me with the things I want; it doesn’t provide me with the things in a timely manner.

It’s terrible; it’s just terrible. And that’s likely because nobody’s looking at the data. Nobody’s looking at it. I think it has the Android store; it’s got a—I think it did have a 1.2 rating at one point. It was a horrible, horrible app that nobody must be looking at that data, or they don’t care, right? If the business doesn’t care about that part of the system, then why provide it, right?

Maybe they’re providing it because they feel they have to, but if you have to provide it because it’s table stakes, it needs to be good; otherwise, it reflects on your company and your business, right? Collecting this data as a product manager is the whole ball game. So we just talked about one metric. We started with an innovation rate, and we ended up connecting all those stories together.

The Visual Studio team at Microsoft gets seven terabytes of data per day, and that’s just from the clients, the Visual Studio desktop clients out in the world that are on networks that allow that data to be sent. Lots of big organizations have those things locked off so that that data doesn’t exit the network, right? So this is just a subset of users, but it’s a subset of data that they can interact with, that they can see what’s going on in the product, that they can make a change and push it out, and then see almost real-time whether they’ve been successful or they’ve not in achieving the hypothesis that they had.

We think adding this feature is going to make a change in people’s behaviour in this way. Let’s look at the telemetry. Did we improve the user engagement with this feature? Did we improve the number of people that used it? If not, we didn’t achieve our outcome with that change. Let’s think about why we didn’t do that. Maybe we want to talk to a couple of users and figure out why they didn’t change their behaviour, right?

All of those types of things are part of building great products. So we’ve talked about some where we can get some of the data. We talked about how to collect the data. Some of it is just, I guess, we talk about to collect it. Some of it is just you stick it in Excel, right? There’s nothing bad about Excel; it’s a great tool.

So you just record it on a regular cadence and stick it in Excel. That could be from customer survey data; that could be a slice of your innovation rate when you’re looking at it. If you’re really on the ball, you might feed all of that data that you have into a data warehouse and then use something like Power BI to analyze it and have dashboards that show you what’s going on near real-time in your product, in your system, when your customers are doing and how they’re interacting with it.

There are leading measures and lagging measures. Leading measures are ones that change frequently, right? That you can make a change to the system and then see that data changing. Lagging measures are things that take a little bit longer to come to fruition. If you surveyed users on their satisfaction, then that’s going to be a lagging measure, right?

You’re making changes to the system; it isn’t going to immediately reflect changes to the data. You would need to analyze it over a longer period of time. Both are important, right? Your leading stuff is changing the work that you’re doing right now; your lagging stuff is perhaps affecting more your overall strategy and what’s going on.

There’s a great example of a feature that the Visual Studio team built called IntelliTrace. If you don’t know what it is, it doesn’t matter, but it was a big, big, expensive feature. It was only in the Enterprise product to start with, and it was a very expensive feature to build. I can’t overstate that enough; it’s like huge amounts of money, more ridiculous amounts of money than we could possibly imagine.

What they found was that nobody was using it. So they built this feature, they spent all this money on this feature, they shipped this feature, and then they’re looking at the telemetry, and it says nobody’s using it. So what do you do? Do you cut that feature? Do you stop investing in that feature? Do we invest more in that feature? We maybe need a little bit more information.

So when they did dive into it, right? Go see a little bit of gamber, right? Go have a look, talk to people. They found that people weren’t using it because they didn’t know it was there, and it was off by default, and they had to go turn it on in the settings. So in the next version of the product, because this was actually when they were on the two-yearly releases, it then took them, let’s say, 18 months or whatever amount of time to get this change in.

So that’s 18 months of people not using the feature, remember, of that feature effectively sitting on a shelf and depreciating while your competitors might have that feature. You do have that feature, but your customers don’t know it’s there, so they don’t use it. They might go to your competitors because they think your competitors have a feature that you don’t.

When they turned it on by default and shipped it out, nobody turned it off. Very few people—the percentage of people that turned it off was very small, and then they were getting that usage data and getting that usage telemetry. That’s why I always say the data that you collect informs but does not control your decision-making.

Because there needs to be more to it than that. You’re playing poker, not chess, right? If you’re playing chess, all of the information that’s available is in your hands, right? You can see the state of the board; you know all the moves; you know all the possible counter moves. That’s where you get to those grandmasters thinking 20, 30 moves ahead.

I don’t know how many moves ahead they think, but they’re thinking so many moves ahead. You know it’s 10 moves to checkmate, right? And then how does this person get out of it? The only way they can do that is to, I guess, maybe fake you into doing something, or you make a mistake, right? I’m going to win unless I make a mistake; I’m going to win this game.

There’s no hidden information. But for building products, it’s more like playing poker. We have incomplete information; we don’t understand all of what’s going on. Now ideally, we want as much information as we can on the things we can find out to fill in enough of the gaps so that when we’re looking at our big picture or even our little picture, we can kind of say, “Well, we think this is what’s happening,” right?

We can infer the missing information. When you’re playing poker, you know the cards in your hand; you know the cards that are open on the table, on the type of pool you’re playing, but you don’t know what your competitors have got in their hand, and you don’t know what the dealer is hiding or what the next card’s going to come off the stack.

Now, if you’re really good at card counting, if you’re telemetry and data, then you can have a more informed decision-making. If you get very good at reading the other players around the table, really reading their emotional intent, right? Those tails that they have, then you get more information that you can use.

That’s who the best poker players are, right? They are poker players that recognize the cards that go through the deck. They’ve got good memories; they recognize the emotional engagement between the players and the game to be able to discern more information about what their competitors are doing without actually knowing, right? It’s a lot of guesswork.

Then they make decisions on whether they’re going to ante up, fold, or increase the bet. They might go all in, right? I’ve definitely got the winning hand; we’re going to go all in and run with this. So that’s what we’re playing as product management.

Understanding the data, collecting the data, viewing the data, analysing the data—some tools for analysing the data. Wow. IT probably is the big, big tool. My background is in the Microsoft space, so I tend to know the Microsoft solutions for these problems. But I know a lot of people like Power BI, even when they’re using other platforms to store the data.

Power BI to put all the data together. Probably things like App Insights is great. Application Insights is Microsoft’s tool for collecting telemetry. It supports open telemetry. There’s a platform called OpenTelemetry that you can use, and you can store the data wherever the heck you want.

But it enables you to collect and stream data direct from customers; that’s your telemetry capture. This is part of the story, right? So you’ve got your data, you’ve analysed your data, you’ve been able to tell the difference between what you think is happening versus what you know is happening, right? Your intuitive versus your actual data.

You need another tool; you need a tool to help enable your organization to more quickly adapt and sync through those processes. The main tool that I would use to collect data to understand our processes, right? It’s exactly the same as understanding our organization. We want to understand our process that our team or our product is our operating model for our product.

I’m probably going to use a Kanban metrics, right? That’s going to give us quantitative data on what’s going on within our system that our teams have created, that we’ve created within our organization to handle work. Then I want to make tweaks to that system and be able to see in the quantitative data that it’s changing, right?

It’s changing in a way that we want it to change, right? We’re improving, reducing our cycle time, improving our throughput—all of those kinds of things. So that’s great. Kanban really allows you to model any system, right? But there are tools out there, technologies out there that you can adopt that allow you to kind of jumpstart that process.

You might be way over here on, “My team, every team in my organization made up their process, and every member of every team does their own thing.” That’s a pretty common place for most organizations to be. We want to have a little bit of standardization while enabling the teams to be free to do those things that we’ve standardized in any way they choose.

This is the same idea that Microsoft talks about: one engineering system. That’s Azure DevOps, right? This idea that you have one place to store all the stuff so everybody knows where to go look. Everybody knows how to go set things up in that thing.

When an engineer moves from one product to another product, they’re not going and hunting for, “Well, where’s the automated builds? What technology is this? I need to learn a new technology to even change and understand this automated build.” You don’t have to do that; we understand that stuff already.

Then it’s just looking at how does it actually run, right? Then we can jumpstart our people coming into that process. The same is true in process, like 1ES for our actual engineering systems. Scrum is a social technology for our social systems, right?

We can jumpstart our move towards agility. Doing Scrum doesn’t mean you’re agile, right? We want to improve our systems, and Scrum can be a good application to say, “Here’s our bounded environment,” right? So that if you’re familiar with Kotter’s Change Model, you create a bounded environment for change.

You create a set of principles that are going to guide that change, and perhaps there’s some rules in there at the start, right? We need a set of rules to get people started so that they’re then following the principles. Then we start taking away, like, “You don’t have to do this; you don’t have to do that.”

When you were new at this, you had to do it this way. You know, like training wheels on your bike. Then you can take the training wheels off, and suddenly you’re riding your bike on your own. If you look at the Scrum Guide, it has a literal set of rules, but it’s based on a set of principles.

So the rules are not a must; the principles are a must, if that makes sense in the Scrum Guide. But the rules are there because most teams, especially when they start out, especially when they’re coming from something else, need an example of how to start doing something. Once they are doing that thing and doing that thing well and understand why those particular structures are in place, they can then start doing something different from those particular structures in order to still fulfil the same outcome from the principles.

That’s why I talk about Scrum as a social technology. We’ve got our engineering, you know, one ES, one engineering system. What is your one ES? Your one engineering system that helps from an engineering perspective ensure that everybody in your organization is able to move between different products, is able to help each other, and collaborate with each other, share knowledge and ideas because they’re all working within the same relative context, even though how they actually do stuff is completely up to them.

Evidence-based management is really your lead into a whole bunch of things that we can do really, really well and to understand how we’re doing things. You can apply evidence-based management to your business; you can apply evidence-based management to your product; you can apply evidence-based management, some of the metrics, to an individual team, and you can figure out what’s happening with your system, how your business is organized.

When you make changes to that system, you can see it’s the empirical data that you need to understand, not just your engagement with the market and your engagement with your users, but how your people are engaging and your products are delivering within that market so that you can adapt your strategic, intermediate strategic, and tactical plans in order to have a successful business, build the best product possible, and generate lots of money.