Evidence-based Management: Gathering the metrics

Gathering the metrics for Evidence-based Management in software organisations can be a strenuous task and I have a number of customers that are fretting on what to collect and from where. Here I try to create an understanding of the ‘what’ that we need to collect.

Updated to reflect the 2020 Scrum Guide! 

Evidence-based Management for Software Organisations is a framework from Scrum.org to help organisations focus their efforts on most valuable improvements and monitor their effect on the organisation. In my experience, organisations recognize agility as a means to improve their value, yet tend to lack focus and commitment to moving towards it. This is mostly because of entrenched traditional ideas and a resistance to change that comes, in the west, from an “if it ain’t broke, don’t fix it” mentality. Let’s forget for a moment that it has been proven over and over again that small experiments that fail quickly are what produce results. Let’s forget as well the 20 years of the success of agility when there is focus and courage. We need evidence…

That evidence is there. You can find study after study that shows that agile practices work, that moving to agile values works better. So why don’t people listen? Still, I hear: “That will not work here”, especially in the enterprise. In the successful enterprise its worse: “We are fantastic and we know best as we have been doing it this way for many years with success”. Unfortunately, the measure of success has traditionally been:

• delivered on time

• delivered on budget

• with the features that were required upfront

This measure of success does not take into account how much ‘value’ is delivered to the customer or users. If you take another look, as the Standish Group in Boston has, at the above traditional measures of success (used both by PMI and Prince2) you find that this measure equates to very little value to the customers. Forrester is currently trying to encourage the organisation to start measuring the value that they are delivering rather than how they are delivering it.

How we measure value up until now in software delivery has been to either rely on the Product Owner, or to start introducing traditional measures from formal process… and we already know where that leads. The dark side!

So what can we measure? Sure Scrum has traditionally focused on the team, and the best measure for a team is “Remaining Work”. From that we can create Velocity and Burndown to help the Product Owner plan and the Development Team deliver. Simple… but organisations are not simple and need a way to really measure success in an agile fashion. If we can provide organisations with some standard agile metrics maybe, just maybe, we can pull them out of the dark and into the light.

This is why, with all the kafuffle over “scaling Scrum”, Scrum.org has been biding its time and doing its research. Most of the scaling Scrum methodologies out there miss the boat on measurements. Not only that they have [come under much fire recently] for providing way too much rigor in the wrong places for a “one size fits all” solution. If you can’t have “one size fits all” at the team level you sure can’t do it at the organisational one…

I have discussed What my father taught me about evidence-based management and metrics that matter over the years:

Evidence-based Management for Software Organisations (n):  A process for measuring, diagnosing and improving the value an organization derives from its software investments. It improves an organization’s ability to compete on its software capabilities by using evidence to focus investments on areas that will create the highest value for the entire organization. Its iterative, incremental approach to guided change helps organizations control the risk of disruption.

You can find out more from The Evidence-Based Management Guide: Measuring Value to Enable Improvement and I want to talk a little more about the gathering of metrics for monitoring.

Let’s face it, it’s not just Organisations that want to understand their return on investment for their hard earned cash spent on improving their process and practices. There are good consultants out there that want to understand the value that they are bringing to their customers so they can inspect and adapt as well. I, for one, want to focus on delighting my customers. To bringing them the most value for what they pay me. I have traditionally done this by simply looking at how happy my customers are. While this is an important metric is only one piece of information in a web of interconnected data that I can use as evidence of that improvement.

That is why I went to Boston last month to complete the training and assessment to become an Evidence-based Management Consultant with Scrum.org. I need to understand how my advice affects an organisation that I am working with. How else can I know if my advice is good? Guesswork? That sounds a little…archaic. There is a better way…

There are really three groups of metrics that make sense at the organisational level. There are metrics that monitor the current value being delivered, the time to market and your organisations ability to innovate. We need to gather these metrics at a point in time as soon as possible to provide a baseline for where the organisation currently is. We can then reassess the organisation at intervals, probably no more than quarterly, which will allow us to look at the trends over time.

Current Value Metrics

The current value metrics look at data that gives an indication of the company’s success in the market place. These metrics should be gathered once across the organisation with the organisation defined as consistently as possible across instances. They might include:

• Revenue per Employee – Gather the revenue generated by the product you are evaluating and employees contributing to its success

• Cost of Product Domains – Gather the IT expenses associated with the product

• Employee Satisfaction – what percentage of employees are engaged?

• Customer satisfaction – what percentage of your customers is marketing your products or services on your behalf?

All of these are easy to collect and most are likely readily available. Many enterprises already do some form of employee and customer satisfaction which we may be able to use. However the timespan of the collection of the metrics should ideally match.

Time-to-Market Metrics

These metrics look at how quickly an organisation is able to bring new innovations to its customers. They might include:

• Release frequency – How often do we put new features in the hands of our customers?

• Release stabilisation – How much time are we spending on getting something “done” to “really done”?

• Cycle time – How long does it take for us to move new features from idea into customer’s hands?

Ability to Innovate Metrics

These metrics look at an organisation’s ability to continue bringing innovations to its customers in the future. They might include:

• Installed Version Index - What percentage of your customers are able to take advantage of new features on the latest release of your product?

• Usage Index – What percentage of our features does not provide value to our customers?

• Innovation Rate – What percentage of our budget is spent on innovation?

• Defects – How much technical debt are we accumulating?

As you have already likely surmised with these metrics it makes sense to gather them per application. We can then average the results to figure what they are across the entire spectrum. While these figures may vary wildly between applications as long as you use the same formula you should get comparable results.

Scrum.org suggests aggregating these into one, easy to track metric – the Agility Index. This number becomes synonymous with the value an organization is getting from its software development capabilities.

Conclusion

For the first time we can now, as consultants, monitor the improvements we are making in organisations. Even better, organisations can use these metrics to demonstrate the value that their investments in agility with training and consulting are bringing them. This is a fabulous step forward in our understanding of software delivery. An easy, simple way of measuring our progress towards process improvement.