blog

Evidence-based Management: Gathering the metrics

Published on
7 minute read

Gathering the metrics for Evidence-based Management in software organisations can be a strenuous task and I have a number of customers that are fretting on what to collect and from where. Here I try to create an understanding of the ‘what’ that we need to collect.

Updated to reflect the 2020 Scrum Guide! 

Evidence-based Management for Software Organisations is a framework from Scrum.org to help organisations focus their efforts on most valuable improvements and monitor their effect on the organisation. In my experience, organisations recognize agility as a means to improve their value, yet tend to lack focus and commitment to moving towards it. This is mostly because of entrenched traditional ideas and a resistance to change that comes, in the west, from an “if it ain’t broke, don’t fix it” mentality. Let’s forget for a moment that it has been proven over and over again that small experiments that fail quickly are what produce results. Let’s forget as well the 20 years of the success of agility when there is focus and courage. We need evidence…

That evidence is there. You can find study after study that shows that agile practices work, that moving to agile values works better. So why don’t people listen? Still, I hear: “That will not work here”, especially in the enterprise. In the successful enterprise its worse: “We are fantastic and we know best as we have been doing it this way for many years with success”. Unfortunately, the measure of success has traditionally been:

This measure of success does not take into account how much ‘value’ is delivered to the customer or users. If you take another look, as the Standish Group in Boston has, at the above traditional measures of success (used both by PMI and Prince2) you find that this measure equates to very little value to the customers. Forrester is currently trying to encourage the organisation to start measuring the value that they are delivering rather than how they are delivering it.

How we measure value up until now in software delivery has been to either rely on the Product Owner, or to start introducing traditional measures from formal process… and we already know where that leads. The dark side!

So what can we measure? Sure Scrum has traditionally focused on the team, and the best measure for a team is “Remaining Work”. From that we can create Velocity and Burndown to help the Product Owner plan and the Development Team deliver. Simple… but organisations are not simple and need a way to really measure success in an agile fashion. If we can provide organisations with some standard agile metrics maybe, just maybe, we can pull them out of the dark and into the light.

This is why, with all the kafuffle over “scaling Scrum”, Scrum.org has been biding its time and doing its research. Most of the scaling Scrum methodologies out there miss the boat on measurements. Not only that they have [come under much fire recently] for providing way too much rigor in the wrong places for a “one size fits all” solution. If you can’t have “one size fits all” at the team level you sure can’t do it at the organisational one…

I have discussed What my father taught me about evidence-based management and metrics that matter over the years:

Evidence-based Management for Software Organisations (n):  A process for measuring, diagnosing and improving the value an organization derives from its software investments. It improves an organization’s ability to compete on its software capabilities by using evidence to focus investments on areas that will create the highest value for the entire organization. Its iterative, incremental approach to guided change helps organizations control the risk of disruption.

You can find out more from The Evidence-Based Management Guide: Measuring Value to Enable Improvement and I want to talk a little more about the gathering of metrics for monitoring.

Let’s face it, it’s not just Organisations that want to understand their return on investment for their hard earned cash spent on improving their process and practices. There are good consultants out there that want to understand the value that they are bringing to their customers so they can inspect and adapt as well. I, for one, want to focus on delighting my customers. To bringing them the most value for what they pay me. I have traditionally done this by simply looking at how happy my customers are. While this is an important metric is only one piece of information in a web of interconnected data that I can use as evidence of that improvement.

That is why I went to Boston last month to complete the training and assessment to become an Evidence-based Management Consultant with Scrum.org. I need to understand how my advice affects an organisation that I am working with. How else can I know if my advice is good? Guesswork? That sounds a little…archaic. There is a better way…

There are really three groups of metrics that make sense at the organisational level. There are metrics that monitor the current value being delivered, the time to market and your organisations ability to innovate. We need to gather these metrics at a point in time as soon as possible to provide a baseline for where the organisation currently is. We can then reassess the organisation at intervals, probably no more than quarterly, which will allow us to look at the trends over time.

Current Value Metrics

The current value metrics look at data that gives an indication of the company’s success in the market place. These metrics should be gathered once across the organisation with the organisation defined as consistently as possible across instances. They might include:

All of these are easy to collect and most are likely readily available. Many enterprises already do some form of employee and customer satisfaction which we may be able to use. However the timespan of the collection of the metrics should ideally match.

Time-to-Market Metrics

These metrics look at how quickly an organisation is able to bring new innovations to its customers. They might include:

Ability to Innovate Metrics

These metrics look at an organisation’s ability to continue bringing innovations to its customers in the future. They might include:

As you have already likely surmised with these metrics it makes sense to gather them per application. We can then average the results to figure what they are across the entire spectrum. While these figures may vary wildly between applications as long as you use the same formula you should get comparable results.

Scrum.org suggests aggregating these into one, easy to track metric – the Agility Index. This number becomes synonymous with the value an organization is getting from its software development capabilities.

Conclusion

For the first time we can now, as consultants, monitor the improvements we are making in organisations. Even better, organisations can use these metrics to demonstrate the value that their investments in agility with training and consulting are bringing them. This is a fabulous step forward in our understanding of software delivery. An easy, simple way of measuring our progress towards process improvement.

measure-and-learn people-and-process blog ebmgt evidence evidence-based-management featured leadership-track merics

Connect with Martin Hinshelwood

If you've made it this far, it's worth connecting with our principal consultant and coach, Martin Hinshelwood, for a 30-minute 'ask me anything' call.

Our Happy Clients​

We partner with businesses across diverse industries, including finance, insurance, healthcare, pharmaceuticals, technology, engineering, transportation, hospitality, entertainment, legal, government, and military sectors.​

ALS Life Sciences Logo
Hubtel Ghana Logo
Slicedbread Logo
Cognizant Microsoft Business Group (MBG) Logo
Kongsberg Maritime Logo
YearUp.org Logo
Teleplan Logo

CR2

Jack Links Logo
Genus Breeding Ltd Logo
Graham & Brown Logo
Milliman Logo
Lean SA Logo

NIT A/S

New Signature Logo
DFDS Logo
Akaditi Logo
Freadom Logo
Washington Department of Enterprise Services Logo
Department of Work and Pensions (UK) Logo
Royal Air Force Logo
Nottingham County Council Logo
New Hampshire Supreme Court Logo
Ghana Police Service Logo
Capita Secure Information Solutions Ltd Logo
Big Data for Humans Logo
Genus Breeding Ltd Logo
Emerson Process Management Logo
Kongsberg Maritime Logo
Graham & Brown Logo