Evidence-based Management: Gathering the metrics

Gathering the metrics for Evidence-based Management in software organisations can be a strenuous task and I have a number of customers that are fretting on what to collect and from where. Here I try to create an understanding of the ‘what’ that we need to collect.

Updated to reflect the 2020 Scrum Guide!

Evidence-based Management for Software Organisations is a framework from Scrum.org to help organisations focus their efforts on most valuable improvements and monitor their effect on the organisation. In my experience, organisations recognize agility as a means to improve their value, yet tend to lack focus and commitment to moving towards it. This is mostly because of entrenched traditional ideas and a resistance to change that comes, in the west, from an “if it ain’t broke, don’t fix it” mentality. Let’s forget for a moment that it has been proven over and over again that small experiments that fail quickly are what produce results. Let’s forget as well the 20 years of the success of agility when there is focus and courage. We need evidence…

That evidence is there. You can find study after study that shows that agile practices work, that moving to agile values works better. So why don’t people listen? Still, I hear: “That will not work here”, especially in the enterprise. In the successful enterprise its worse: “We are fantastic and we know best as we have been doing it this way for many years with success”. Unfortunately, the measure of success has traditionally been:

  • delivered on time
  • delivered on budget
  • with the features that were required upfront

This measure of success does not take into account how much ‘value’ is delivered to the customer or users. If you take another look, as the Standish Group in Boston has, at the above traditional measures of success (used both by PMI and Prince2) you find that this measure equates to very little value to the customers. Forrester is currently trying to encourage the organisation to start measuring the value that they are delivering rather than how they are delivering it.

How we measure value up until now in software delivery has been to either rely on the Product Owner, or to start introducing traditional measures from formal process… and we already know where that leads. The dark side!

So what can we measure? Sure Scrum has traditionally focused on the team, and the best measure for a team is “Remaining Work”. From that we can create Velocity and Burndown to help the Product Owner plan and the Development Team deliver. Simple… but organisations are not simple and need a way to really measure success in an agile fashion. If we can provide organisations with some standard agile metrics maybe, just maybe, we can pull them out of the dark and into the light.

This is why, with all the kafuffle over “scaling Scrum”, Scrum.org has been biding its time and doing its research. Most of the scaling Scrum methodologies out there miss the boat on measurements. Not only that they have [come under much fire recently] for providing way too much rigor in the wrong places for a “one size fits all” solution. If you can’t have “one size fits all” at the team level you sure can’t do it at the organisational one…

I have discussed What my father taught me about evidence-based management and metrics that matter over the years:

Evidence-based Management for Software Organisations (n):  A process for measuring, diagnosing and improving the value an organization derives from its software investments. It improves an organization’s ability to compete on its software capabilities by using evidence to focus investments on areas that will create the highest value for the entire organization. Its iterative, incremental approach to guided change helps organizations control the risk of disruption.

You can find out more from The Evidence-Based Management Guide: Measuring Value to Enable Improvement and I want to talk a little more about the gathering of metrics for monitoring.

Let’s face it, it’s not just Organisations that want to understand their return on investment for their hard earned cash spent on improving their process and practices. There are good consultants out there that want to understand the value that they are bringing to their customers so they can inspect and adapt as well. I, for one, want to focus on delighting my customers. To bringing them the most value for what they pay me. I have traditionally done this by simply looking at how happy my customers are. While this is an important metric is only one piece of information in a web of interconnected data that I can use as evidence of that improvement.

That is why I went to Boston last month to complete the training and assessment to become an Evidence-based Management Consultant with Scrum.org. I need to understand how my advice affects an organisation that I am working with. How else can I know if my advice is good? Guesswork? That sounds a little…archaic. There is a better way…

There are really three groups of metrics that make sense at the organisational level. There are metrics that monitor the current value being delivered, the time to market and your organisations ability to innovate. We need to gather these metrics at a point in time as soon as possible to provide a baseline for where the organisation currently is. We can then reassess the organisation at intervals, probably no more than quarterly, which will allow us to look at the trends over time.

Current Value Metrics

The current value metrics look at data that gives an indication of the company’s success in the market place. These metrics should be gathered once across the organisation with the organisation defined as consistently as possible across instances. They might include:

  • Revenue per Employee – Gather the revenue generated by the product you are evaluating and employees contributing to its success
  • Cost of Product Domains – Gather the IT expenses associated with the product
  • Employee Satisfaction – what percentage of employees are engaged?
  • Customer satisfaction – what percentage of your customers is marketing your products or services on your behalf?

All of these are easy to collect and most are likely readily available. Many enterprises already do some form of employee and customer satisfaction which we may be able to use. However the timespan of the collection of the metrics should ideally match.

Time-to-Market Metrics

These metrics look at how quickly an organisation is able to bring new innovations to its customers. They might include:

  • Release frequency – How often do we put new features in the hands of our customers?
  • Release stabilisation – How much time are we spending on getting something “done” to “really done”?
  • Cycle time – How long does it take for us to move new features from idea into customer’s hands?

Ability to Innovate Metrics

These metrics look at an organisation’s ability to continue bringing innovations to its customers in the future. They might include:

  • Installed Version Index – What percentage of your customers are able to take advantage of new features on the latest release of your product?
  • Usage Index – What percentage of our features does not provide value to our customers?
  • Innovation Rate – What percentage of our budget is spent on innovation?
  • Defects – How much technical debt are we accumulating?

As you have already likely surmised with these metrics it makes sense to gather them per application. We can then average the results to figure what they are across the entire spectrum. While these figures may vary wildly between applications as long as you use the same formula you should get comparable results.

Scrum.org suggests aggregating these into one, easy to track metric – the Agility Index. This number becomes synonymous with the value an organization is getting from its software development capabilities.

Conclusion

For the first time we can now, as consultants, monitor the improvements we are making in organisations. Even better, organisations can use these metrics to demonstrate the value that their investments in agility with training and consulting are bringing them. This is a fabulous step forward in our understanding of software delivery. An easy, simple way of measuring our progress towards process improvement.

Upcoming Training Opportunities

These are the next five classes we have, and you can check out our full public schedule of classes.

Timezone:
Live Virtual Professional Scrum Product Owner online 5th June 2023
Virtual
5-8 Jun, 2023
09:30-13:30 BST
4 half-days
Live Virtual PAL Evidence-Based Management Online on 19th June 2023
Virtual
19-20 Jun, 2023
09:00-13:00 BST
2 half-days
APS 19th June 2023
Virtual
19-22 Jun, 2023
09:00-13:00 EDT
4 half-days
Live Virtual Professional Scrum Master Online on 26th June 2023
Virtual
26-29 Jun, 2023
09:00-13:00 EDT
4 half-days

We can deliver any of our courses as private in-house training over Microsoft Teams & Mural. We also recommend training based on your accountabilities or role, you can go directly to recommended courses for Scrum MastersProduct OwnersDevelopers and Agile Leaders.

Create a conversation around this article

Share on Facebook
Share on Twitter
Share on Linkdin

Related Courses

Evidence-Based Management

...

𝗣𝗿𝗼𝗳𝗲𝘀𝘀𝗶𝗼𝗻𝗮𝗹 𝗔𝗴𝗶𝗹𝗲 𝗟𝗲𝗮𝗱𝗲𝗿𝘀𝗵𝗶𝗽 𝘄𝗶𝘁𝗵 𝗘𝘃𝗶𝗱𝗲𝗻𝗰𝗲-𝗕𝗮𝘀𝗲𝗱 𝗠𝗮𝗻𝗮𝗴𝗲𝗺𝗲𝗻𝘁 (𝗣𝗔𝗟-𝗘𝗕𝗠) is delivered over two half or one full day(s) and includes the industry recognised 𝗣𝗔𝗟-𝗘𝗕𝗠 𝗮𝘀𝘀𝗲𝘀𝘀𝗺𝗲𝗻𝘁. It shows how leaders can guide their teams toward continuously improving customer outcomes, organizational capabilities, and business results. EBM focuses on customer value and intentional experimentation to systematically improve an organization’s performance and achieve its strategic goals.

Read more

Martin Hinshelwood nkdAgility.com
https://youtu.be/_fFs-0GL1CA Why do you encourage people to follow a certification path in their career journey? I would encourage you to follow a scrum certification path for the same reason that people go to university. The same reason that people do any course or certification. It gets you a foot in …
Martin Hinshelwood nkdAgility.com
https://youtu.be/QGXlCm_B5zA What will you learn on the PSM II course? There are two main things that most scrum masters will learn on the PSM II or Advanced Professional Scrum Master course. That they haven’t been working effectively as a scrum master to date. That they are there to empower and …
Martin Hinshelwood nkdAgility.com
In Scrum Events across the world, I hear repeated the phrase “that’s how agile works” when describing behaviours that are both unprofessional and the very opposite of an agile mindset. These behaviours will inhibit agility and are a result of a lack of understanding of the underlying principles. We need …
Martin Hinshelwood nkdAgility.com
https://youtu.be/klBiNFvxuy0 What is the most common Aha moment people have in a scrum course? It depends on the scrum course they are attending. The content presented in the Applying Professional Scrum (APS) course leads to very different epiphanies when compared to the content presented on an Advanced Professional Scrum Master …

OUR TEAM

We believe that every company deserves high quality software delivered on a regular cadence that meets its customers needs. Our goal is to help you reduce your cycle time, improve your time to market, and minimise any organisational friction in achieving your goals.

naked Agility Limited is a professional company that offers training, coaching, mentoring, and facilitation to help people and teams evolve, integrate, and continuously improve.

We recognise the positive impact that a happy AND motivated workforce, that has purpose, has on client experience. We help change mindsets towards a people-first culture where everyone encourages others to learn and grow. The resulting divergent thinking leads to many different ideas and opportunities for the success of the organisation.