The Evidence-Based Management Guide: Improving Value Delivery under Conditions of Uncertainty

The Evidence-Based Management Guide

Improving Value Delivery under Conditions of Uncertainty May 2024 Scrum.org

This publication is offered for license under the Attribution Share-Alike license of Creative Commons, accessible at http://creativecommons.org/licenses/by-sa/4.0/legalcode and also described in summary form at http://creativecommons.org/licenses/by-sa/4.0/ . By utilizing this EBM Guide, you acknowledge and agree that you have read and agree to be bound by the terms of the Attribution Share-Alike license of Creative Commons.

Purpose of the EBM Guide

Organizations exist for a reason: to achieve something that they think they, uniquely, can achieve. They often express this purpose in different ways, at different levels, to create purpose and alignment about what they do:

• A Vision Statement, an expression of the change that the organization wants to make in the world.
• A Mission Statement, an expression of why the organization is uniquely capable of achieving the Vision Statement.
• Goals, on several different levels and timescales, that help the organization achieve its Mission and Vision.

Organizations form goals to make concrete progress toward achieving their Mission and Vision. Without goals, the Mission and Vision are simply lofty aspirations. Furthermore, without effective Mission and Vision statements goals lack a compelling purpose, especially for those working under conditions of uncertainty.

This Guide defines EBM and its concepts.

Definition of Evidence-Based Management

Evidence-Based Management (EBM) is a framework that helps people, teams, and organizations make better-informed decisions to help them achieve their goals by using intentional experimentation and feedback.

EBM Helps Organizations Achieve Their Goals in a Complex World

Complex problems don’t have easy solutions. In order to solve them, organizations must experiment by defining, working toward, and achieving larger goals in small steps. Each step involves comparing the actual result of the experiment with its desired outcome, and adapting the next step accordingly (see Figure 1).

EBM focuses on three levels of goals:

• Strategic Goals, important things that the organization feels it needs to achieve to realize its Mission and Vision. These goals are so big and far away, with many uncertainties along the journey that the organization must use empiricism to achieve them. Because a Strategic Goal is aspirational and the path to achieving it is uncertain, the organization needs a series of practical targets, like Intermediate Goals.
• Intermediate Goals, achievements of which will indicate that the organization is on the path to a Strategic Goal. The path to the Intermediate Goal is often still somewhat uncertain, but not completely unknown.
• Immediate Tactical Goals, which are the current focus of the organization’s improvement efforts.

To progress towards Strategic and Intermediate goals, organizations form hypotheses about improvements they can make to move toward their Immediate Tactical Goals. These hypotheses form the basis of experiments that they run to try to improve. They measure the results of these experiments (evidence) to evaluate their progress toward their goals, and to determine their next steps (new hypotheses), which may include adjusting their goals based on what they have learned. This is empiricism in action with EBM.

For more on complexity, see the Scrum Theory section of the Scrum Guide at https://www.scrumguides.org/scrum-guide.html

Figure 1: Reaching strategic goals requires experimenting, inspecting, and adapting Figure adapted from Mike Rother’s Improvement Kata ( http://wwwpersonal.umich.edu/~mrother/The_Improvement_Kata.html )

Setting Goals

Organizations must define measurable goals that will indicate whether that goal is achieved. These measurable goals, measures, and experiments should be made transparent in order to encourage organizational alignment.

Case Example: Infectious Disease Response

• The Strategic Goal is to eradicate the effects of the disease as measured by the number of people who fall ill and suffer significant illness. Measurement is important to understand if progress is being made and if the strategic goal is relevant across time. In this example, the goal is focused on the effects of the disease, and not on the means for achieving the desired impact. For example, the goal is not to vaccinate a certain percentage of the population against the disease. While that may be an activity necessary to achieving the Strategic Goal, it is not the Strategic Goal.
• An example of an Intermediate Goal is the successful completion of a trial of a vaccine against the disease. This is still ambitious and measurable, and achieving it may require the completion of many different activities. It is a necessary step on the path to achieving the Strategic Goal.
• Examples of Immediate Tactical Goals may include activities like isolating symptoms, evaluating a therapy, sequencing the DNA of a virus or bacterium, and so forth. These are critical near-term objectives toward which a team or group of teams will work.

The Strategic Goal is usually focused on achieving a highly desirable but unrealized outcome for a specific group of people. Achieving the goal results in improved happiness, safety, security, or well-being of the recipients of some product or service. In EBM, we refer to this as Unrealized Value, which is the satisfaction gap between a beneficiary’s desired outcome and their current experience. Unrealized Value is described in greater detail below, in the Key Value Areas section.

Understanding What is Valuable

Organizations measure many different kinds of things. Broadly speaking, measures fall into five categories:

• Inputs. These are things that the organization spends money on. While necessary to produce value, there is no correlation between the amount of input and the value that customers experience. Inputs establish constraints on experiments, e.g. an organization may establish limits on how much a team may spend (the input) to test an improvement idea.
• Activities. These are things that people in the organization do, such as perform work, go to meetings, have discussions, write code, create reports, attend conferences, and so forth.
• Outputs. These are things that the organization produces, such as product releases (including features), reports, defect reports, product reviews, and so on.
• Outcomes. These are desirable things that a customer or user of a product experiences. They represent some new or improved capability that the customer or user was not able to achieve before. Examples include being able to travel to a destination faster than before, or being able to earn or save more money than before. Outcomes can also be negative, as in the case where the value a customer or user experiences declines from previous experiences, for example when a service they previously relied upon is no longer available.
• Impacts. Results that the organization or its non-customer stakeholders (such as investors) achieve when customers or users of a product achieve their desired outcomes. Examples include things like increased revenue or profit, improved market share, and increased share price. Positive Impacts are only sustainably achievable when customers experience improved outcomes.

The problem most organizations face, which is often reflected in the things they measure, is that measuring activities and outputs is easy, while measuring outcomes is difficult. Organizations may gather a lot of data with insufficient information about their ability to deliver value. However, delivering valuable outcomes to customers is essential if organizations are to reach their goals. For example, working more hours (activities) and delivering more features (outputs) does not necessarily lead to improved customer experiences (outcomes).

While it is possible for organizations to improve impacts without improving customer outcomes, doing so usually harms the organization, such as when it reduces product quality to improve profitability, or when it sells products below cost to increase revenue and market share but harms profitability. Achieving impacts is important, but they have to be achieved in a sustainable way that does not harm the organization’s long-term viability.

Making Progress Toward Goals in a Series of Small Steps

The first step in the journey toward a Strategic Goal is understanding your Current State to frame your thinking about where and how you need to improve. For example, if your goal is to improve the satisfaction of your customers you will need to know what your customers experience today and what they would like to experience in the future. You will probably also need to understand your own capability for delivering value, i.e. how fast you are able to make improvements in the value that your customers will experience, so that you can set realistic short and medium term goals.

The Experiment Loop (shown in Figure 1) helps organizations move from their Current State toward their Immediate Tactical Goal, their Intermediate Goal, and ultimately their Strategic Goal, by taking small, measured steps, called experiments, using explicit hypotheses. This loop consists of:

• Forming a hypothesis for improvement. Based on experience, form an idea of something you think will help you move toward your Immediate Tactical Goal, and decide how you will know whether this experiment succeeded based on measurement.

• Running your experiments. Make the change you think will help you to improve, and gather data to support or refute your hypothesis.

• Inspecting your results. Did the change you made improve your results based on the measurements you have made? Not all changes do; some changes actually make things worse.

• Adapting your goals or your approach based on what you learned. Both your goals and your improvement experiments will likely evolve as you learn more about customers, competitors, and your organization’s capabilities. Goals can change because of outside events, and your tactics to reach your goals may need to be reconsidered and revised, for example:

  • Was the Immediate Tactical Goal the right goal?
  • Are the Intermediate and Strategic Goals still relevant or do they need to be adapted?
  • If you failed to achieve the Immediate Tactical Goal but you think it is still important to achieve, how might you do better next time?
  • If you achieved your Intermediate or Strategic Goals you will need to formulate new goals.

Hypotheses, Experiments, Features, and Requirements

Organizations can spend a lot of money implementing features (distinguishing characteristics) and other requirements in products, only to find that customers don’t share the company’s opinion on their value; beliefs in what is valuable are merely assumptions until they are validated by customers. This is where hypotheses and experiments are useful.

A hypothesis is a belief that doing something will lead to something else, such as delivering feature X will lead to outcome Y. An experiment is a test that is designed to prove or reject some hypothesis.

Every feature and every requirement really represents a hypothesis about value. One of the goals of an empirical approach is to make these hypotheses explicit and to consciously design experiments that explicitly test the value of the features and requirements. The entire feature or requirement need not actually be built to determine whether it is valuable; it may be sufficient for a team to simply build enough of it to validate critical assumptions that would prove or disprove its value.

Explicitly forming hypotheses, measuring results, and inspecting and adapting goals based on those results are implicit parts of an agile approach. Making this work explicit and transparent is what EBM adds to the organizational improvement process.

EBM Uses Key Value Areas to Examine Improvement Opportunities

In addition to using hypotheses and experiments to move toward goals, EBM provides a set of perspectives on value and the organization’s ability to deliver value. These perspectives are called Key Value Areas (KVAs). These areas examine the goals of the organization (Unrealized Value), the current state of the organization relative to those goals (Current Value), the responsiveness of the organization in delivering value (Time-to-Market), and the effectiveness of the organization in delivering value (Ability-to-Innovate).

Market value KVAs (UV, CV) reflect customer outcomes. Organizational capability KVAs (A2I, T2M) reflect the organization’s ability to deliver valuable customer outcomes and may be measured in terms of either outcomes or outputs. Input, activity, output, and impact measures do not tell an organization anything about organizational capability to deliver valuable outcomes.

Focusing on these four dimensions enables organizations to better understand where they are and where they need to go.

Figure 2: Key Value Areas provide lenses to examine improvement opportunities.

Each KVA focuses on a different aspect of either value, or the ability of the organization to deliver value. Delivering business value (Current Value) is important, but organizations must also show that they can respond to change (Time-to-Market) while being able to sustain innovation over time (Ability-to-Innovate). And they must be able to continually make progress toward their long-term goals (Unrealized Value) or they risk succumbing to stagnation and complacency.

Current Value (CV)

Measures that quantify the value that the product delivers today

The purpose of looking at CV measures is to understand the value that an organization delivers to customers and stakeholders at the present time; it considers only what exists right now, not the value that might exist in the future. Questions that organizations need to continually re-evaluate for current value are:

1. How happy are users and customers today? Is their happiness improving or declining?
2. How happy are your employees today? Is their happiness improving or declining?
3. How happy are your investors and other stakeholders today? Is their happiness improving or declining?

Considering CV helps an organization understand the value that their customers or users experience today.

Example: While profit, one way to measure investor happiness, will tell you the economic impact of the value that you deliver, knowing whether customers are happy with their purchase will tell you more about where you may need to improve to keep those customers. If your customers have few alternatives to your product, you may have high profit even though customer satisfaction is low. Considering CV from several perspectives will give you a better understanding of your challenges and opportunities. Customer happiness and investor happiness also do not tell the whole story about your ability to deliver value. Considering employee attitudes recognizes that employees are ultimately the producers of value. Engaged employees that know how to maintain, sustain and enhance the product are one of the most significant assets of an organization, and happy employees are more engaged and productive.

Unrealized Value (UV)

Measures that quantify the potential future value that could be realized if the organization met the needs of all potential customers or users

Looking at Unrealized Value measures helps an organization to maximize the value that it realizes from a product or service over time. When customers, users, or clients experience a gap between their current experience and the experience that they would like to have, the difference between the two represents an opportunity; this opportunity is measured by Unrealized Value.

Questions that organizations need to continually re-evaluate for UV are:

1. Can any additional value be created by our organization in this market or other markets?
2. Is it worth the effort and risk to pursue these untapped opportunities?
3. Should further investments be made to capture additional Unrealized Value?

The consideration of both CV and UV provides organizations with a way to balance present and possible future benefits. Strategic Goals are formed from some satisfaction gap and an opportunity for an organization to decrease UV by increasing CV.

Example: A product may have low CV, because it is an early version being used to test the market, but very high UV, indicating that there is great market potential. Investing in the product to try to boost CV is probably warranted, given the potential returns, even though the product is not currently producing high CV. Conversely, a product with very high CV, large market share, no near competitors, and very satisfied customers may not warrant much new investment; this is the classic cash cow product that is very profitable but nearing the end of its product investment cycle with low UV.

Ability to Innovate (A2I)

Measures that quantify the effectiveness of an organization in delivering new capabilities

The goal of looking at A2I measures is to maximize the organization’s ability to deliver new capabilities and innovative solutions. Organizations should continually re-evaluate their A2I by asking:

1. What prevents the organization from delivering new value?
2. What prevents customers or users from benefiting from that innovation?

Improving A2I helps an organization become more effective in ensuring that the work that it does improves the value that its products or services deliver to customers or users.

Example: A variety of things can impede an organization from being able to deliver new capabilities and value: spending too much time remedying poor product quality, needing to maintain multiple variations of a product due to lack of operational excellence, lack of decentralized decision-making, inability to hire and inspire talented, passionate team-members, and so on. As low-value features and systemic impediments accumulate, more budget and time are consumed maintaining the product or overcoming impediments, reducing its available capacity to innovate. In addition, anything that prevents users or customers from benefiting from innovation, such as hard to assemble/install products or new versions of products, will also reduce A2I.

Time-to-Market (T2M)

Measures that quantify how quickly the organization can deliver and learn from feedback they gather from experiments

The reason for looking at T2M measures is to minimize the amount of time it takes for the organization to deliver something that is potentially valuable. To know this they must measure the result so that they know whether they actually improved the value their customers experienced. Questions that organizations need to ask to evaluate their T2M are:

1. How fast can the organization learn from new experiments and information?
2. How fast can you adapt based on the information?
3. How fast can you test new ideas with customers?

Improving T2M helps improve the frequency at which an organization can potentially change CV.

Example: Reducing the number of features in a product release can dramatically improve T2M; the smallest release possible is one that delivers at least some incremental improvement in value to some subset of the customers/users of the product. Many organizations also focus on removing non value-added activities from the product development and delivery process to improve their T2M.

Example Key Value Measures (KVMs) for each KVA are described in the Appendix.

Inspecting and Adapting Based on Experiment Results

Once you have gathered measures from your experiments to improve value, you will need to inspect or evaluate your results against your goals to see if your improvement ideas worked. Examining measures in each of the Key Value Areas will help you to maintain a balanced perspective.

Immediate Tactical Goals should improve Current Value and reduce Unrealized Value. Even when Immediate Tactical Goals are focused on organizational effectiveness or speed of obtaining feedback, considering CV and UV helps the organization keep customer satisfaction in sight. Each KVA is a different lens that helps you focus on different aspects of your performance towards the goals you are trying to achieve.

Similarly, when your Immediate Tactical Goals are focused on improving effectiveness (A2I) or the speed at which you can obtain feedback (T2M), you never want to ignore or take for granted your customers’ experiences. When an organization targets improvements only in A2I and T2M without monitoring CV and UV, they are focused only on internal processes that may not help them further satisfy customers or achieve value. This can lead to, or be an indication of, a lack of outcome-based goals.

If you succeed in achieving your Immediate Tactical Goal, congratulations! Your next step will be to form a new Immediate Tactical Goal that, when achieved, will take you closer to your Intermediate Goal. Continue devising experiments, or things you can try, to achieve that goal.

If you’ve actually achieved your Intermediate Goal, even better! Now you’ll need to form a new Intermediate Goal that, when you achieve it, will move you closer to your Strategic Goal. You’ll also need to form a new Immediate Tactical Goal to provide you with a nearer target to work toward.

Sometimes you’ll find that your goals need adjusting. You might discover that a goal is no longer relevant, or that it needs to be refined. This can happen to your goals at any level. And sometimes you’ll fail to reach your Immediate Tactical Goal because your experiment did not produce the results you had expected. This is not a bad thing, and what you learned helps you to devise new experiments that may yield better results.

End Note

Evidence-Based Management is free and offered in this Guide. Although implementing only parts of EBM is possible, the result is not Evidence-Based Management.

Acknowledgements

Evidence-Based Management was collaboratively developed by Scrum.org, the Professional Scrum Trainer Community, Ken Schwaber and Christina Schwaber.

Appendix: Example Key Value Measures

To encourage adaptability, EBM defines no specific Key Value Measures (KVMs). KVMs listed below are presented to show the kinds of measures that might help an organization to understand its current state, desired future state, and factors that influence its ability to improve.

Current Value (CV)

• Revenue per Employee: The ratio (gross revenue / # of employees) is a key competitive indicator within an industry. This varies significantly by industry.
• Product Cost Ratio: Total expenses and costs for the product(s)/system(s) being measured, including operational costs compared to revenue.
• Employee Satisfaction: Some form of sentiment analysis to help gauge employee engagement, energy, and enthusiasm.
• Customer Satisfaction: Some form of sentiment analysis to help gauge customer engagement and happiness with the product.
• Customer Usage Index: Measurement of usage, by feature, to help infer the degree to which customers find the product useful and whether actual usage meets expectations on how long users should be taking with a feature.

Unrealized Value (UV)

• Potential Market Share: The potential market share that the product might achieve if it better met customer needs.
• Customer or User Satisfaction Gap: The difference between a customer or user’s desired experience and their current experience.
• Desired Customer Experience or Satisfaction: A measure that indicates the experience that the customer would like to have.

Time-to-Market (T2M)

• Build and Integration Frequency: The number of integrated and tested builds per time period. For a team that is releasing frequently or continuously, this measure is superseded by actual release measures.
• Release Frequency: The number of releases per time period, e.g. continuously, daily, weekly, monthly, quarterly, etc.
• Release Stabilization Period: The time spent correcting product problems between the point the developers say it is ready to release and the point where it is actually released to customers.
• Mean Time to Repair: The average amount of time it takes from when an error is detected and when it is fixed.
• Customer Cycle Time: The amount of time from when work starts on a release until the point where it is actually released.
• Lead Time: The amount of time from when an idea is proposed, or a hypothesis is formed until a customer can benefit from that idea.
• Lead Time for Changes: The amount of time to go from code-committed to code successfully running in production. See the DORA 2019 report.
• Deployment Frequency: The number of times that the organization deployed (released) a new version of the product to customers/users. See the DORA 2019 report.
• Time to Restore Service: The amount of time between the start of a service outage and the restoration of full availability of the service. See the DORA 2019 report.
• Time-to-Learn: The total time needed to sketch an idea or improvement, build it, deliver it to users, and learn from their usage.
• Time to Remove Impediment: The average amount of time from when an impediment is raised until when it is resolved.
• Time to Pivot: The elapsed time between when an organization receives feedback or new information and when it responds to that feedback.

Ability to Innovate (A2I)

• Employee Engagement: A measure of the degree to which employees are aligned with and bought-in to the organization’s goals.
• Innovation Rate: The percentage of effort or cost spent on new product capabilities, divided by total product effort or cost.
• Defect Trends: Measurement of change in defects since last measurement.
• On-Product Index: The percentage of time teams spend working on product and value.
• Installed Version Index: The number of versions of a product that are currently being supported.
• Technical Debt: Reflects extra development and testing work due to “quick and dirty” solutions.
• Production Incident Count: The number of times in a given period that the Development Team was interrupted to fix a problem in an installed product.
• Active Product (Code) Branches: The number of different versions (or variants) of a product or service.
• Time Spent Merging Code Between Branches: The amount of time spent applying changes across different versions.
• Time Spent Context-Switching: Includes time lost to interruptions and switching between tasks.
• Change Failure Rate: The percentage of released product changes that result in degraded service and require remediation. See the DORA 2019 report.