video

Say-Do Metrics: Avoiding Agile Banditry in Your Organization

Published on
6 minute read

In Agile environments, there’s often a temptation to rely on metrics that seem to offer clarity and control over a project’s progress. One such metric is the “say-do” metric, which measures what a team says they will do versus what they actually accomplish. While this may appear useful on the surface, it’s often a slippery slope that leads to vanity metrics, reduced psychological safety, and, ultimately, a focus on outputs rather than outcomes.

In this post, we’ll explore the dangers of say-do metrics, why focusing on outcomes is critical, and how to steer clear of what I call Agile banditry. Let’s dive in.

What Are Say-Do Metrics?

Say-do metrics compare a team’s estimated project outcomes with the actual results. A common example is tracking original estimates vs. actual hours worked on a project. On the surface, this seems like a valuable way to measure performance. However, as I’ve seen time and again, say-do metrics are ripe for manipulation, leading teams down the wrong path.

A Real-Life Example of Misleading Metrics

Let me share an example from an organization I worked with many years ago. The head of the PMO (Project Management Office) proudly showed me a presentation of metrics sent to leadership. The data compared original estimates with actuals for five ongoing projects, each in the thousands of hours range.

This initially sounds impressive, right? But then I noticed something odd: one project had an original estimate of 5,232 hours and an actual time spent of… 5,232 hours.

Wait, what? 🧐

When Metrics Become Vanity Metrics

I pointed out this highly suspicious match between estimated and actual hours. The head of the PMO sheepishly admitted that, to avoid the leadership’s wrath after last year’s budget issues, they allowed project managers to submit change requests that adjusted original estimates. By the time the final report was due, they had managed to align their estimates perfectly with the actuals. This data was going to be presented to leadership, and key funding decisions would be made based on it.

Here’s the problem: this data wasn’t reflective of reality. It was complete fiction. These kinds of vanity metrics can have disastrous consequences when leadership uses them to make funding or project prioritization decisions.

Key Lessons from Misleading Metrics:

The Impact on Psychological Safety

Monitoring say-do metrics not only skews the data but also undermines psychological safety within the team. When teams feel pressured to match their estimates to the actuals, they’re incentivized to manipulate data rather than report the truth.

Here’s what happens:

This lack of transparency and manipulation of data fosters a culture of fear. If you’re not delivering exactly what you promised, you’re seen as a failure, even if you delivered something far more valuable.

🚫 Avoid Agile Banditry

To prevent this, stop relying on say-do metrics. Focus on fostering transparency, creating psychological safety, and delivering outcomes that matter, not just outputs that look good on paper.

Outputs vs. Outcomes: What’s the Difference?

Focusing on Outputs

Outputs are the tangible results of work completed, often measured by the number of tasks, features, or deliverables produced. In a traditional Agile environment, this might mean delivering a specific number of features or completing a certain amount of work within a sprint.

However, focusing too much on outputs can lead to situations where teams prioritize quantity over quality. This often happens when metrics like original estimates vs. actuals take center stage.

Focusing on Outcomes

Outcomes, on the other hand, are the value derived from the work. It’s not just about completing 10 features; it’s about delivering 9 that are incredibly valuable to the end-user or stakeholder. When we shift our focus from outputs to outcomes, we measure success based on the impact of our work, not just the completion of it.

Let me share an example from my time at an MVP Summit at Microsoft. One year, our group of MVPs spent days determining the five most valuable features that we believed Microsoft should develop for Azure DevOps.

The following year, Brian Harry, a key figure at Microsoft, took the stage and said something that initially surprised us: “Of the five things you said were the best things we could build, we built none of them.”

That could have been devastating – after all, we’d spent a significant amount of time and effort figuring out those five features. But then he explained that what they did build was far more valuable to customers. They had focused on outcomes, not just ticking off our list of recommendations. And in the end, they blew us away with the value they delivered.

🔥 Key takeaway:

How to Shift Focus to Outcomes

If your organization is stuck in an output-focused mindset, it’s time to make a shift. Here are some steps to help move the focus from say-do metrics to outcomes:

1. Encourage Transparency

Create an environment where teams feel safe sharing the real challenges and roadblocks they’re facing. Open and honest communication leads to better problem-solving.

2. Measure Value Delivered

Instead of measuring tasks completed or hours worked, focus on the value delivered to the end user. Are the features being delivered making a positive impact?

3. Incorporate Flexibility

Allow teams the flexibility to adapt to changing circumstances. Agile is all about responding to change. Strictly adhering to say-do metrics leaves little room for the iteration and improvement that Agile thrives on.

4. Continuous Improvement

Encourage teams to focus on continuous improvement rather than just hitting arbitrary targets. Celebrate the learning and adaptations that come from not getting things perfect the first time.

Say-Do Metrics: A Bandit’s Tool

At the end of the day, say-do metrics are a tool of Agile banditry. They allow organizations to create a veneer of success while covering up the true picture. And when leadership makes decisions based on these vanity metrics, the consequences can be dire.

What You Should Do Instead:

If your organization is grappling with Agile banditry and misleading metrics, my team at Naked Agility can help. We specialize in helping teams and organizations get back on track, focusing on delivering real value rather than playing games with data.

video Agile Scrum agile project management agile product development agile product management project management product development product management professional scrum trainer scrum training scrum certification scrum.org DevOps consultant DevOps coach DevOps engineer agile coach agile consultant agile trainer scrum framework scrum methodology scrum approach agile leadership leadership.

Connect with Martin Hinshelwood

If you've made it this far, it's worth connecting with our principal consultant and coach, Martin Hinshelwood, for a 30-minute 'ask me anything' call.

Our Happy Clients​

We partner with businesses across diverse industries, including finance, insurance, healthcare, pharmaceuticals, technology, engineering, transportation, hospitality, entertainment, legal, government, and military sectors.​

Akaditi Logo
Brandes Investment Partners L.P. Logo
Graham & Brown Logo
Qualco Logo
ALS Life Sciences Logo
Sage Logo
Emerson Process Management Logo
Cognizant Microsoft Business Group (MBG) Logo
SuperControl Logo
Milliman Logo
Schlumberger Logo
Capita Secure Information Solutions Ltd Logo
Bistech Logo
ProgramUtvikling Logo
Illumina Logo
Slaughter and May Logo
Lean SA Logo
Microsoft Logo
Ghana Police Service Logo
Washington Department of Transport Logo
Washington Department of Enterprise Services Logo
Department of Work and Pensions (UK) Logo
Royal Air Force Logo
Nottingham County Council Logo
Alignment Healthcare Logo
ALS Life Sciences Logo
Kongsberg Maritime Logo
Flowmaster (a Mentor Graphics Company) Logo
DFDS Logo
Schlumberger Logo