In Agile environments, there’s often a temptation to rely on metrics that seem to offer clarity and control over a project’s progress. One such metric is the “say-do” metric, which measures what a team says they will do versus what they actually accomplish. While this may appear useful on the surface, it’s often a slippery slope that leads to vanity metrics, reduced psychological safety, and, ultimately, a focus on outputs rather than outcomes.
In this post, we’ll explore the dangers of say-do metrics, why focusing on outcomes is critical, and how to steer clear of what I call Agile banditry. Let’s dive in.
What Are Say-Do Metrics?
Say-do metrics compare a team’s estimated project outcomes with the actual results. A common example is tracking original estimates vs. actual hours worked on a project. On the surface, this seems like a valuable way to measure performance. However, as I’ve seen time and again, say-do metrics are ripe for manipulation, leading teams down the wrong path.
A Real-Life Example of Misleading Metrics
Let me share an example from an organization I worked with many years ago. The head of the PMO (Project Management Office) proudly showed me a presentation of metrics sent to leadership. The data compared original estimates with actuals for five ongoing projects, each in the thousands of hours range.
- All five projects were within a 20% margin between estimated and actual time.
- Three projects were within a 15% deviation.
- Two projects boasted a margin of 10% or less.
This initially sounds impressive, right? But then I noticed something odd: one project had an original estimate of 5,232 hours and an actual time spent of… 5,232 hours.
Wait, what? 🧐
When Metrics Become Vanity Metrics
I pointed out this highly suspicious match between estimated and actual hours. The head of the PMO sheepishly admitted that, to avoid the leadership’s wrath after last year’s budget issues, they allowed project managers to submit change requests that adjusted original estimates. By the time the final report was due, they had managed to align their estimates perfectly with the actuals. This data was going to be presented to leadership, and key funding decisions would be made based on it.
Here’s the problem: this data wasn’t reflective of reality. It was complete fiction. These kinds of vanity metrics can have disastrous consequences when leadership uses them to make funding or project prioritization decisions.
Key Lessons from Misleading Metrics:
- Vanity metrics paint an unrealistic picture to keep leadership happy.
- They create a false sense of security and trust in the data.
- They divert focus from what really matters – delivering value to customers.
The Impact on Psychological Safety
Monitoring say-do metrics not only skews the data but also undermines psychological safety within the team. When teams feel pressured to match their estimates to the actuals, they’re incentivized to manipulate data rather than report the truth.
Here’s what happens:
- Teams start gaming the system to avoid negative feedback.
- Leadership becomes disconnected from the true challenges on the ground.
- The focus shifts from outcomes to merely ticking off boxes.
This lack of transparency and manipulation of data fosters a culture of fear. If you’re not delivering exactly what you promised, you’re seen as a failure, even if you delivered something far more valuable.
🚫 Avoid Agile Banditry
To prevent this, stop relying on say-do metrics. Focus on fostering transparency, creating psychological safety, and delivering outcomes that matter, not just outputs that look good on paper.
Outputs vs. Outcomes: What’s the Difference?
Focusing on Outputs
Outputs are the tangible results of work completed, often measured by the number of tasks, features, or deliverables produced. In a traditional Agile environment, this might mean delivering a specific number of features or completing a certain amount of work within a sprint.
However, focusing too much on outputs can lead to situations where teams prioritize quantity over quality. This often happens when metrics like original estimates vs. actuals take center stage.
Focusing on Outcomes
Outcomes, on the other hand, are the value derived from the work. It’s not just about completing 10 features; it’s about delivering 9 that are incredibly valuable to the end-user or stakeholder. When we shift our focus from outputs to outcomes, we measure success based on the impact of our work, not just the completion of it.
Let me share an example from my time at an MVP Summit at Microsoft. One year, our group of MVPs spent days determining the five most valuable features that we believed Microsoft should develop for Azure DevOps.
The following year, Brian Harry, a key figure at Microsoft, took the stage and said something that initially surprised us: “Of the five things you said were the best things we could build, we built none of them.”
That could have been devastating – after all, we’d spent a significant amount of time and effort figuring out those five features. But then he explained that what they did build was far more valuable to customers. They had focused on outcomes, not just ticking off our list of recommendations. And in the end, they blew us away with the value they delivered.
🔥 Key takeaway:
- Prioritize outcomes over outputs.
- Focus on what delivers the most value, not just what was originally promised.
How to Shift Focus to Outcomes
If your organization is stuck in an output-focused mindset, it’s time to make a shift. Here are some steps to help move the focus from say-do metrics to outcomes:
1. Encourage Transparency
Create an environment where teams feel safe sharing the real challenges and roadblocks they’re facing. Open and honest communication leads to better problem-solving.
2. Measure Value Delivered
Instead of measuring tasks completed or hours worked, focus on the value delivered to the end user. Are the features being delivered making a positive impact?
3. Incorporate Flexibility
Allow teams the flexibility to adapt to changing circumstances. Agile is all about responding to change. Strictly adhering to say-do metrics leaves little room for the iteration and improvement that Agile thrives on.
4. Continuous Improvement
Encourage teams to focus on continuous improvement rather than just hitting arbitrary targets. Celebrate the learning and adaptations that come from not getting things perfect the first time.
Say-Do Metrics: A Bandit’s Tool
At the end of the day, say-do metrics are a tool of Agile banditry. They allow organizations to create a veneer of success while covering up the true picture. And when leadership makes decisions based on these vanity metrics, the consequences can be dire.
✅ What You Should Do Instead:
- Shift your focus from outputs to outcomes.
- Foster psychological safety within your teams.
- Measure value delivered, not tasks completed.
- Embrace transparency and open communication.
If your organization is grappling with Agile banditry and misleading metrics, my team at Naked Agility can help. We specialize in helping teams and organizations get back on track, focusing on delivering real value rather than playing games with data.