Flaw of Averages: Conference Report from Mile High Agile

I just recently returned from Mile High Agile, a great regional conference based here in Denver. I am somewhat required to say that because I’m on the Board of Directors for Agile Denver, the primary sponsor, but I do truly believe it. One of the sessions was from Prateek Singh about probability and predictions. I thought it would be worthwhile to discuss the premise of the talk – the flaw of averages.

What is the Flaw of Averages?

The Flaw of Averages by Sam Savage
The Flaw of Averages by Sam Savage

Prateek talked about a book written about this very thing called, not surprisingly, The Flaw of Averages by Sam Savage. It’s still on my reading list, but I plan to tackle it this summer. The gist is that people are generally bad at thinking about risk and statistics. In particular, when we look at averages, we have to consider that not just are we looking at successes but also at failures. It’s a little complicated, but the idea is that an average or a mean is just a middle point in a set of data, including the extremes. Because it usually includes such varied data, our averages are inaccurate.

Predictions and estimates are incredibly important. NOAA, or the National Oceanic and Atmospheric Administration, uses predictions to inform people of severe weather, especially hurricanes. And, in the case of hurricanes, they don’t just give a specific location for the hurricane. They provide precise information about the current location of the hurricane’s eye and then provide a plot showing the possible cone of uncertainty. As new data comes in, that cone is modified to show the new projections. At all times, though, there’s never a precise answer of where the hurricane will be – just where it is.

How Does It Matter?

The biggest question we get asked as development teams is “how long will something take?”. Most people look at how long things have taken in the past, take the average, and then use that as our answer. Let’s say that we have three stories that took 1 day, 10 days, and 20 days. We would then tell our Product Owner that we think it will take about 10 days to deliver this next piece of work because that’s the average. But if we tell that Product Owner that it will take 10 days, there’s actually a 50% chance that we’ll be wrong.

What we want to do, is find a way that we can provide good estimates to our business owners. Our goals should end up moving toward something predictable. If we can provide good estimates, we build trust with our business sponsors. And the better the estimates, the more trust.

What Can We Do Differently?

The Flaw of Averages Demonstrated
The Flaw of Averages Demonstrated

Well, Prateek discussed that we need to avoid the flaw of averages. The best way we can do this is to review our performance and begin looking at how confident we are in our estimates. We can evaluate how our team performs. One way is looking at a scatter plot of our work and see what outliers exist and how well we deliver overall. An example would be the plot to the right. The plot is a sample from Actionable Agile.

When we look at this (and you’ll want to “embiggen” it by clicking on it), we can see that overall we deliver pretty well, but there are stories that take longer. What we can see, though, is that the average time is pretty low. But there are a number of stories that aren’t completed in less than the average time. But we can start talking about how confident we are at different time spans. In the chart, you can see that the average is 6 days but that we complete 85% of our stories within 15 days. So if we want to provide our business owners with a good estimate of how long something might take, we can provide that 15-day number. While it’s possible that we can complete it within 6 days, there’s still a good likelihood it will be more than that.

When we begin to consider the amount of work that we can get completed in a sprint, we typically take the average of the last three sprints. If our velocity has been relatively consistent, this works fine. If that velocity has bounced around dramatically, though, we need to be cautious about those averages. The more variability in our data, the less useful our calculations will be. Simply being aware of the flaw of averages can help us be better.

Closing Thoughts

When we look at our commitments, we need to consider the variability of our information and provide some context to our business owners and sponsors. Product Owners should know how much confidence we have in our estimates. Not that we need to try to sandbag our estimates. What we want to do, though, is be open, honest, and transparent about the data. When we bring openness and honesty into our relationships with our business partners, we build trust that goes both ways.

Feliz entrenamiento, mis amigos! (Happy coaching, my friends!)

Have questions?

Bill DeVoe