Why your estimates don’t matter

February 15, 2013

Let’s pretend for just a second that we need estimates in order to perform our business. Some of you will say that we do and some will probably say that estimates are a big waste. But for the moment, let’s at least pretend that they have a place.

dice

Usually we do estimation in order to provide some kind of predictability in our deliveries. It’s just that an estimate is not enough on its own. Knowing that something will take 6 man weeks to implement has no value unless we know that we have 6 man weeks at our disposal. We need to combine our estimate with some kind of capacity measure in order to get predictability. There’s a big difference if our team can give the task 6 man weeks worth of attention within the next two week iteration or if they’re overloaded with other work and need 4 calendar months to finish the requested 6 man weeks.

So we need an estimate AND a capacity in order for the estimate to have any value. The thing is that it’s not enough either. When we estimate, we also need to agree on what we’re estimating. We need to have the same view on the expected quality; both external and internal quality. Everyone involved needs to know how the solution is supposed to behave; if the customer expects a Lexus but a developer is about to build a go-cart, the estimate will have no value. Everyone involved needs to have the same view on the level of confidence for the internal quality; if the developer is ok with Windows 95 quality but the tester is expecting a pacemaker level of confidence, the estimate will have no value.

So now we need an estimate AND a capacity AND an understanding of quality in order for the estimate to have any value. The thing is that if we make an estimate and it’s wrong, the effects will fade over time (unless we’re dealing with systematic estimation errors). If a requirement was estimated to take 5 days but actually took 10 days (a 100% estimation error), the effect on a six-month project will be less than 4%. An error in capacity on the other hand will multiply if left to itself. If a team is working in two-week sprints and plans are made with a 10% error in capacity, this error will multiply for each sprint and for a six-month project, we’ll have to add another two sprints to the end in order to finish what we had initially planned. But even worse is the cost of poor quality. These costs tend to rise exponentially with time. The longer time a poor assumption or a bug goes unnoticed, the more code will get built on top of that error and either multiplying the instances of the actual error or at least building dependencies to the error.

In short:
Error in estimate – impact decreasing linearly with time
Error in capacity – impact increasing linearly with time
Error in quality – impact increasing exponentially with time

But where do people put their attention when plans fail? They usually address the estimate and way too often put blame on the teams for not doing good enough estimates. Apart from being unethical since estimates are nothing but guesses, it’s also a waste of time since any deviations from the plan are much more likely to come from errors in capacity measurements (or worse; capacity estimates) or a mismatch in the understanding of what quality (or functionality) was being estimated.

So if predictability is what you’re looking for, don’t invest much in your estimates, instead you should make sure that your capacity is known and that quality (internal as well as external) is well understood. And that’s why your estimates don’t really matter.

Advertisements

22 Responses to “Why your estimates don’t matter”


  1. Really great post. I like arguments and reasoning. I’d wish it was understood to more “traditional” project managers 😉

    What I personally like very much is one of the last sentences i.e. “don’t invest *much* in your estimates”. Formal and typical “estimation meetings” acceptable somehow for people looking for detailed prediction and “command & control” style can be used to deliver as little estimates as possible and changed into “understanding meetings”. This kind of conversation between product, development and QA people brings us closer to desired understanding of subject including capacity and expected quality.

  2. Payson Hall Says:

    Excellent post, thank you for taking the time to write it. I agree with your points, but there is an omission that is important. We also do estimates to support decision making. Each feature, each project should be “worth doing” before it is undertaken. This is a decision made based upon an estimate of the value of doing something compared to the estimate of the cost of doing something or not doing something (augmented by risk considerations in the big leagues… estimation risk, risk of failure, etc.).

    Estimates support decision making. A poor estimate (too high) might cause deletion of a valuable function or cancellation of a project that really had a good ROI. A poor estimate (too low) causes waste by encouraging mis-allocation of resources to low value pursuits… and may ultimately result in organizational failure (called bankruptcy, unless you are a large financial institution or a government entity).

    I appreciate your sentiment at a tactical level, but encourage you to also consider the strategic. If you are told that remodeling your house will require $200K and proceed with the project… then discover half way through that you will need $300K (after they have demolished the roof and walls) and you don’t have the extra $100K – it might personally bankrupt you (or at least cause a legal tussle with the contractor). In this situation, the fact the error is linear is academic. You made an ill-informed decision in good faith based upon bad information.

    • Rasmus Rasmussen Says:

      “Context is king”

      In a typical IT project having relatively big learning element, and where there are options to take decisions late, then estimating becomes less valuable compared to “rebuilding roof of football stadium”.

      There are many IT projects (or did I just happen to get hit by all of them?) where there are lots of waste in trying to estimate a world far too complex to predict. Also, the effects of these false plans generates big costs in different ways.

      Estimates gives you bad decision making. if possible I would rather lean on “feedback provides decision making” because estimates are… estimates. Good feedback loops helps putting the value of estimates in its true light.

      • Payson Hall Says:

        Agree with you that context is important. Disagree that estimates give you bad decision making… would turn that around to say, your decision making can be no better than the quality of your estimates. This should cause decision makers to reflect soberly on the quality of the information that they are getting from estimates.

        Also, pardon the construction analogy. I tend to consult on IT projects in the $250M+ range… I know estimating them is hard. Managing expectations about the quality of the estimates is hard. It is a reflection of the immaturity of our discipline and our inability to describe large systems well (the “quality” dimension that the author describes above).

        This underscores the value of phase-limited commitment. Estimates may be flawed, but they are important. If you can afford a $300M system, but can’t afford a $600M system then you will live or die based on the quality of your estimates.

    • morgsterious Says:

      Thanks for commenting and I’m glad you liked the post. I understand your concern about wanting information about costs up front but I don’t think that problem differs from the one about predictability. You will still need to know your capacity for the estimate to have any value. A simple ROI will not suffice, you will also need to take cost of delay into account and to do that, capacity must be known. And the same goes for quality; you must have an agreement and understanding on the material to use for the roof when remodeling, if the chimney should have new flashing, if safety benches should be mounted etc. Those things are quite simple to foresee when it comes to roofing, they are part of the continuous learning when it comes to product development.

      Cheers!

      • Payson Hall Says:

        I think your point about estimates and predictability is sound (that’s one of the things I liked about your original post), but imagine the situation where a vendor is bidding on the system. Capacity is now (largely) the vendor’s concern… the quality of the definition remains key (as you mention above).

    • dianezw Says:

      Many people “do estimates to support decision making.” That does not make it a sound decision. Most software development projects are so complex and involve so many unknowns that an estimate is merely a guess. (Read this great post: http://zuill.us/WoodyZuill/2013/01/22/a-thing-i-can-estimate/)

      Determining an ROI on a guess does not seem sound to me. It is far better to determine a small need, build a solution and then evaluate the costs, benefits, etc. to inform the decision of whether you should proceed. If it is truly a need, like your roof example, then you have to pay for one solution or another. If your builder had communicated the expenses throughout the project, then you would know that you are heading toward being over budget and could make offsetting decisions, like choosing another roofing material or forgoing the jacuzzi tub.

      Btw, I generally don’t like building examples because while houses are complex systems, they are created through a series of repeatable tasks. An experienced builder can get very good at estimating since the steps to building a house are very similar even if the plans are different. Software development is not the same.

      Cheers! – dzw

  3. Kronda Says:

    The substance of your post is really great. I had to work extra hard to get through it though because of the constant use of the term ‘man weeks’ which I find really offensive.

    I’m sure you don’t think of it that way or you wouldn’t have used the term but I just wanted to make you aware of something that has the potential to alienate half your audience and diminish your message (which again, is a great one).

    • morgsterious Says:

      Hi Kronda,
      I’m sorry if my wording offended you. I normally use story points if I do any estimates but for the sake of making the post accessible to people not used to story points, I chose time instead and it never occurred to me that the unit could be offensive (though it is a unit that should be avoided for a number of other reasons, see The Mythical Man Month). I’ll try to be more careful in the future.

      Cheers!

  4. Pete Says:

    Some great angles here, but there are a couple of things in here that might skew the point a little.

    Firstly, estimates (points)should always be effort based, not duration. There is no reason for story point estimation to care about capacity.

    Secondly, estimation should iterate and become more accurate over time.

    Finally, if the quality isn’t understood, you’ve got a bigger problem than estimation!

    I think if you roll all these things up it can make estimation seem pointless (excuse the pun), but estimates exist at story level, sprint level, release level and higher. And frankly they’re all a bit different and offer different value. Done iteratively and well, they are immensely useful, especially in environments where some are not so used to agility.

  5. Köhlmark Says:

    I’ll quote Eisenhower here “In preparing for battle, I have always found that plans are useless but planning is indispensable”.

    This notion is something I agree with. The value of planning/estimation is getting a common understanding of the problems and the anatomy of the work ahead.

    The most valuable outcome of a planning meeting is not the plans themselves but rather the increased understanding of the work.


  6. Great post, Morgan!
    I’ll keep it right beside this one by Duarte Vasco when it comes to the flaws of estimation:
    http://softwaredevelopmenttoday.blogspot.se/2012/01/story-points-considered-harmful-or-why.html


  7. […] Why your estimates don’t matter Written by: Morgan Ahlström […]


  8. […] Estimation has been a hot topic in software development blogosphere recently. Morgan Ahlström contributed to it with a nice blog post several days ago: Why Estimates Don’t Matter. […]


  9. […] estimations constituent un sujet de controverse constant. L’auteur de l’article Why your estimates don’t matter se penche sur la question et nous apporte un excellent […]


  10. I understand your argument, however there something … wrong ? underneath.
    Estimates doesn’t really matter, if and only if you don’t care enough about capacity and quality.
    Capacity, imo, is no related to the quality of estimates : estimates are about effort, no about planning. As long as you didn’t change the planned path , the number of stages in which you make a journey do not change the distance between the starting point and the finish line. Priorities often changes, inducing lots of context switching and as much loss of time : one should simply know that changing priorities has a cost, that is not related to what was estimated. But it’s true that most of the time, bad estimates are blamed instead.
    Quality, on the other hand, have the impact you described, but only when not enough is done to keep it under control on a daily basis. Early automated tests and static code analyzers can do an amazing job on this area. They are introducing a clear and unquestionable measure of the quality that is delivered, and that is one big part of the predictability you were looking for.

    Yes, if nothing really matters, estimates doesn’t really matter. But in any other cases, they matter !!

  11. Paul Bowler Says:

    Great post! I also share the opinion that trying to get good as estimation is solving the wrong problem: http://www.radicalcompany.com/2013/01/a-problem-of-estimation/


  12. […] testing is an interesting topic. This blogpost by Morgan Ahlström nicely emphasizes that estimates are guesses. Martin Jansson of the Test Eye […]


  13. I tend to agree with Olivier Demeijer: Estimates matter to understand the scope of a project. But as soon as you produce a mess or the requirements change, the ORIGINAL estimate is useless.

    IMHO estimating your daily tasks serves as a valuable feedback loop. All you have to do is splitting up larger tasks into smaller items and that’s your “estimation”. You’ll find, that you can do a fixed number of small items every day on average (your throughput, if you like to speak with buzzwords).


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: