It’s time to let you in on The Secret. Don’t tell anyone, but “How long does it take” is the wrong question.
I planned for this post to be about statistics and how to calculate probabilites. Instead it turned into a rant, of sorts. You see, there’s another Secret buried here: If you really take the time to learn how to estimate risk correctly, this will help you learn the following fact about the outcome of your project: It is very uncertain. I guess that’s helpful. Sort of.
“That’s just baloney,” I can hear some of you say. “My projects are consistently on time and on budget,” you continue. Maybe so. Let’s examine the famous “project triangle” for a moment.
Ever wondered why it isn’t a square? Because quality is a result of the other three, some might say. “That’s just baloney,” you can hear me say. I think it’s because cost, scope and time is easy to measure and easy to adjust. Quality? We always deliver perfect quality! We are professionals! Let’s just put that in the middle, and hope no one pays too much attention.
You, the guy who screamed “We always deliver on time and on budget!” a moment ago. How about scope? “Hah! We delivered all the functionality as well! All the developers bitched about them having too much to do in too little time, but my project plan showed them wrong!”
That’s impressive. But even when we go to the painstaking length of figuring out the risk and uncertainty of a project PROPERLY – You know, with Statistics and stuff, not just, God forbid, guessing or anything – it still comes out as being very uncertain. So how come you can consistently deliver your projects on time, scope and budget? Your team don’t have much choice, do they? They have to reduce quality. Luckily for you, no one will notice until the project is over. And they couldn’t really measure it if they did. What IS quality anyway? Sounds like a made up word to me.
So what did I mean by “how long does it take” being the wrong question?
There are a couple of problems. First of all, what is this “it” that we are doing? When you start the project, odds are neither you or the customer knows what the end result is supposed to look like. You may think that you do, but you don’t.
If it is a waterfall project, the customer will say “This is what I want” while dropping the dreaded Complete Requirement Specification on your desk. Then you go away for three months with a team of developers to create whatever is in the specification. “Here is your product!” you exclaim with great enthusiasm when you come back. After the customer has tested it, she goes “Nope, that’s not it.”
If it’s an agile project, your team of developers only go away for maybe a couple of weeks at the time. Each time you come back and show a little bit of the product, the customer tests it and goes “Nope, that’s not it either. And where’s the rest?”
However you go about it, both you and the customer learn a lot DURING the project. Mostly what they don’t want. Needless to say, this process takes quite a bit longer than just doing stuff once. No one knows what done looks like until, well, you’re done. And please remember, we’re not BUILDING a product, we’re CREATING it. The first of its kind. Your developers aren’t packing meat in boxes along a conveyer belt, they’re freaking SCIENTISTS! So stop pretending you are a production facility. The product you’re creating has never been built before. If it has, please go and buy that instead.
And that’s not even half of it. What did you say were doing again? A project, right? Well, most of the time we’re not really creating projects, we’re creating PRODUCTS. The funny thing about products is that they don’t end when the project ends. That’s kind of when they start. In terms of cost, that means roughly 90% of the total cost of a product comes AFTER the initial release. Maintenance, bug fixing, additional development, paying back technical debt, people spending most of their day swearing over how horrible it is, and so on.
Here’s the Third Secret: The less time you spend on quality during the “project” phase, the higher the total cost of the product.
So whenever you pat yourself on the back for delivering “on time” (usually a random point in time decided by anything except the actual amount of work that needs to be done before that date), that probably means you have reduced the overall quality of the project, and increased the total cost of the product you delivered. That doesn’t really sound like back patting performance, if you ask me.
Did you think this series of posts would culiminate in How To Estimate Accurately In 3 Easy Steps? Well, it sort of didn’t. If you’re still looking for that guide, you should probably read “The Flaw of Averages: Why We Underestimate in the Face of Uncertainty” by Sam L. Savage.
I’m not sure it will make your estimates more accurate, but at least you’ll understand why.
(This is part #2 in my mini series of blog posts about estimates. Part #1 can be found here)
With the help of detailed analysis, an experienced developer may figure out the most likely outcome of a project. But exactly HOW likely is the most likely outcome?
In the previous post we were forced to guess the duration of a project, and ended up guessing the project would take 300 hours. As a side note, the result after detailed analysis will probably be quite close to the same number. This is known as anchoring. In order to keep each post relatively short, we will save that for another post.
Let’s assume you’re given time to do a detailed analysis of the proposed project, consider the risk and are provided with a “complete specification”. You spend maybe half a day thinking about it, jot down some detailed estimates, and add them all together. The total sum is 290 hours, and you figure that to be the most likely outcome of the project. Your assumption may be exactly wrong for any number of reasons, but for the sake of argument, let’s assume you’re actually correct.
You have now figured out that the project will most likely take around 290 hours. But exactly HOW likely is it that the estimate matches the actual end result? 60%? 80%? You may be surprised to learn that the actual number is probably quite a bit lower than that.
Above illustration represents the actual outcome of 42 Norwegian projects from several different companies. (Magne Jørgensen / scienta.no)
Let’s say you have historical data from several past projects, telling you both your estimated, most likely outcome, as well as the actual outcome. If you put them in a graph with the x-axis representing the actual effort in percent of the estimated effort, and the y-axis representing the percentage of the total number of projects, it would probably look something like the graph above. Confused yet? Let me explain that again.
The highest bar in the above graph is the one marked “100” on the x-axis, and the value of that bar is around 33. That means that roughly 33% of the projects were completed on the estimated time. Since it’s the highest bar, being on time appears to be the most likely outcome. The next bar to the right, marked “125” on the x-axis, indicates that a little over 15% of the projects actually spent 125% of the estimated time, and so on.
So why do you care? Because it sheds light on what “most likely” really means. And it may not be what you think. Actually, our most likely outcome (hitting the estimate spot on) is NOT very likely to happen. Yes, it surely is more likely than any other individual result, but it is LESS likely to happen than all the other results combined. We actually have a whopping 67% likelyhood of NOT hitting our estimate, even though (due to some unknown miracle) our estimate is the most likely result of all possible results.
To make matters worse, the combined likelyhood of the bars to the LEFT of the 100% bar (representing actual outcomes that were lower than our estimate) is only about 7%. So we got 33% likelyhood of hitting our estimate, and 7% likelyhood of being below.
If you provide your boss with your “most likely” estimate, you will in this example leave him with a 60% chance of blowing the customers budget.
I’ll leave you with some time to think that through. Part #3 of this mini series can be found here.
I know a whole bunch of developers in different companies and different businesses. What they all love most about their jobs, is when they are asked to produce an estimate. It’s the highlight of their week!
You’ve had a creative discussion with your boss and perhaps a couple of representatives from the customer. It’s all fun and games, until your boss turns to you and asks The Question.
“So, how long does it take?”
Did he just do that? In front of the customer, no less? How on earth are you supposed to answer that reliably without any time to analyse the details? You venture a somewhat vague answer, even though you know it’s no use.
“Hm, perhaps somewhere between 200 and 400 hours?”
“What does that mean, between 200 and 400 hours? Just give me a number!”
“Oh, uhm, I guess around 300 then?” you reply desperately, automatically reaching for the average of the first two numbers you threw out there. Surely that can’t be too much off?
Any one number representing a possible future outcome of something, will never be anything except just that. One possible outcome. That means somewhere out there, you’ve got a whole bunch of other possible outcomes as well. So If you say it might take 200 hours, without the backing of additional data, the chance of it taking more or less is about 50% either way.
So you’re giving your boss a definite number based on your guess on the outcome of something, without any information about other just as likely outcomes. Does that sound like a useful number to you? I didn’t think so.
Estimates are usually needed to figure out wether to invest in something, and to set a budget. If you give me a number where it’s a 50% chance of me blowing my budget, then I’d say you’re not really helping me over here. It’s heads or tails wether I’m in trouble with the customer for overspending. So when I ask for an estimate, it’s implied that I need a number that isn’t very likely to be too low.
So if your boss (or your customer) insists on getting that one number, what to do? First we need to understand more about the problem of “just give me a number”. More on that in part #2 on this mini-series about estimating!
As I’ve mentioned before, accurate estimates are hard. To make matters even worse, you will never become an expert at it.
It’s just not possible. If you practice enough, you may become an expert at playing chess or tennis. Why? They are both activities with predictable rules and clear consequences to any action. The concept isn’t difficult to grasp, in order to master it you just need to practice (a lot).
The saying goes that no chess game is ever the same, but an expert chess player will still have an intuitive feel for the game. Looking at a chess board in the middle of a game, she can immediately tell you the preferred moves and the likely outcome of the game.
Estimating complex projects is another story. You probably have to deal with unknown uknowns. There are the stuff you know, and the stuff that you know that you don’t know. But what usually kick you in the groin, are all the things that you don’t know that you don’t know. Any number of things may happen during the project that will dramatically affect the outcome.
To make matters worse, our brain has a nasty habit of completely ignoring unknown information. In the context of estimating, that means unless you know absolutely everything about the task at hand, you will underestimate the work required. You have to force yourself to consider the known unknowns, and include them in your estimate.
Your brain also loves replacing a hard question with a simpler one. And most of the time, you won’t notice. “Will it be a good investment to buy this house?” is a complicated question, that most people readily replace with “Do I like this house?”. Much easier.
The same thing happens when someone asks you to estimate a complex system based on some simple drawings of the user interface. Your brain goes into lazy mode, and attempts to replace the difficult question “How long would it take to design, build and test all the logic behind that system while working in a team of several people?” with the much simpler “How long would it take me to build a system that looks like that?”
And the answer to that question, as we all know, results in a useless estimate.
The third problem is that we are all too confident in our own abilities, and in hindsight we are usually convinced that we had a hunch or gut feeling about the true outcome of the project. So whenever a project goes over budget, we usually “knew” deep inside. We just need to follow our gut the next time! And whenever a project is actually on time, we have probably added features until we spent the available time. And whenever a project is delivered before the deadline…just kidding, that never happens.
So the next time you are asked to estimate something: Pay attention to your gut feeling, that is your experience talking. But force your brain to keep working. Consider what you don’t know, and be honest with yourself and everyone else about the fact that your estimate will never be 100% accurate. The larger the project, the more uncertain your estimate will be.
If you want to know more about how your brain works in relation to decision making and predicting the future, I strongly recommend “Thinking, Fast and Slow” by Daniel Kahneman.
The guys in the agile community have been writing, talking and moaning about estimates for ages. “Accurate estimates are impossible! What’s the point? Let’s stop estimating, it’s just guessing anyway!“
Let me try to explain.
We all agree that we’re basically guessing, right? That may be the case, but if you live in the real world, you actually sell stuff for a living. In order to sell stuff, you need to have customers. And what’s the problem with customers? They usually demand to know what something costs before it’s produced. Go figure. Anyway, that means we need to estimate wether we like it or not.
A rough estimate from a developer or architect is usually the basis for a fixed price offer. If the estimate is good, you’ll make money. Sometimes it’s not, and then you’re up shit creek. If bad luck and an inexperienced developer team up, rough estimates can be several hundred percent off the mark. So why do we keep working like this?
The big problem with this transaction is trust. The customer rarely trust the software company enough to pay for a project by the hour. The history books are full of dishonest companies who have destroyed the trust in the IT industry. Now we all have to work hard in order to regain that trust. You need to convince each and every customer that you actually want to add value to their business by creating great software. That’s what we’re all about. Right?
If you gain enough trust within your market, I strongly suggest that you attempt to operate with budget limits instead of a fixed price. Agree on a minimum scope, and explain to the customer that you will do your best to include as much functionality as possible within the budget you have agreed upon. This way you reduce the risk on your end, and the customer doesn’t have to pay for the built-in risk premium of a fixed price. A better deal for everyone.
Great stuff, happy campers all around! Now we can stop estimating and focus on real work.. But wait a minute. How can you (or the customer) know wether the project will be worthwhile? How do you establish a reasonable budget? When presented with the required time to market, how can you tell if you will be able to deliver on time? You guessed it, by estimating. Seems like there’s no way to win this game.
To make matters worse, the game changes as you go along. As your company grows, your development process will change. You add quality assurance, you add a project manager. Your team grows, and suddenly lots of time is spent talking about how to do stuff instead of actually doing it. Challenging SLAs means more time spent on release management and server maintenance. The environment surrounding your team changes all the time, and the correct answer to “how long will it take to..” changes with it.
At the end of the day, an estimate can never be more than an educated guess. But fear not, all is not lost: If you manage to build a team with focus on quality, keep the turnover low and maintain a stable, honest environment with room for making mistakes – you might just end up with a bunch of guys that are amazingly good at guessing.