Why projects take longer than you think

[MadTux]

Because if I get bored/annoyed with them, they have to sit in some dark corner until I get bored/annoyed with something else and need something else to work on 

[ebastler]

Because if I get bored/annoyed with them, they have to sit in some dark corner until I get bored/annoyed with something else and need something else to work on 

Thanks for a refreshing perspective after all the whining...   

[MT]

Engineering is always the bottle-neck doing product development. Up against impatient executives, managers, investors - it's been getting worse.
So much so that safety lapses are the only beacon saying "stop pushing and rushing us".
Not to self-blame our (engineer) estimating skills at all.

A lot of engineers are realizing "it's no fun anymore" because they started off doing fun hobby projects and then onto the corporate treadmill, which is all about doing things "as quickly as possible".
Creativity is gone, there is no reward for working harder or faster, and you get abused by being chastised for "late" efforts.
The trolls in Accounting can take weeks to issue a P.O. to Digi-key, but engineering can't waste a day.

These days it seams all about galley slaveing, rowing the corporate boat to the whip of the greedy masters:

Why galley slaving when FED prints the fiat bills anyway! Picking lingonberries mending the gardening rather then the boat, life to short to bow to wannabe masters.

[coppercone2]

when you total up the responses to the thread the answer is simple: its completely random, making mangers gamblers. 

[coppercone2]

also meetings, its a wild factor

[schmitt trigger]

And engineering turnover.
At the project’s eleventh hour, a key engineer who had been with the project since its conception, quits and a poor rookie is hired to lead the project to completion.
Barely.

[AG6QR]

The original article is about software.  A confounding issue in software is that you are ALWAYS doing something that has never been done before.  Or else you're an idiot.

In many other endeavors, your experience on the previous job helps you estimate the next one.  If you've built ten houses of 1500 square feet, then you've probably got a pretty good idea of how long the eleventh one will take.

But in software, if you ever find yourself doing something that's similar to stuff you did last year (or last week), or stuff you expect to do next year/week, then you'll write a program to automate the task, and the time required for that repetitive task drops to near zero.  There's nothing better than a computer to handle repetitive work.

These days, many software tasks consist of finding the right libraries and tools, figuring out how they work, and hooking them together.  Uncommon tasks still may require writing complex custom software, but if you're writing complex custom software that's similar to what you wrote last time, you should put it into a library so that you can efficiently re-use what you previously wrote.

The result is that wise and productive programmers spend all their time doing stuff they've never done before.  And doing new stuff is always hard to estimate.

That may also be true to some degree for some other fields, where people always find themselves doing new things.  The thing about software, though, is that it is so easy to automate the repetitive parts, so as soon as there's repetition, it's automated and the time required for that part of the job goes away.

Contrast software with jobs like lawn mowing.  I mow my own lawn, and I can always estimate the time required with exquisite precision, because the task is well understood and I don't get better at it.

[CatalinaWOW]

The original article is about software.  A confounding issue in software is that you are ALWAYS doing something that has never been done before.  Or else you're an idiot.

In many other endeavors, your experience on the previous job helps you estimate the next one.  If you've built ten houses of 1500 square feet, then you've probably got a pretty good idea of how long the eleventh one will take.

But in software, if you ever find yourself doing something that's similar to stuff you did last year (or last week), or stuff you expect to do next year/week, then you'll write a program to automate the task, and the time required for that repetitive task drops to near zero.  There's nothing better than a computer to handle repetitive work.

These days, many software tasks consist of finding the right libraries and tools, figuring out how they work, and hooking them together.  Uncommon tasks still may require writing complex custom software, but if you're writing complex custom software that's similar to what you wrote last time, you should put it into a library so that you can efficiently re-use what you previously wrote.

The result is that wise and productive programmers spend all their time doing stuff they've never done before.  And doing new stuff is always hard to estimate.

That may also be true to some degree for some other fields, where people always find themselves doing new things.  The thing about software, though, is that it is so easy to automate the repetitive parts, so as soon as there's repetition, it's automated and the time required for that part of the job goes away.

Contrast software with jobs like lawn mowing.  I mow my own lawn, and I can always estimate the time required with exquisite precision, because the task is well understood and I don't get better at it.

Interesting point of view.  But clearly not totally accurate.  It is foolish to automate everything you have done before.  There should be a trade between the non-zero costs of automating the task and the estimate of the number of times you will do the task again.  And there you are again, predicting the future with inadequate data.

To use your house building analogy, obviously if you expect to build 1500 square foot houses with the same layout over and over again you will automate. Prefabricate sections.  Build jigs and other tooling.   If you have just built one and don't have a customer on the line for the second one you don't know whether to automate any part of the task.  And in both houses and software the next customer will often want something "better".  1450 square feet.  Or 2500 sq. ft.  Now obviously that is a situation that begs for automating with a parameter.  But the obvious problem is that a scale factor isn't enough.  You probably have to add rooms as the size goes up.  And maybe trim level options.  Regional variations.  (Ranch style for Californians, Salt Boxes for New Englanders...).   All of which can be automated, but pretty soon the automation is a major project and you still haven't lined up the second customer.

Most design jobs have the qualities you ascribe to software.  Less akin to lawn mowing where you are doing the same thing over and over.  But in both software and hardware there are some jobs which pretty predictable.  The heavens may fall on me for saying this but web page design is one.  And I used to work for a power company "designing" new service installations.  I was an intern between undergraduate years in college and had been over educated and over qualified for the job for several years when I took it.   

[Brumby]

Building the automation system will have its problems over and above the problems of building the base product.  Subtle variations that an experienced human would just cope with without a second thought become intricate and sometimes very difficult to automate.  For starters, you would have to know that you need to capture that information and then you would have to encode it for repetitive execution.  Plus you need to include some mechanism for identifying cases that are outside those which have been encoded AND take an appropriate action - such as pausing for human intervention.

But, back on the original point:

The only way which works is splitting a project up in overseable tasks and then add the numbers.

I once had a software project where I was the team lead.  The project was to create a new system based on an existing one, so a lot of the curly bits had already been addressed.  I did the project estimate by breaking everything down to the basic components and then looking at each and asking myself "How long would it take me to do this on a good day?".  I then doubled that number.  I took the project total of 120 person-days to management and after an hour's "discussion" it got whittled back to 60.    At the end of the project, I checked the actual time taken: 125 person-days - but I still didn't get the project in on time, did I?   

[SilverSolder]

Less akin to lawn mowing where you are doing the same thing over and over.

Even lawn mowing is surprisingly complicated if you do a lot of it.  Tools...   tractor, riding mower, self propelled push, push, manual?  Totally automated lawn robot?  Strimmer?  Electric? Battery? Two stroke?  Four stroke?  etc.  -   

Efficient lawn mowing depends on knowing the specific lawn in advance, or you end up like the garden companies, with a trailer full of different implements!

[MT]

And when the lawnmower get an system add on like MCAS your in deep shit! And out the window goes everything!

[tszaboo]

C) Because there is a price target, I have to have several designs made, organize meetings with suppliers, manufacturers, gather information about prices, availability, compare them, make several effort estimates. Even organizing the meetings is several weeks.
D) You need a custom thingamabob for it, where the tooling needs several months to be made.

For these pair of items, it usually means having things made in China. This is where The Troubles begin:

From the "lost-in-the-translation" discussions, to the crazy meeting hours (in Chinglish, of course), to their holidays not aligning with ours, to the way the see things vs. your point of view...... and of course, receiving the first production batch which will be found to have a "minor" issue that will render them unusable, more crazy meetings attempting to figure out the issue, agreeing on sorting  criteria, figuring out a correct rework procedure, dealing with the inevitable scrap, paying overtime fees to meet deadlines.... and finally having to air-freight the reworked material.

In the end, the cost will be several times the original quote.

Well, not really. We make high quality stuff, high margin. Local assembly house, sourcing and design. B2B.
But that doesn't mean that I can just go crazy with the designs whenever I want.

I still have to do that most of the time, because there just isn't time. We told our manager, that we could employ two extra engineers who could do just cost cutting, cause there is so much extra money left in the designs. You know, when you have only a few months to spit out a product, you over engineer.

[donotdespisethesnake]

I have been pondering this for about 30 years, ever since I started writing software professionally.

Of course, the cart is always put before the horse. People ask "why is the project taking longer than it should", rather than "why was the time estimate wrong". For some reason, I guess because it comes first, the estimate is regarded as gospel truth, and if the actual work didn't match the estimate then it must be due to bad developers.

This despite the fact it is pretty generally known you can't predict the future. Software projects simply contain too many "unknown unknowns" to come up with accurate, detailed estimates.

I have noticed that the unknown unknowns all lead to more work required, there are vanishingly few times when it is unexpectedly discovered that less work is required. Even the known unknowns are badly accounted for, e.g. a study found developers only spent 50% of the time coding, even though they are nominally 100% on a project. The "lost time" is all the workplace overhead, like meetings, training, holidays, sickness, supporting legacy projects etc.

On the flip side, all the drivers from business management are the wrong way. i.e. they want less time, less cost, less resources. There is a strong tendency for them to refuse to listen to anything they don't want to hear. If they get an answer that doesn't fit, they ask again until they get an answer they like. Even if the developers are not "yes" men, the management will always take their most optimistic estimate.

It's a classic case where the "wisdom of crowds" breaks down, because there is a strong bias towards the "least cost" answer, rather than the median answer.

Unfortunately, I have come to the conclusion that the naive and ignorant people in charge always win the argument, so the industry will never improve.

[Tomorokoshi]

...
On the flip side, all the drivers from business management are the wrong way.
...

A running joke here is something like:

Manager: "When will that task be done?"
Engineer: "August."
Manager: "Marketing August, or Engineering August?"

Marketing August is the 1st. Engineering August is the 31st.

[SilverSolder]

Of course, the cart is always put before the horse. People ask "why is the project taking longer than it should", rather than "why was the time estimate wrong".

Often things are underestimated for the "positive" reason that people want to do the project...   because they think it is cool or find it satisfying for other reasons, so they tend to underestimate how hard it will be,  and/or overestimate how good the payoff will be!

[Tomorokoshi]

I recently had to estimate a number of tasks for a project, and assign cursory time estimates to the various tasks.

From a simple linear point of view, the time estimates usually take the form of BASE +/- ERROR or BASE +/- %ERROR. With this one gets estimates like:

40 hours +/- 8 hours.
5 weeks +/- 20%.

Or other such things.

However, with a very technical project that requires a lot of research to even understand the implications of some of the requirements, this really isn't adequate and presumes I understand much more about how to accurately estimate tasks than is possible at the early stages.

Therefore, when I presented my list of tasks with a time associated with each task, I included the disclaimer that the times in hours were accurate to a factor of 5. How can this be concisely stated? By using dB(t)!

So I get estimates like:
40 hours +/- 14 dB(h) for a range of 8 hours to 200 hours.
5 weeks +/- 14 dB(w) for a range of 1 week to 25 weeks.

Close enough for engineering work.

[ebastler]

So I get estimates like:
40 hours +/- 14 dB(h) for a range of 8 hours to 200 hours.
5 weeks +/- 14 dB(w) for a range of 1 week to 25 weeks.

Close enough for engineering work.

And the best thing: Nobody will ask you for estimates again anytime soon!