Why overplanning makes projects fail?
43 points by spikey_sanju 7 months ago | 31 comments- steveBK123 7 months agoSerendipity is real and hugely important to innovation.
That said, I've worked exactly 0 jobs that had too much planning in 20 YOE. However I worked lots of jobs with over-measuring. Constant check-ins, syncs, status meetings, team wide weeklies, etc. Measuring every step we take before we take the next one, but no vision as to wear this journey is supposed to actually lead.
This presents its own danger, and Big Agile Industrial Complex is the current manifestation of this problem.
I think its a micro-optimization that creates a macro problem.
We did not get from the Wright Brothers to a Boeing 777 with lots of minute check-ins and week-long visions of micro deliverables along the way.
You DO get improvements applying that process to say- churning out the current plane on the factory floor in a 25% more efficient manner, step by step. Not a lot of planning for what 6 months looks like there.
So one needs to match the type of problem they are solving with the correct amount of vision & planning.
- dkarl 7 months agoSame here, I've never worked at a place that did too much planning. However, I've been at a lot of places that wildly overvalued plans.
"Plans are useless, but planning is indispensable."
The culprit is usually someone who sees their job as checking boxes off a list. In extremely complex situations, box-checkers can be valuable to help ensure that nothing falls through the cracks, but everything goes to shit when you elevate people like that to any position involving the word "manager." Project manager, product manager, people manager -- it's a disaster to try to reduce these jobs to checking boxes. Immediately they'll start to say things like, "Why did we spend so much time creating a plan if we're not going to follow it?" They'll create deadlines for finalizing plans and insist that everything after the deadline has to robotically follow the plan. They see their job as 80% making sure every box on the plan is ticked and 20% stopping you from doing anything there isn't a box for. If they do that and the project fails, they point fingers at people for not adding all the right boxes to the project plan according to the schedule for making boxes.
Good planning isn't about checking boxes. It's about forcing yourself to consider all aspects of a project, crystallizing your plan at a certain point in time, and periodically reconsidering and re-crystallizing your plan on a cadence that makes sense for the project.
- intelVISA 7 months agoSounds like a layer of management that doesn't understand the works below, or the vision above, and has to maintain value theater.
- steveBK123 7 months agoYou just described median middle manager behavior.
- steveBK123 7 months ago
- goalieca 6 months agoOver-planning often comes from a place of management wanting observability and predictability. They want to be in control and know all the risks.
This ends up inversing all of the agile Manifesto.
* Individuals and interactions over processes and tools
* Working software over comprehensive documentation
* Customer collaboration over contract negotiation
* Responding to change over following a plan
- steveBK123 6 months ago* Customer collaboration over contract negotiation
This is a bad assumption for the agile premise in many large non tech orgs. Many cases internal tech tools just do not have users with this mindset.
Here’s what I wanted, why isn’t it already this way, go fix it, stop showing me stuff that isn’t done yet.
Or the other flavor where every touch point changes their feature of the week such that you are always behind what they actually want.
It’s entirely possibly to iterate such that no one is happy with delivery.
- steveBK123 6 months ago
- spikey_sanju 7 months agoYeah, planning can feel like a trap sometimes. But take the example of the discovery of penicillin. It was a fluke... If the scientist had followed rigid steps, he might’ve missed that moment. Flexibility and being open to surprises is key.
- dkarl 7 months ago
- csours 7 months agoPlanning and Learning are toxic to each other -
As you learn, the scope of the plan must change.
As you plan, you reduce the scope of what you may learn.
Example: We are migrating legacy code to new hosting with a small change. The plan is straightforward and simple. As we progress through migration we learn about unsupported dependencies and bad code practices in the legacy code, so we have to replan.
Previous comment: https://news.ycombinator.com/item?id=41183984
--- Bottom line:
If you don't address budget when you talk about planning, you are missing the biggest problem.
- perrygeo 7 months agoI've noticed one obvious commonality to failed projects: the plans are made in a vacuum of information, before data is gathered. This leaves land mines everywhere in the execution path, leading to big risks and large error bars on estimates.
Instead of course correcting to any new data, failed projects tend to double down on the process. Even as more information comes to light, it gets systematically ignored in favor of "sticking to the plan". This is the death of software.
Of course I've seen projects that have basically no plan at all beyond the 2 week sprint. It's not the presence or absence of a plan that hurts, it's about the quality of the plan. Duck tape and prayers don't constitute a plan either.
We need a tight feedback loop between planning and experimentation. Otherwise planning goes off the rails with bad assumptions, and experiments can go off the rails chasing new technology for its own sake. A planning process without a data-gathering period and an iteration loop to refine the design isn't software engineering!
- xyzzy123 7 months agoMost good plans will have actions / tasks that reduce risk or uncertainty as highest priority.
A good heuristic in the absence of a detailed high level plan is "work on the highest risk things first".
- spikey_sanju 7 months agoPlans are guesses... treating them as facts is the real danger. Adapt as you learn, or they become traps.
- xyzzy123 7 months ago
- jajko 7 months agoStrict detailed plans give illusion of more control. But unless given domain and technology is very familiar and whole team knows it very well, unforeseen stuff cascading everywhere will wreak havoc on it.
Have you seen a very detailed plan yet very flexible release date mixed together?
- steveBK123 7 months ago+1 Illusion of control is absolutely the correct phrase for these types of problems
You can run into this with certain types who want to talk through every possible eventuality in their head they have imagined, so they can feel like they have control of outcomes. Of course the stuff that happens in life is none of the things they imagined.
- steveBK123 7 months ago
- vouaobrasil 7 months agoWhen I worked full-time, the biggest demotivator was when my manager wanted to plan every aspect of a project. It took all the fun out of exploring. And now I work for myself, and I am much more productive than ever because I just let most projects go where the wind blows...
- spikey_sanju 7 months agoYeah, I totally get it...
When you let things flow instead of sticking to a rigid plan, you end up finding better ideas. It’s like the pressure’s off, and the real work happens naturally. That freedom makes everything more productive ;)
- spikey_sanju 7 months ago
- spikey_sanju 7 months agoHere are a couple of Reddit threads on accidental inventions. It’s wild how small, unexpected moments led to some of the biggest discoveries ever. Uncertainty has its own kind of magic—it can spark the best ideas.
- What was invented by accident?https://www.reddit.com/r/AskReddit/comments/ygqwwl/what_was_...
- What is the best accidental invention?https://www.reddit.com/r/AskReddit/comments/1cajrnr/what_is_...
- hermitcrab 7 months ago>A scientist found mold in a petri dish that killed bacteria—and boom, we got penicillin.
Casually leaving out the vast effort of a whole team of people to develop the drug:
https://www.sciencemuseum.org.uk/objects-and-stories/how-was...
Fleming gets the credit, but Florey and a whole team of people of other people did the hard work.
- from-nibly 7 months agoPresumably while having a plan
- from-nibly 7 months ago
- slightwinder 6 months agoIt's basically the difference between top-down and bottom-up. For good planning, you need to have a good knowledge about your domain, and lay out all the routes and solutions, and then just execute them. But new greatness usually comes from the unexpored, dark corners of your knowledge-map, so it's by definition kinda impossible to plan for greatness and success. Though, you can plan for raising the chances of finding success, but this is a different domain of planning, so it's not something anyone can do just because they can plan other types of projects. And let's not forget that humans are very depending on mindset and energy, to manage their limited mental abilities. Executing one type of plan will naturally prevent you from doing other things.
- Nina1000 7 months agoThis is a great reminder of how unexpected success often comes from things we don’t plan for. Focusing too much on perfection can sometimes limit creativity, but you can’t just sit back and wait for things to happen either. It’s all about balance.
- spikey_sanju 7 months agoThis is the whole point of the post...
Success rarely comes from what we aim for—it’s often about timing & flow. Perfection can hold us back, but staying open leads to breakthroughs.
Few irl examples:
1. Spend hours perfecting a video, 100 views. Post a raw clip, and it goes viral.
2. A polished product flops, but a scrappy weekend tool takes off.
3. A masterpiece gets ignored, but a casual sketch goes viral.
4. A failed dish turns into the next food trend.
Don’t wait for perfect. Keep creating, stay open, and let the unexpected happen.
P.S. Success is personal. But if you aim for something and it doesn’t happen, that’s a failure in that context.
- elpocko 7 months agolol, your "irl examples" are fantasies, made up on the spot by you to support your own made-up claim that "Success rarely comes from what we aim for". You should at least google some of the existing but exceedingly rare examples so your claims are not just hot air.
- spikey_sanju 7 months agoMade up? Hardly. I hear you, but reality isn’t as tidy as a research paper. Planning only gets you so far; timing and luck do the rest. IYKYK!
- spikey_sanju 7 months ago
- elpocko 7 months ago
- spikey_sanju 7 months ago
- 6 months ago
- jiggawatts 7 months ago“It’s too hard to change the plan now.”
That one sentence alone is enough to explain the failures and I’m not even accounting for the very real cost overheads of planning.
Don’t try to change this! You can’t convince a PM that their job is not needed, their pay depends on it being a necessary function and they will fight dirty to protect that income no matter what.
- AnimalMuppet 7 months agoThe very real cost and time overheads of planning.
It's not just that the plans are too rigid. It's not just that you don't know enough to plan in that much detail. It's that the planning takes time that you could be doing something useful with.
Note well: I am not advocating for no planning. I am against over planning.
- spikey_sanju 7 months ago"It's too hard to change the plan now"
+1 Agreed. Plans fail when sticking to them matters more than adapting...
TBH, the best plans know when to shift.
Flexibility > Rigidity
- jiggawatts 7 months agoThis sentence isn't something I made up, it's a sentence I heard this week.
A project started a year ago, planning an infrastructure (cloud IaaS) deployment of a complex product 12 months into the future.
Now, there is a turnkey software service (SaaS) version of the product that covers the same features and more for less money and takes literally seconds to spin up.
They're stuck in the mud trying to work around complex network routing, split-DNS, and other issues with the IaaS solution. They're going to spend months applying workarounds to problems that don't need to be solved because the SaaS version has none of those problems. They're already starting to talk about getting MFA and SSO integrated Q1 next year even though the SaaS uses MFA+SSO by default.
It's incredible how much inertia these things build up, and how much time, effort, and money is wasted "sticking to the plan" even if those plans are invalidated by changes to reality.
If the project manager simply said "let's rip up the plan and use the SaaS", they'd instantly make themselves redundant and might actually be let go when their contract is up for renewal. If they keep "fighting the fight" to make a bad plan succeed, then they're the hero and will be gainfully employed for the foreseeable future.
- jiggawatts 7 months ago
- AnimalMuppet 7 months ago
- spikey_sanju 7 months agoWhy do some of our best ideas come out of nowhere? This post talks about how planning too much can kill creativity, and why greatness often shows up when you least expect it.
- csours 7 months agoThe scariest and most exciting words are "That's funny ..."
---
The most exciting phrase to hear in science, the one that heralds new discoveries, is not “Eureka” but “That’s funny...” —Isaac Asimov (1920–1992)
- spikey_sanju 7 months agoExactly. 'That's funny…' is when you realize you were looking at it wrong all along...
- spikey_sanju 7 months ago
- csours 7 months ago
- ElevenLathe 6 months agoFake planning is the real evil. This is the kind where your entire quarter/fiscal year is spoken for in an elaborate tree of initiatives/milestones/epics/tasks painstakingly negotiated over months by your boss and many other parties without any input from you. First day of the quarter, you are invited to learn what inscrutable drivel is written in your tickets, and what totally random time estimates it has been given. Since this is immutable now and your performance review depends on being seen to have completed it, you start plotting ways to trick various people into explaining what they thought they were writing down. Of course you're also "doing agile" so you have to pack all these tickets into "sprints" (a synonym for fortnights). Since all your work for the quarter is already known, this is a simple matter of deciding on day 1 the exact order you will work on it all for the next 3 months. Any deviation from this plan is considered unprofessional. You are doomed. You pour a cocktail, boot your workstation and log into your daily standup with the other prisoners. It's a beautiful day but you'll be inside playing with JIRA and arguing over the meaning of "implement" versus "create" with a "technical PM" that doesn't know what Linux is.