Waterfall Should Have Never Existed: Part 1

I’d like to begin by sharing a brilliant quote that puts the latest project management fashion, Agile, into humbling perspective: “A Waterfall project is just an Agile (Scrum) project with one huge sprint! An Agile project is just a Waterfall project almost entirely implemented using Rolling Wave Planning! (from George Intzesiloglou, UK on 2020-Sep-11 at ProjectManagement.com)

In the following two-part article, I will argue that:

1. ‘Agile’ is not agile.
2. ‘Waterfall’ is not the same as Critical Path.
3. ‘Waterfall’ should never have existed.
4. Hybrid approaches between ‘Agile’ and ‘Waterfall-that-should-not-be’ should not exist either.
5. A true hybrid approach between ‘adaptive project management’ (Agile) and ‘predictive project management’ (like Critical Path) should exist, can exist, and does exist. This is most viable.

In other words, be prepared to have your world shaken up a little bit. To entice you to continue to read, I promise to follow my claims with a solution for doing both Agile and Critical Path concurrently. In fact, you do not have to choose between Agile and Critical Path. Both can be done in one schedule!

I will explain how you can marry these two seemingly‑opposite approaches to scheduling in one project in Part 2 of this article. I will even recommend using ‘Agile’ to manage-down (i.e. your project team). At the same time, I will recommend Critical Path (CPM):

• To manage-up (i.e. your executives),
• To manage-out (i.e. your clients), and
• To manage-in (i.e. yourself as the project manager)

Let’s use CPM to manage the overview of a project … and your sanity, don’t forget. Before we get to that; however, let’s consider why Agile became so popular to begin with.

Why Did Agile Become So Popular?

For at least twenty years and counting, the world around us has become more and more software driven, and, as a result, more digital. Electric vehicles are about 50% software, in terms of value, whereas fossil fuel cars are mostly hardware. Banks have essentially been software developing organizations for a long time. Fintech is accelerating this trend. In the last twenty years, records management began moving away from paper records to electronic ones. We see this happening in the medical world nowadays. The wall of paper files in your doctor’s office will soon be gone.

Obviously, as digital transformation has occurred, and the number of software projects increased exponentially. This has led to a dramatic growth in software development projects and in the number of software project managers we’ve seen and will continue to see. It’s accompanied by a flourishing consulting industry eager to sell the latest management fad to anyone with a heartbeat and willingness to listen. At the turn of the millennium, software projects were mostly organized in a Waterfall way and were reported to have poor results (Standish Group reports). In 2001, a group of software thought-leaders got together and hammered out the Agile Manifesto. They called themselves the ‘Agile Alliance’ and the Agile movement was born. It is riding the wave of digitizing our world.

The Agile movement set up associations (‘Agile Alliance’ and ‘DSDM Consortium’). The latter, Dynamic Systems Development Method (DSDM) came from the Rapid Application Development movement. These associations and thought leaders, authors, and researchers developed several flavors of Agile: Scrum, Scrum XP (eXtreme Programming), ScrumBan, DSDM, DevOps, and Lean-Agile, and market them vigorously. There are even resellers for the DSDM Consortium if you want to sell DSDM.

In the process of this development, Agilists hijacked a few techniques and presented them as their own. For example, KanBan and incremental scheduling (a.k.a. ‘Rolling Wave’ scheduling which preceded Agile by many years). Many private companies jumped on the opportunity and have developed software applications to support the Agile way of delivering projects: JIRA, @Task, Monday, Trello, Wrike, VersionOne, etc. As you can see, a whole new cottage industry grew out of the ‘Agile’ movement.

Since about 2010, Agile was challenged to start providing solutions at the executive level within organizations. These organizations needed an approach that would start providing predictions. One response to this was Dean Leffingwell private company, Scaled Agile, Inc. He created the ‘Scaled Agile Framework’ methodology (SAFe) and SAFe-assessment tools for organizations, as well as ‘SAFe certification’ for professionals. At the time of writing, over 600,000 practitioners have been trained worldwide in SAFe by over 350 ‘Scaled Agile Partners’ Simultaneously, Scrum was put into the public domain and responded with its ‘Scrum of scrums,’ which seems to flounder in the shadow of Scaled Agile.

When executives started to hear about ‘Agile,’ they kind of liked the term and, hence, the term ‘the agile organization’ was born. In short, ‘Agile’ began shaping up for the big leagues (i.e. the executive level).

Please note I am using the uppercase ‘Agile’ (as in ‘Agile Project Management’) and the lowercase ‘agile’ (as in ‘more adaptive’). I will use these consistently going forward to separate the ‘Agile’ project delivery method from the common sense meaning of the word ‘agile’ executives felt attracted to.

The ‘agile’ Organization

Just about every executive wants to have an ‘agile organization.’ The word ‘agile’ is just like the word ‘Quality.’ You cannot say ‘No‘ to it; you cannot NOT want it. Executives cannot say ‘No’ to ‘agile,’ because it is a word that is inherently positive in nature and wanted in a dynamic, capitalist society. The idea of ‘agile’ is something most executives want for their organization. Who does not want their company to be flexible and responsive to our constantly changing world? There are enough changes right now: the COVID-19 pandemic, the Black Lives Matter (BLM) movement against systemic racism, the opposing forces of globalization versus protectionism, the arrival of new technologies like machine learning and artificial intelligence, and our changing climate.

In the context of a quickly changing world, executives are being sold on the concept of making their organizations more ‘agile’ (lowercase!) by adopting ‘Agile’ (uppercase!) project management practices. In retrospect, I think it is a widespread misunderstanding that if executives implement ‘Agile Project Management’ (Uppercase), they will make their organizations more ‘agile’ (lowercase). After all, Agile is ‘a highly structured way of delivering projects.’; in other words: ‘Agile’ is not ‘agile’! In my mind, ANY form of project management makes an organization more ‘agile’ as in ‘adaptive’. After all, project management, by its definition, is ‘managing a temporary endeavor that creates a unique product, service, or result’ (PMBOK^® Guide, any edition). Hence, project management ALWAYS brings about change, and ‘bringing about change’ is perceived as being ‘agile.’

Over time, it amazed me how ‘Agile Project Management’ (uppercase!) ran away with the main prize of getting the ear of C-level executives: PMI had tried for many years to get their ear after all. Executives, in their eternal quest to making their organization more ‘agile’ grew interested in ‘Agile Project Management’. The good thing is that ‘Agile Project Management’ has brought the topic of ‘project management’ into executive discussions and into the boardroom of organizations, and so PMI followed along.

In my mind, implementing ‘predictive project management’ techniques (i.e. Critical Path, Resource Critical Path, Critical Chain, Earned Value, Earned Work) make an organization at least as ‘agile’ (in the sense of ‘more adaptive’) as the ‘adaptive project management’ technique of uppercase ‘Agile’ would.

Agile Will Not Conquer the World

Agile will never conquer the entire world of projects! There are many types of projects that will NEVER be planned in an Agile way: construction projects, plant shutdown and turnaround projects, software enhancements on core operations systems (as little as possible downtime allowed), and many others. Can you imagine that a cloud scraper would be built in an Agile way: “Oh, we built the next floor, let’s go back to the client now … see if they like this floor and then we will ask them for a budget to build the next floor! Not sure if it will be financed … oh well, we will just stop this project for a while…” This will never happen; Agile will not and should not be implemented in the construction industry—and many other industries for that matter that can only use, sell or lease finished products (as opposed to ‘minimum viable products’, a concept from Agile). Just today, I heard a project manager who builds nuclear power stations say: ‘Agile never made sense to me!’.

Currently, Agile is only applied in research, software development, IT, and marketing projects. As far as I have observed, broadly confirmed by the 14^th ‘Annual State of Agile’ report from VersionOne: “The survey results indicate that agility is largely confined to [software] development, IT, and operations.” (page 4). In my estimation, Agile will be applied in at most 50% of our world’s projects and, in a few years, less than that as I see in my crystal ball. Why? I think that the market share of Agile will shrink in the future for two reasons:

• Firstly, executives, sponsors, and investors will never stop asking for forecasts that Agile does not provide, cannot provide, and does not want to provide. Agile is in the school of ‘adaptive project management,’ after all. ‘Agile’ allows the continuous adding of scope to or removing it from the project that is delivered in several fixed periods with a team of fixed size. In this respect, Agile is not the opposite of predictive project management, because none of the predictive techniques prescribes that the scope should be fixed. They use change requests to manage it. Only ‘predictive project management’ (i.e. Critical Path, Resource Critical Path, Critical Chain, Earned Value, Earned Work, etc.) are designed to, and are able to and will always provide time forecasts by identifying the dependencies between the items in the WBS. The key difference between ‘adaptive’ and ‘predictive’ techniques is that all predictive techniques expect you to identify all dependencies between activities (create complete and correct network logic), whereas Agilists are not expected to do that.
• The second reason that I think Agile will shrink in the future is that it is, in its essence, non-committal in terms of finishing the product or service, and in terms of time (deadline) or cost commitments (budget). Agilists will not commit to building a complete product, neither will they tell you in advance how much it will cost, or how long it will take. Agilists will get you to start investing and then keep you going in a way that reminds me of how consulting firms would string their clients along in the 1990’s using contracts of the ‘Time & Materials’ With a healthy dose of sarcasm, ‘Time & Materials’ can be summarized as: “I will work as slow as possible for as long as possible and expect to be paid for this!” Consulting firms knew how to maximize their profits in the 1990’s! Since, Agile is a new incarnation of Time & Materials contracting in my mind, I am afraid that Agile will be pushed back to a much smaller market share in the world within the foreseeable future. It was no surprise that Time & Materials contracting was relegated to the sidelines by the late 1990’s and replaced by ‘Firm Fixed’ price contracts. For ‘Firm Fixed’ price contracts, you need to establish the entire scope and produce accurate predictions. Predictive project management is much better at forecasting than Agile ever will be.

Many people think that ‘predictive project management’ (i.e. Critical Path) is the same as ‘waterfall.’ This is nonsense! ‘Waterfall’ means that, before the project starts, you plan an entire project out in (somewhat) chronological phases. The concept of ‘phases’ feels good, and essentially, ‘phases’ are an arbitrary way to cut up a large piece of bread into smaller slices that you can actually digest. Phases can even overlap in time, because there are no transition criteria that must be met before a project goes from one phase over into the next. This is where Dr. Robert Cooper has improved the phase-based thinking significantly by adding transition criteria to the phases. He captured this in his famous Stage-Gate methodology: Each gate has certain criteria that MUST be met before a project can move to the next stage (phase). Notice that he replaced the word ‘phase’ with the word ‘stage’; he could have called his invention ‘phase-gate’, but he didn’t.

Since ‘Agile’ was a reaction to ‘Waterfall that-should‑never‑have‑existed,’ what does this mean for ‘Agile’ itself? Is it a solution that is in a desperate search for a problem? I do not think so. When I state ‘Waterfall’ should have never existed, I am not saying ‘Agile’ project delivery should never have existed. Agile has its place and will hold its place. Agile fits well in a situation where people are working at the frontier of what people know in its broadest sense (knowledge-constrained projects). Obviously, R&D projects are in the field of knowledge-constrained projects. In software development projects, customers often find it difficult to verbalize all requirements, which imposes a knowledge-constraint to the project manager. If you are interested in the topic of picking the right scheduling approach, please retrieve my (free) White Paper ‘Choosing the Best Scheduling Approach for Your Project’ and look at its Project Ideals and Constraints (PIC) matrix: click here ( https://projectprocorp.com/collections/white-papers/products/choosing-the-best-scheduling-approach-for-your-project ) or buy my textbook ‘Forecasting Programs’ (same website, but under ‘Books’). Both discuss what place in this world all scheduling methods that I’ve mentioned in this article, have, including ‘Agile’.

‘Waterfall’ Should Never Have Existed!

Predictive project management has never used ‘phases’ for as long as I have been at it (soon 30 years!). Ever since the first PMBOK^® Guide was published in 1983, PMI encouraged ‘deliverable-oriented’ Work Breakdown Structures (WBS) rather than ‘phase-oriented’ WBS’s. In contrast to ‘phases’, a ‘deliverable’ is simply defined as ‘a measurable thing of value that someone is willing to pay for’ (source: my textbook ‘Forecast Scheduling’). A ‘phase’ is not measurable, but a ‘deliverable’ is measurable by definition! So, given the fact that ‘Waterfall’ is based on a phase‑oriented breakdown of the work in a project AND ‘phases’ never were the PMI-recommended breakdown-orientation for developing WBSs, I would argue that ‘Waterfall’ never really existed (except in the minds of some software development project managers and their misguided world of Software Development Life‑Cycles (SDLC) of the 1990’s). Hence, I’d like to proclaim: Waterfall is dead! Waterfall should have never existed!

Why do we continue to refer to ‘Waterfall’ if it was a dead-end in the first place? Let’s simply stop talking about ‘Waterfall’; it is a dinosaur now; it is dead! Let’s get rid of it; it’s time to move on!

To Hybrid or Not to Hybrid?

In the past few years, more and more organizations are publicly admitting to the fact that Agile was not implemented as per the Agile textbook, but in a modified way, a mix between Agile and Waterfall-that-should-never-have-existed. Many people are leery of a ‘hybrid‘ approach that indeed raises an important question: Do we end up with the best of both worlds OR the worst of both?

Several project managers have written accounts in which they describe how they ended up with the worst of those two worlds of Agile and Waterfall-that-should-never-have-existed. For example, when there are many conflicting deadlines and the portfolio managers are not setting clear priorities, the autonomy of the agile teams will indeed get compromised: “Do this first instead of your sprint work!” Is this particular case, was it a real ‘hybrid’ approach? Not in my mind! At most, this is ‘Agile’ with a ‘portfolio manager override.’

In my opinion, a true ‘hybrid’ form is a combination of an adaptive (Agile) and a predictive method (like Critical Path). When I realized that Critical Path is just a different approach and not the opposite of Agile, I started thinking about the following important question: Can Agile perhaps be combined with Critical Path in one project schedule? I’ve seen many situations where a project team was asking for Agile while executives asked for forecasts. There is a clear use-case for combining Agile with Critical Path in my experience.    

Can Agile be Combined with Critical Path in One Schedule?

I mentioned already that Agile and Critical Path are NOT each other’s opposite: Agile and Waterfall are opposites, and instead of coming up with some shade of gray that works for everyone, I would like to present a fresh and completely different approach. What if Agile and Critical Path happily existed side-by-side in a single project schedule without affecting each other (much)?

How would that work? Well, there are a few dilemma’s to resolve when you try to combine two approaches that look at first sight very different:

1. The WBS of an Agile schedule is sprint-oriented (adaptive) with user stories within each sprint, whereas the WBS in a Critical Path schedule (predictive) is broken down in a deliverable-oriented way with activities within each deliverable.
2. Agilists estimate in anything except hours, whereas predictive folks need the estimates expressed in hours to create their Gantt Charts and calculate Critical Path.
3. Agilists don’t like identifying all dependencies between activities, but predictive people need all dependencies, the right types of dependencies and all entered correctly into their scheduling application. They need complete and correct network logic.

I have solved the first two dilemmas, and propose that Agilists need to compromise on the third to make combining BOTH approaches in ONE schedule work, which I will describe in Part 2 of this article series!

What do you think, is there a way to use Agile with your team and Critical Path with your executives? Please comment below and look for part 2 of this article, which will lay out solutions to each of these dilemmas we’ve discussed.

Read Part 2 of Waterfall Should Have Never Existed