Why Waterfall was a big misunderstanding from the beginning – reading the original paper

2 Mar

You probably know that there is the Waterfall process – by now feared by most as “the father of the plan driven approach”. Some may even know that Waterfall was supposedly “invented” by the paper “Managing the Development of Large Software Systems” by Dr. Winston W. Royce in 1970. But have you ever read it? Why should you? It’s probably just an endless explanation of how great Waterfall is… or is it?

What if I told you that it is the exact opposite? If I told you that it is a paper identifying this pattern (without naming it Waterfall) and describing its deficiencies and proposing enhancements? Don’t believe me? Well let’s see what Mister Royce writes right after introducing the pattern on page 2:

I believe in this concept, but the implementation described above is risky and invites failure.

Doesn’t sound so positive, does it? Oh and that’s not even the best part! The best part is that the paper actually includes some thoughts that I would classify as agile.

Wait what?

Yes back in 1970 the paper, that supposedly invented Waterfall was describing its deficiencies and actually contained some agile thoughts. I really recommend you to go ahead and read this paper. Or stay with me and let me explain it to you – or do both.

The reason why I’m bringing all of this up is that it seems that all of this is a little known fact. I mentioned it to two of my lecturers (both PhDs) during discussions about agile vs. plan driven. The fact that they didn’t know this kind of shocked me, so I decided that this probably needs some more exposure – and this is all the exposure I can give.

But all is not lost, I actually learned this fact from a lecturer, Dr. Joachim Schnitter, at my home institute – so thanks to him!

The misunderstanding

Royce doesn’t define the Waterfall model and recommend it, he identifies the pattern and shows one of the major problems of the Waterfall model as the testing phase occurs at the end of the development process:

The testing phase which occurs at the end of the development cycle is the first event for which timing, storage, input/output transfers, etc., are experienced as distinguished from analyzed.

He further on states that faults found there will most likely result in a major redesign of the software, he describes the devastating effects of this as:

The required design changes are likely to be so disruptive that the software requirements upon which the design is based and which provides the rationale for everything are violated. Either the requirements must be modified, or a substantial change in the design is required. In effect the development process has returned to the origin and one can expect up to a 100-percent overrun in schedule and/or costs.

To me this analysis is pretty much right on target in describing Waterfalls biggest problem, highlighting that he doesn’t recommend this approach at all. He is suggesting improvements.

So how is this paper agile?

It is certainly not totally agile, but it contains thoughts that go in the right direction and really resemble some current agile practices. Winston Royce proposes 5 enhancements to the “Waterfall” model, they are as follows:

  1. Program design comes first
  2. Document the design
  3. Do it twice
  4. Plan, control and monitor testing
  5. Involve the customer

Let’s focus on the latter 3 as the first 2 aren’t pretty agile to my mind (especially not number 2 as it highly emphasizes documentation):

Do it twice

Royce basically advocates that if your software product is original, the version delivered to your customer should actually be the second version of the product, at least for critical areas. He goes on to explain:

Note that it is simply the entire process done in miniature, to a time scale that is relatively small with respect to the overall effort.

He explains that this is done to identify problems early on, so they can be tackled appropriately later on. All of this sounds like a “spike” to me. You go on to build a part of your system, which you have little knowledge about, in order to increase your knowledge about this part of the system . You do so in isolation and throw the code away afterwards. He says that you should do this, but for the whole system.

Plan, control and monitor testing

In this section he once again argues that it is bad, that the testing comes last. However he does not jump to the conclusion that testing should come first, but he greatly emphasizes the importance of testing for the success of a software project. He also says that code should be checked by a second party, as many errors are easy to spot. I know it’s a very far stretch but that reminds me a bit of one of the motivations for Pair Programming, although he doesn’t suggest anything like it.

Involve the customer

This basically stands for itself as it is also one of the four core values of the Manifesto for Agile Software Development. He more explicitly says:

Involve the customer – the involvement should be formal, in-depth, and continuing.

He also states what it is like when the customer is not involved and the development team is basically left alone:

To give the contractor free rein between requirement definition and operation is inviting trouble.

He suggests 3 points after the initial requirements generation, where “the insight, judgment, and commitment of the customer can bolster the development effort.” These are: Preliminary Software Review (after Prelimnary Software Design), Critical Software Review (after Program Design) and Final Software Acceptance Review (after Testing). Although this is not an on site customer yet, it is much better than “make a contract and then let the contractor work alone in the dark on the system according to the contract” – don’t you think?

So was Winston Royce agile?

Certainly not, but he had some really good ideas on how to improve software development, some of which may be classified as “agile”. But why have most of us never heard of this before?

So how did Waterfall become so popular if the deficiencies were so well-known?

What I write now is just a rumor, nothing more. I’ve no way of telling whether it is true or not and clearly no reference.

So from what I’ve heard sometime some guy of the Department of Defense was given the task to look for a software development process to develop a new system. He found the paper of Royce, read the first page (which is quite positive) and saw the figure at the top of page 2 depicting what we now know as the “Waterfall” model. As it was simple and seemed good he went on to propose the depicted process, without ever reading about the deficiencies and proposed enhancements. Maybe it’s just an urban legend, maybe it’s true – I don’t know. Either way I’m sad that Winston Royce has been so misunderstood.

Lesson learned

It can be very beneficial to go back to the roots and read the original sources, even over 40 years after their initial publication. If some more people would have done this, maybe we would have (mostly) gotten rid of Waterfall as a Software Process by now. However it is still there and going strong. To each his own, but I prefer agile methodologies, practices and associated software development processes. To me they are way more productive, effective and fun. And it is always good to know that a CMMI level 5 company can highly benefit from a switch to Scrum – doubling the productivity and reducing defects by 38% (and many more benefits). I encourage you to go ahead and read about it in the Scrum Papers, the section (paper) is titled “Scrum and CMMI Level 5: A Magic Potion for Code Warriors”. Have fun reading it and I hope you enjoyed this read as well!

9 Responses to “Why Waterfall was a big misunderstanding from the beginning – reading the original paper”

  1. Ezekiel July 31, 2012 at 03:00 #

    Thanks for this excellent article and insight.

  2. 小椋俊秀 (@ogura_toshihide) May 5, 2014 at 04:41 #

    I found who used the term “waterfall” first.

    Bell & Thayer state “The same top-down approach to a series of requirements statements is explained,without the specialized military jargon, in an excellent paper by Royce ;he introduced the concept of the “waterfall” of development activities.In this approach software is developed in the disciplined sequence of activities shown in Figure I. ”

    Title:”Software requirements: Are they really a problem?”
    Authors:T. E. Bell T. A. Thayer
    Published in:· Proceeding ICSE ‘76 Proceedings of the 2nd international conference on Software engineering
    Pages 61-68
    IEEE Computer Society Press Los Alamitos, CA, USA ©1976

    http://dl.acm.org/citation.cfm?id=800253.807650&coll=DL&dl=GUIDE&CFID=266647296&CFTOKEN=26547871

    My report is open to the public.
    http://barrel.ih.otaru-uc.ac.jp/handle/10252/5163
    (Sorry, It’s written by Japanese)

    • PragTob May 26, 2014 at 16:55 #

      Thank you for sharing your findings! Can’t read Japanese unfortunately :(

  3. Ian Edwards March 16, 2015 at 18:17 #

    I liked this article it was an eye operner.

    I’ve now read Dr. Royce’s paper myself, and he never used the term waterfall. He talked about the process he had used, presumably the NASA method, and suggested 5 ways to improve it. Nothing more apart from some very nice illustrative diagrams.

    Having read Bell and Thayer’s paper, which talks about requirements identification problems, they said Dr Royce had introduced the concept of “the ‘waterfall’ of development activities. Why they did that I don’t know other than I think that was their own pet name for the 7 steps in the project delivery plan he used. More a cascade than a waterfall I would suggest, and Royce explained that all those steps could refer iteratively to the previous or the next one as needed, not exactly a one-way flow, right? Royce explain, the purpose of his paper in my view, how problems were encountered when the stage in hand (such as text phase) had to go back two or more steps to get the job done. And he gave 5 recommendations to improve the planning model to make that kind of situation work better or to avoid it in the first place.

    How a two line refereence in an obscure IT paper in 1976 ncorrectly quoted an obscure IT paper written in 1970 would spawn a wholesale industry myth about about a methodology that never was is beyond me.

    ***

    What the 7 Step Planning method Royce used for his nine or so years building space systems was called we don’t know but maybe ‘NASA Project Life Cycle’ would come close.

    Who at NASA invented it back in the 1950’s I can only guess but I imagine it was adapted from the physical engineering world. With physical engineering, such as a bridge, or tower block, it is important to have the requirements and design agreed up front. It is costly to tear down half way through and start again. Royce advcated pilots and micro models to reduce that risk in the software side.

    I think modern day agilers maybe perhaps are not so aware that back then there was little or no computing in businesses and everything was a greenfield start with a turn-key operational delivery governed by the Finance Manager. The ‘NotReallyaWaterfall’ planning tool used by Royce worked well for the big projects done in that era. And he identified how it could be improved too. It’s a little more nuanced today given the legacy systems to retire, interconnectivity/integration expected, and the (my view) unreasonable expectations of modern management to conjure projects in short order and on the cheap with minimum amount of management oversight.

    So whilst projects today may be run differently, I do not think they are done better, regardless of the method used. I think agilers are running to stand still in that sense. No offence intended if anyone disagrees. Managers in my experience sees agile as quicker and cheaper. Plug and play and ‘lets’ not waste any time working out what we want first’ works for them. I reckon you do get features earlier but will still pay the same or more in the long run. And then once a project has delivered, regardless of how it was managed, it starts to age because needs change and maintenance is the hidden cost of solutions, not just the original build. It’s a wonderful world.

Trackbacks/Pingbacks

  1. Failing Fast « Manage to Test - May 3, 2012

    […] won’t pass.  Now that’s fast failure.  In the past, such problems (programmers and testers misunderstanding requirements) might not have gotten caught until much later in the process.   Of course, all this presumes the […]

  2. Dreams of Lean | Vlad Pasman's Blog - October 5, 2014

    […] of the criticism of “MANAGING THE DEVELOPMENT OF LARGE SOFTWARE SYSTEMS” is misguided. And I am not the only one thinking that way. I don’t see anything in it that would not fit a good lean or agile process. It is the people who […]

  3. On Cooking Software and Waterfalls | Gigi Labs - January 13, 2015

    […] Waterfall model a lot more reasonable to use. If you’re too lazy to read the paper itself, this blog post contains some pretty decent commentary about […]

  4. 1p – Why Waterfall was a big misunderstanding from the beginning | Exploding Ads - May 8, 2015

    […] 1p – Why Waterfall was a big misunderstanding from the beginning […]

  5. 1p – Why Waterfall was a big misunderstanding from the beginning | Profit Goals - May 8, 2015

    […] https://pragtob.wordpress.com/2012/03/02/why-waterfall-was-a-big-misunderstanding-from-the-beginning… […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 895 other followers

%d bloggers like this: