You probably know that there is the Waterfall process – by now feared by most as “the father of the plan driven approach”. Some may even know that Waterfall was supposedly “invented” by the paper “Managing the Development of Large Software Systems” by Dr. Winston W. Royce in 1970. But have you ever read it? Why should you? It’s probably just an endless explanation of how great Waterfall is… or is it?
What if I told you that it is the exact opposite? If I told you that it is a paper identifying this pattern (without naming it Waterfall) and describing its deficiencies and proposing enhancements? Don’t believe me? Well let’s see what Mister Royce writes right after introducing the pattern on page 2:
I believe in this concept, but the implementation described above is risky and invites failure.
Doesn’t sound so positive, does it? Oh and that’s not even the best part! The best part is that the paper actually includes some thoughts that I would classify as agile.
Wait what?
Yes back in 1970 the paper, that supposedly invented Waterfall was describing its deficiencies and actually contained some agile thoughts. I really recommend you to go ahead and read this paper. Or stay with me and let me explain it to you – or do both.
The reason why I’m bringing all of this up is that it seems that all of this is a little known fact. I mentioned it to two of my lecturers (both PhDs) during discussions about agile vs. plan driven. The fact that they didn’t know this kind of shocked me, so I decided that this probably needs some more exposure – and this is all the exposure I can give.
But all is not lost, I actually learned this fact from a lecturer, Dr. Joachim Schnitter, at my home institute – so thanks to him!
The misunderstanding
Royce doesn’t define the Waterfall model and recommend it, he identifies the pattern and shows one of the major problems of the Waterfall model as the testing phase occurs at the end of the development process:
The testing phase which occurs at the end of the development cycle is the first event for which timing, storage, input/output transfers, etc., are experienced as distinguished from analyzed.
He further on states that faults found there will most likely result in a major redesign of the software, he describes the devastating effects of this as:
The required design changes are likely to be so disruptive that the software requirements upon which the design is based and which provides the rationale for everything are violated. Either the requirements must be modified, or a substantial change in the design is required. In effect the development process has returned to the origin and one can expect up to a 100-percent overrun in schedule and/or costs.
To me this analysis is pretty much right on target in describing Waterfalls biggest problem, highlighting that he doesn’t recommend this approach at all. He is suggesting improvements.
So how is this paper agile?
It is certainly not totally agile, but it contains thoughts that go in the right direction and really resemble some current agile practices. Winston Royce proposes 5 enhancements to the “Waterfall” model, they are as follows:
- Program design comes first
- Document the design
- Do it twice
- Plan, control and monitor testing
- Involve the customer
Let’s focus on the latter 3 as the first 2 aren’t pretty agile to my mind (especially not number 2 as it highly emphasizes documentation):
Do it twice
Royce basically advocates that if your software product is original, the version delivered to your customer should actually be the second version of the product, at least for critical areas. He goes on to explain:
Note that it is simply the entire process done in miniature, to a time scale that is relatively small with respect to the overall effort.
He explains that this is done to identify problems early on, so they can be tackled appropriately later on. All of this sounds like a “spike” to me. You go on to build a part of your system, which you have little knowledge about, in order to increase your knowledge about this part of the system . You do so in isolation and throw the code away afterwards. He says that you should do this, but for the whole system.
Plan, control and monitor testing
In this section he once again argues that it is bad, that the testing comes last. However he does not jump to the conclusion that testing should come first, but he greatly emphasizes the importance of testing for the success of a software project. He also says that code should be checked by a second party, as many errors are easy to spot. I know it’s a very far stretch but that reminds me a bit of one of the motivations for Pair Programming, although he doesn’t suggest anything like it.
Involve the customer
This basically stands for itself as it is also one of the four core values of the Manifesto for Agile Software Development. He more explicitly says:
Involve the customer – the involvement should be formal, in-depth, and continuing.
He also states what it is like when the customer is not involved and the development team is basically left alone:
To give the contractor free rein between requirement definition and operation is inviting trouble.
He suggests 3 points after the initial requirements generation, where “the insight, judgment, and commitment of the customer can bolster the development effort.” These are: Preliminary Software Review (after Prelimnary Software Design), Critical Software Review (after Program Design) and Final Software Acceptance Review (after Testing). Although this is not an on site customer yet, it is much better than “make a contract and then let the contractor work alone in the dark on the system according to the contract” – don’t you think?
So was Winston Royce agile?
Certainly not, but he had some really good ideas on how to improve software development, some of which may be classified as “agile”. But why have most of us never heard of this before?
So how did Waterfall become so popular if the deficiencies were so well-known?
What I write now is just a rumor, nothing more. I’ve no way of telling whether it is true or not and clearly no reference.
So from what I’ve heard sometime some guy of the Department of Defense was given the task to look for a software development process to develop a new system. He found the paper of Royce, read the first page (which is quite positive) and saw the figure at the top of page 2 depicting what we now know as the “Waterfall” model. As it was simple and seemed good he went on to propose the depicted process, without ever reading about the deficiencies and proposed enhancements. Maybe it’s just an urban legend, maybe it’s true – I don’t know. Either way I’m sad that Winston Royce has been so misunderstood.
Lesson learned
It can be very beneficial to go back to the roots and read the original sources, even over 40 years after their initial publication. If some more people would have done this, maybe we would have (mostly) gotten rid of Waterfall as a Software Process by now. However it is still there and going strong. To each his own, but I prefer agile methodologies, practices and associated software development processes. To me they are way more productive, effective and fun. And it is always good to know that a CMMI level 5 company can highly benefit from a switch to Scrum – doubling the productivity and reducing defects by 38% (and many more benefits). I encourage you to go ahead and read about it in the Scrum Papers, the section (paper) is titled “Scrum and CMMI Level 5: A Magic Potion for Code Warriors”. Have fun reading it and I hope you enjoyed this read as well!
Journeys of a not so young anymore Software Engineer 
Thanks for this excellent article and insight.
I found who used the term “waterfall” first.
Bell & Thayer state “The same top-down approach to a series of requirements statements is explained,without the specialized military jargon, in an excellent paper by Royce ;he introduced the concept of the “waterfall” of development activities.In this approach software is developed in the disciplined sequence of activities shown in Figure I. ”
Title:”Software requirements: Are they really a problem?”
Authors:T. E. Bell T. A. Thayer
Published in:· Proceeding ICSE ‘76 Proceedings of the 2nd international conference on Software engineering
Pages 61-68
IEEE Computer Society Press Los Alamitos, CA, USA ©1976
http://dl.acm.org/citation.cfm?id=800253.807650&coll=DL&dl=GUIDE&CFID=266647296&CFTOKEN=26547871
My report is open to the public.
http://barrel.ih.otaru-uc.ac.jp/handle/10252/5163
(Sorry, It’s written by Japanese)
Thank you for sharing your findings! Can’t read Japanese unfortunately 😦
I liked this article it was an eye operner.
I’ve now read Dr. Royce’s paper myself, and he never used the term waterfall. He talked about the process he had used, presumably the NASA method, and suggested 5 ways to improve it. Nothing more apart from some very nice illustrative diagrams.
Having read Bell and Thayer’s paper, which talks about requirements identification problems, they said Dr Royce had introduced the concept of “the ‘waterfall’ of development activities. Why they did that I don’t know other than I think that was their own pet name for the 7 steps in the project delivery plan he used. More a cascade than a waterfall I would suggest, and Royce explained that all those steps could refer iteratively to the previous or the next one as needed, not exactly a one-way flow, right? Royce explain, the purpose of his paper in my view, how problems were encountered when the stage in hand (such as text phase) had to go back two or more steps to get the job done. And he gave 5 recommendations to improve the planning model to make that kind of situation work better or to avoid it in the first place.
How a two line refereence in an obscure IT paper in 1976 ncorrectly quoted an obscure IT paper written in 1970 would spawn a wholesale industry myth about about a methodology that never was is beyond me.
***
What the 7 Step Planning method Royce used for his nine or so years building space systems was called we don’t know but maybe ‘NASA Project Life Cycle’ would come close.
Who at NASA invented it back in the 1950’s I can only guess but I imagine it was adapted from the physical engineering world. With physical engineering, such as a bridge, or tower block, it is important to have the requirements and design agreed up front. It is costly to tear down half way through and start again. Royce advcated pilots and micro models to reduce that risk in the software side.
I think modern day agilers maybe perhaps are not so aware that back then there was little or no computing in businesses and everything was a greenfield start with a turn-key operational delivery governed by the Finance Manager. The ‘NotReallyaWaterfall’ planning tool used by Royce worked well for the big projects done in that era. And he identified how it could be improved too. It’s a little more nuanced today given the legacy systems to retire, interconnectivity/integration expected, and the (my view) unreasonable expectations of modern management to conjure projects in short order and on the cheap with minimum amount of management oversight.
So whilst projects today may be run differently, I do not think they are done better, regardless of the method used. I think agilers are running to stand still in that sense. No offence intended if anyone disagrees. Managers in my experience sees agile as quicker and cheaper. Plug and play and ‘lets’ not waste any time working out what we want first’ works for them. I reckon you do get features earlier but will still pay the same or more in the long run. And then once a project has delivered, regardless of how it was managed, it starts to age because needs change and maintenance is the hidden cost of solutions, not just the original build. It’s a wonderful world.
Did you even read Royce’s paper, or is your article an attempt at meta humor (claiming Waterfall is the product of someone who didn’t read the article, when you didn’t read the article yourself)? If you read Royce’s paper, you’ll see that points #1 and #2 are both distinctly pro-waterfall, that #3 isn’t at all what you represent it to be, and that Royce says that he supports Waterfall, but simply feels it needs some modifications.
Hi there, I did read the article multiple times (although by now a couple of years ago, if you have a look at the date).
The misunderstanding is that he correctly identified the short comings of waterfall and proposed improvements, which were never implemented but they lead into an agile direction.
You’re essentially correct in your take on Royce’s observations regarding the so-called (and much maligned) waterfall model of the software development lifecycle. Royce’s observations were noted by others in the software development industry at that time (as was I with TRW), and led to improvements such as RUP, Barry Boehm’s spiral model, and any number of recommended modifications that DoD adopted. There are plenty of successful projects that were developed following the waterfall and other similar models, especially one-of-a-kind, large scale systems with a well-defined problem space. Most of these project DID involve iterative methods and prototyping (in the case of interactive systems) that were integrated into the waterfall framework. Yes, there are limitations, but agile has become more of a “religion” these days, ignoring other useful approaches (IMHO). For example, agile has mixed results at scaling up to very large projects and in providing sufficient documentation for rigorous contractual arrangements.
I agree with the statement “agile has mixed results at scaling up to very large projects.” I would like to add that traditional methods have a solid track record of failure at very large software development projects.
Actually, he doesn’t alter the so-called “Waterfall” model (Figures 2, 3, and 4) much, he simply adds more BDUF in an effort to mitigate the parts that “invite failure”. He even states, a few paragraphs down, “However, I believe the illustrated approach to be fundamentally sound. The remainder of this discussion presents five additional features that must be added to this basic approach to eliminate most of the development risks.” Consequently, I think it’s slightly revisionist to wave Royce’s paper about as though he was really just misunderstood and was, in fact, a closet Agilist all along.
Hi there,
I never claimed for Winston Royce to have been agile. To cite myself (well from almost 4 years ago) “So was Winston Royce agile? Certainly not, but he had some really good ideas on how to improve software development, some of which may be classified as “agile””.
Without re-reading my post now, I claimed that he didn’t really invent the waterfall model as it went on to “succeed” (or wreak havoc). He identified the model along with some crucial faults and proposed some fixes, many of which share the same values as modern agile techniques. And as you cited he said “MUST be added” but in practice were never or rarely added. Therefore I think he was largely misunderstood, as in my opinion the paper focuses more on the improvements than the actual base process.
In the end, I’m happy that this post is till being read 🙂
I would say he was much more agile than many agilists today. Why?. His article is the result of his retrospective view on the process he was using, building, improving. He thinks, he tries to improve, he is humble, he challenges the status quo, he constructively proposes improvement actions.
What is even more, nowhere in the article he uses the word waterfall, that was added by the people who ran with the picture on the second page without reading it.
Thank you for sharing your findings. You may find that the DoD move to “waterfall” was driven by DoD Directive 5000.29 and STD 2167, which specified.
“…Directive 5000.29 and STD 2167, as interpreted, have made it difficult to apply these modern [evolutionary, rapid prototyping and incremental development] methods.
Although some parts of the recent Draft DOD-STD-2167A appear to encourage modern methods, the draft as a whole continues to reinforce exactly the document-driven, specify then-build approach that we believe causes so many of DoD’s software problems.” (Defense Science Board, 1987).
Report of the Defense Science Board Task Force on Military Software, Unclassified.
SEPTEMBER 1987
The Pentagon, Roon 3D1020
Washington, D.C. 20301-3140
(Note the “Defense” is American spelling, not me).
Re: “Scrum”, the name was plagarised from Takeuchi and Nonaka’s 1986 paper “The new product development game”, repackaged, and commericalised (see the “inventor’s” website for a sanitised version of the truth). Work backwards through referenced articles in reputable papers’ bibliographies and you will find there is a long and varied history associated with iterative and incremental development (what it really is), and the Manifesto is little more than a marketing tool used to mobilise the uninformed to hand over their hard earned cash for an “Agile” certificate.
One should also read these Resources on that topic:
a) https://pdfs.semanticscholar.org/1684/bc535f5b81ad0bc2ea11ce94471027df1089.pdf – or search Google Scholar for “Misconceptions of the Agile Zealots”
b) http://herdingcats.typepad.com/my_weblog/2015/09/the-water-fall-myth.html
Yes, Royce is sometimes erroneously blamed/credited for single-pass waterfall when he was actually trying to propose an alternative (which may have been better, but still not very agile).
The technique of outlining dependent phases as boxes connected by lines traces back at least as far as H.L. Gantt from the Frederick Taylor era, doesn’t it?
For those coming to this in 2018 and beyond. The link in the article is forbidden, but I believe I found the paper here as well: http://www-scf.usc.edu/~csci201/lectures/Lecture11/royce1970.pdf
Reblogged this on Mohamed Guda.