Waterfall or Agile?

This page describes both Waterfall and Agile software development philosophies. In reality, developments are not 100% waterfall or agile, but a blend of the two.

This is a ‘back to basics’ look at the two philosophies. Separate pages will consider the implications and necessary modifications when each is applied to different bespoke software or product software projects.

Contents:

• Why yet another page?
• waterfall: modelled on construction projects
  • We have an agreed design and the process to build this is well known.
• agile: modelled on engineering design projects
  • no one has ever done this before
• Will the real agile step forward please?
• hybrids?
• Budgeting
• The Team
• Waterfall Programmer vs Agile Developer.

Why yet another page?

In short, to try to separate the ‘religion’ from this topic, and to step back from the ‘bespoke only’ or ‘product software only’ perspective in almost every other page on this topic I have found.

An example of the ‘religion’ attitude is an article here which suggests Apple computers has poor software quality because they use ‘waterfall’ rather than ‘agile’.  It is almost certain that Apple would use a methodology encapsulating some of both ‘agile’ and ‘waterfall’,  but the implication is that label ‘waterfall’ is itself sacrilege.  Whether any company, (and even perhaps Apple), embraces or does not embrace a particular  software methodology, does not itself automatically deliver sound, or unsound, software development.  In such a large company, it is almost certain there is at least some software developed under each model.

The other issue is to take each concept and envisage only a chosen adaptation for the authors’ field of software.  Not only are bespoke and product software almost polar opposites, but even within each category, there are factors that affect ideal methodology.  So most pages compare only for a specific environment, which is great if all your projects match that description, but less helpful for those with different or varied projects.

Waterfall:  Modelled on constructions projects.

The process of building something (i.e. a house or a bridge) has been practiced and refined over centuries.  The waterfall approach is simply taking this construction model, and applying the model to the building of software.  Using a Gantt chart,  the steps look more like a cascade of waterfalls.  But if you view a cascade from the front (see pic) then you have the appearance of water falling from step to step.  The waterfall is from step to step, not all at once.

The principle of waterfall is that each step is completed before the next begins. A complete understanding of requirements should be in place before design starts.  ‘Building” or development, should not commence without a completed design. Once built, tests verify the building meets the design.  These are the same steps that have been proven to work on construction projects over centuries.

Agile: Modelled on engineering design.

While waterfall considers a software project as similar to a construction project, agile effectively considers the entire software project, as being an engineering design phase.

With engineering design, you have a problem, and the challenge is to find a solution.  With a building such as house, even when a exact identical house exists, if you need another, you still have to build one.  With engineering, if a solution exists, there is no need to engineer the same solution again, as engineering is establishing the design, and software has the same situation: if the design already exists, then there  is nothing to be done.

If it is known that some existing design can be modified to provide a solution, for modified design, all that is require is to make the modifications.  If two buildings are almost identical, you have build them both, completely.

With engineering projects, at the start of the project you often do not know what the final result will be.  An example from the tangible world is the design phase for mass-produced products. You have a team of designers who engineer a design, but although you may have a fixed budget and time frame, it is only at the end of the design phase that you know what the final product will look like.

The principles of agile can be considered be:

• what is being created is new and innovative and not an exact replica of an existing program
• new ideas will need prototyping and revision
• the best solution is not always clear at the outset
• actual product requirements may not even be clear until the product is trialled

The process of agile software development can be considered as:

• define an initial set of requirements
• build an as simple as possible prototype as quickly as possible
• now repeat the following steps until satisfied:
  • analyse what how the prototype could be improved, and add those improvements to requirements
  • analyse steps that can be made within 1-2 weeks to meet more requirements and perform those steps

Note: this is representing the agile philosophy itself. Any actual implementation of agile will also depend on some other choices as to how agile will be implemented.

Consider a car manufacturer who produces a new model of a certain car every year.  The design team, with an allocated number of people and resources, is given a fixed time to come up with next years design.  Agile development is often similar to this process, with major new software releases on fixed intervals, with the new features available at that time.

There was a core team who produced the original agile manifesto. The only ‘real agile’ is what is described in that manifesto.  Anything else is an evolution, a derivation or someone’s idea of how to achieve either, those original goals, or an evolution or derivation of those goals.  I have heard “that are so many different types of agile”, and while there can be many different interpretations of those goals, most “different types of agile” seem to be nothing to do with those goals.  The allowance of the specification to continually change during development and continuous delivery are the core of those goal, as is self organising teams, and these are probably the key test of ‘agile at all or not’.  This clearly rules out the classic waterfall: where  specification fixed before coding and where delivery only follows acceptance testing,  and teams are managed by project managers, not self managed.

Agile is not a registered brand.  So anyone can go back to meaning of the word ‘agile’ and say something ‘we are actually agile because we hot desk’ in that they have some practice linked to the dictionary definition of agile, without that practice being related to the work of the agile group.  Sort of like coming out with a new theory of relativity, that has nothing to do with light, gravity or matter, but given how children look like their grandparents because they are related to each other.   The problem is that agile for projects has become associated with the principles of the original group, (just as the word relativity with the work of Einstein), and this does seem to me as appropriating the reputation of that original group. To me, anything that still follows the original principles, even if adding more ideas, can still be called agile.  Anyone not following the original principles, to avoid misappropriation, should really either use a new term or simply say ‘modified agile’.

Hybrids? Solutions using a combination.

Waterfall mindset: We have a specification and the process to build this is well known.

With construction projects, you could be building an exact replica in an effectively identical site, but surely there will be something new?  In fact, the more daring the building, the more often that some part of the project will actually encompass an engineering design project, even with prototypes!  The fact that almost every waterfall project encounters some steps which better fit the agile pattern, is the reason so many waterfall projects suffer time and cost blowouts.

Agile mindset: No one has ever done this before.

Unlike a building, if making a copy of software does not need a new project.  Just with engineering, there is no problem to solve if the solution already exists.  But, surely someone has done part(s) of it before?  Won’t there are least be construction projects to build the prototypes?

The reality is that both agile and waterfall can start to approach a series of short waterfalls placed back to back. Each just approaches from the opposite side.

A very short waterfall stage loses many of the characteristics of waterfall, and a long build of a single prototype becomes more waterfall.  In the end, no project is ‘pure’ and all have elements of both.  Software is always doing something new, but just how new?

The more new, the more no one has genuinely done anything like this before, the greater the need for an agile mindset.  The more ‘just like a previous one but with these straightforward but time-consuming changes‘, the more a waterfall mindset may work.

Budgeting

No one can predict the exact cost of an exactly specified system containing elements that no one has ever done before.

As waterfall is based on the construction model, the budgeting process it quite like getting quotes having a house built to match the plans you have had drawn up.  You can have faith you will get something that matches the plans, even if it may be a little late and the real problems only occur if you realise during construction that result will not match what you actually need.

Agile is more like working with the architect.  You may have a fixed quote but you have to commit blindly without knowing in advance if you will like the result.  With agile, you can control the budget, but at the expense of accepting what is produced within that budget.

The result is that many projects are forced into a waterfall mindset from the budgeting phase, and can only live with an agile mindset for the limited spend of the design phase.

The Team

Just as waterfall breaks a project into steps, it separates the roles needed for those steps.  Consider construction projects. Design people are expensive, but during ‘construction’ you may only need labourers who cost less.  Similarly, a waterfall methodology separates project managers, annalists, and programmers. Programmers who just program become lowest cost resource, and can be more easily outsourced and managed by project managers.

Agile requires the whole team act like engineers.  As the work is always different even the construction has less guidance and more decisions on the fly by the ‘engineer’.  The result is more costly resources and a much flatter team structure.  The team project manages themselves using scrum or other methodology, but this requires additional skills.

Waterfall Programmer vs Agile Developer

In Waterfall, a model with a parallel to construction projects, the programmer takes the role of the labourer in construction projects.  The lower cost workers do the labour to turn the design created by the designers into reality.

There was a time when managers and even clerks never typed, as there were specialist typists to the typing.  But the role of typist is no more as now clerks, managers and even the CEO will usually do their own typing.

In the tech world of agile continuous development, the waterfall programmer has gone the way of the typist.  In this world everyone can program, even the CEO.  Programming is no longer an occupation, but a skill.  Just as now an author now needs to be able to type, but their job is not just typing, an agile developer needs to be able to code, but their job is not just coding.  A developer mainly needs to be able to design solutions.  Those solutions will need to be turned into code, but the role is more how to test the solution, code the solution and make it maintainable.  A waterfall programmer is not automatically ready to be agile developer.

The evolution of the programmer is described in this summary of a video by ‘Uncle Bob’, one of the original authors of the agile manifesto.