Execution: Waterfall, Agile, Lean & Nested Planning

To listen to proponents of the various disciplines, you could often believe that one or the other discipline is right while the others are either severely out of date or just inherently wrong. In actual fact, the forms are complementary. Each has its role in the process of program and project management. And each works for processes and projects.

Waterfall is the earliest discipline, developed around restricted resources and capabilities. As systems have become more complex, and environments have become more flexible, the ability to move to an Agile approach became more feasible. Lean is a more holistic approach that helps provide an overall structure in which a work environment exists. Nested Planning is being introduced to impress the many levels of planning and the interdependency of the levels. As a result, topics are:

Waterfall
Agile
Lean
Nested Planning
Putting It Together

Waterfall

Responsible for some of the largest projects in history, including placing a man on the moon, Waterfall provides a means of managing a project in a fashion that allows for the tracking of all of the pieces needed for a project to be successful. While often portrayed as a very sequential process (Requirements, Design, implementation, Testing, Installation, Operations and Maintenance), in fact it demonstrates much more flexibility. Operations are frequently in parallel. Subprojects are the rule rather than the exception. Flexibility is in the management of the process, not restricted by its structure. It is the maturation of the Waterfall process that leads to concepts like Dependencies and their tracking, Lead Times, Prototyping, Tiger Teams, Time Boxing and others that lead to the principles now used in Agile. Improvements in overall management lead to the concepts of Lean Management. In general, the ability to manage across large numbers of projects and teams still fall back on the concepts of the Waterfall process, especially the Gantt Chart. And while a lot of work accomplished in software development is better accomplished in an Agile approach, many are still done with Waterfall processes. Most notably the implementation of infrastructure and packaged software, still required mainstays in most organizations.

Waterfall did establish a lot of needed structure over time. Disciplines such as Governance, Testing, Risk Management, and User Acceptance processes were all established as part of the Waterfall process. Unfortunately, an inherent inflexibility was established that did not have to happen. Lead times of 18 months could be required to provide changes to systems. These occurred because it was easier to be safe with a process that was predictable rather than address the flexibility needed for IT to be responsive to the real world. The approach taken by the Waterfall process enabled a “contract” to be established between the business and IT. Unfortunately, this approach to systems development caused a lack of flexibility in the process.

Return to Top

Agile

Agile tries to address some of the more severe limitations of Waterfall, as it is generally practiced. There are some practices that have proven especially advantageous as more interactive systems are being developed. One of these is the development of the User Experience (UX) or Human-Centered Design (HCD). Understanding the business process more thoroughly and having individuals who need to work with the interface helps to make for better design. It also presses for full-time involvement from the user since the best results come from active user involvement throughout the process.

Faster iterations are another hallmark of Agile. The concepts of Continuous Integration (CI) and Continuous Delivery (CD) have made this possible. However, this is accompanied by the concept of DevOps or DevSecOps where the Development, Security and Operations aspects of systems are much more closely coordinated. Again, this is enhanced with the Cloud implementations that provide much more flexibility in making development environments available. And, of course, releases should not be done unless tested, so the much improved automation of testing has made this possible, especially for regression testing.

With more frequent releases, planning can now be evaluated on a continual basis. And while Value-Driven development is not new, there is a higher level of use, especially since priorities can now be regularly evaluated. The idea is to develop that Minimal Viable Product (MVP) so that the product can be provided as soon as possible. Generally, development teams are kept small (8-10) (also called Scrum Teams) so that they an be self-directed. These teams depend on all of the members to help solve the problems, not restricted to individual capabilities. They use the full capabilities of the team, instead of maybe pigeon-holed team member assignments. Delivery and activities are all time-boxed; that is, kept within specific timeframes (as short as two weeks). For larger projects, there may be multiple scrum teams, where teams may gather together to keep synchronized. Since Scrum teams are self-directed, there are no real project managers. Though concepts like Value Train Engineers may exist to maintain coordination across teams. Agile also continually uses retrospectives to perform lessons-learned sessions to identify improvements and then actively incorporate the recommendations into their process.

Bottom line, the concept of Agile is to significantly shorten the time it takes to release usable code, in many cases from months to days or hours. This comes from significant improvements in technology (e.g., automated testing and cloud), coordination (DevOps/ DevSecOps) and emphasis on users (involvement throughout the project) and design (UX and HCD).

Return to Top

Lean

Lean, originally built for manufacturing, is built around five principles:

Define Value – Identify the value to be obtained from the process
Map the Value Stream – identify the steps and eliminate those that do not provide value to the process
Create Flow – After removing the waste, identify the final process flow
Establish Pull – Customer pulls the value as needed. Think Just-In-Time processes
Pursue Perfection – Make continuous process improvement part of your culture

These steps provide the planning process around developing your planning and priorities.

While we are here, we might want to discuss the 8 Wastes identified in Lean processes. Remember Lean started in manufacturing, however, there are analogies we should pay attention to:

Defects – Often metrics have a tendency to measure “production”. In Agile, there is a tendency to have frequent releases. However, it is important to ensure that the releases are of high quality. Approaches exist for accomplishing this goal, but metrics are needed to monitor this goal.
Overproduction – Keep the code tight. And evaluate reusability. There are often needs to provide code or services that are consistent among applications. Consideration should be seriously given to potential reuse of the code within and across applications.
Waiting – This can occur many ways. Not paying attention to coordination and lead times, so processes or resources are not ready on time? Especially in the area of infrastructure, there is a need to identify lead times for delivery so that interdependent deliverables are not losing time waiting for needed resources.
Not Utilizing Human Resources – In Agile, the use of wholly contained Scrum teams actually makes better use of the Human Resources if done properly. Since the whole team is responsible for the deliverables, team members actually stretch beyond their comfort zones to help solve problems.
Transportation – As code moves to production, efficient approaches to doing regression testing and implementing code greatly assists in effective and efficient implementation.
Inventory – All systems have backlogs. Constant pruning allows the management and teams to focus on what is important. Once items enter the development cycle, they should go to completion. Doing too many things at one time results in unfinished work (in-process items) and “excess inventory”.
Motion – Constantly review your processes and eliminate efforts that are not delivering results. However, those who would like to eliminate management reporting as unproductive, remember that transparency is an essential part of Agile and project management as a whole. You can argue about how reporting is done, but not that it needs to be done.
Excess Processing – There is a concept of “Minimum Viable Product” in the Agile world. What is essential to get the product or service out the door? “Added value” can be added after the necessities are in place.

Return to Top

Nested Planning

Planning is a multi-layered effort. The implementation of a plan should always be done in the context of the needs of the organization and its customers. Some organizations may be so large that individual divisions may start their own planning independent of the total organization, though all are done in the context set by the top of the organization.

The characteristics of the layers change as you move into the inner layers. Timeframes for execution shorten as the tasks move from high level general execution to lower level specific deliverables. Strategic planning may be an annual or semi-annual event. Scrum meetings are daily. The layers include:

Strategic Planning – Strategic planning is done at the highest level of the organization. Generally, this includes the definitions of the customers, the general goals of the organization and a general identification of products, services and value streams. At the same time, some effort is needed at looking internally to identify “enablers” which may include infrastructure, and internal capabilities that will enable the capabilities of the organization.
Product/ Value Streams – Whenever we talk about products, we are including services in the discussion. At all times, we are identifying the value streams that result, since this is the build up to the value of the organization to your customer and clients. The value streams may be defining multiple services and products. At all times, it includes the impact to the client and how that is built out (at the high level).
Product Roadmaps – Having identified the value streams, we now need to develop roadmaps for achieving these values. In many cases, it may be in refining and adding to the list of products and services, and then defining how we progress to the desired goals. Product backlogs will be developing here.
Release/ Program Increment Plan – Having identified roadmaps, we are now beginning to proceed to execution. Given the related lead times for development and the available resources, how does the work proceed. Remember, the objective is to deliver early and often. So, we are looking at the concept of the Minimal Viable Product (MVP). So what can be done to provide value as early as possible? Organization of the Product backlogs is being established here.
Iteration/ Sprint Plan – The sprint backlog is defined as well as the timeboxing of individual tasks. While an Iteration or Sprint is more of a Agile concept, in a Waterfall world, you are identifying the sequences of development and execution as well. It is important to identify specific tasks that are the focus of implementation. Disbursing across too many activities dilutes the focus and the execution.
Daily Plan/ Scrum Meeting – This is basically a daily review of events. This is a good practice for Agile or Waterfall. The objective is to ensure planned work is moving on a daily basis so that issues are identified and resolved quickly. Without this 15 minute daily exercise, issues may get ignored until they are bona fide problems.

Return to Top

Putting It Together

Putting this together, we will look at the Waterfall, Agile and Lean approaches and apply them to the Nested Planning. In general, all of the areas apply to the various levels of the Nested Planning. It just may vary somewhat in approach.

Waterfall

Portfolio/ Program/ Project Management – Starting with Strategic Management on down, the organization needs to understand that it is managing various levels of activity. And more importantly, it needs visibility to the progress of the activities. Especially with complex endeavors, with the levels of dependencies, the sooner management knows of the roadblocks that are developing, the sooner it can address the issues.
Risk – Risk is a situation with an exposure to danger. And identifying and tracking these areas is critical for resolving them before they become major roadblocks. Each level of planning will identify its own set of risks. But all risks will be visible to the whole organization. Funding, technology availability, cooperativeness among departments, availability of skill sets are just a few areas. As implementation begins, identification of budget or time overruns will occur as actual work exposes issues.
Stakeholders – Who are the people that need to be involved? Who has an interest? Where do you need commitments? These need to be identified so that proper communications are developed. “In the old days”, requirements were collected and the user saw the results when the work was completed. This was, more often then not, a failure. The world changes too frequently. What the analyst understood was not what the user meant. Human/ user interfaces need to be exercised as they are developed, since what looks good in the first discussion frequently fails in reality. Commitment by stakeholders must be made throughout the project. Even in the period dominated by Waterfall projects, the assignment of full-time stakeholders was a critical element to the success of projects.
Dependencies/ Lead times – Applications are continually more complex. Especially when considering Continuous Integrations and Continuous Deliveries. The need to identify dependencies among development teams, among various applications, between the application and the process development, as well as the implementation of infrastructure is critical to success. After identifying the dependencies and charting them, one should do their best to eliminate them, in some cases creatively. When dependencies can be removed, delays can be avoided.
Communications – The willingness to accept a system is very dependent on the need to understand. What is the user getting? When are they getting it? Is management aware? What are the business implications and are expectations going to be be met. A communications plan will be necessary to ensure that everyone is properly informed throughout the process.
Controls – As systems are developed, it is essential that various levels of controls are included: within the system and in the running of the development. Financial systems need to have their checks and balances. Volume information must be reliable and provide consistent data whether the data is collected in one direction or the other. Interfaces need to be able to show all data received were actually acted on. And projects need to be able to report resources and dollars spent to control costs.
Budget – Beginning with the establishment of the Value Streams, the organization needs to be able to allocate resources and dollars that can be monitored to ensure effective use. These are distributed as you move to the various levels.
Quality Management – What steps are being taken to ensure that a viable and sustainable system is being implemented. Frequent implementations mean that efficient and effective testing is accomplished. What is being done to validate that changes are not breaking the system elsewhere? One approach is effective regression testing, which uses previous tests to verify operations remain consistent from one release to the next.
Metrics – What is success? How is it measured? In some cases, it is meeting the schedule. In others it is how low error rates are for the newly installed code. Type of use by the system may be a metric? What enables management to determine, in fact, that an endeavor is successful?

Agile

The areas contributed by Agile:

Multi Disciplinary/ Self-Managed Teams – In Waterfall, teams were generally single discipline and much larger. Agile assumes much smaller, and more numerous, teams that can address all of the challenges of the pieces of work that they have. Extending this to the outer levels of planning, we would look at IT partnering with business level management to form joint teams to resolve issues and manage the portfolios, programs and projects. In general, in the small Agile teams, the members are co-located. However, where this is not possible, tools are now available to provide “virtual co-location”.
Stakeholder Integration – The most successful projects are not completed by IT staff alone. The most successful projects are a partnership between the business and IT with full-time allocation resources from the business as well as IT. In the outer levels, this means an allocation of management time to oversee the direction of the projects.
Continuous Integration/ Continuous Delivery – This process is relatively new. Part of making this possible is the use of the cloud, which means that resources can be dynamically added and reduced as necessary. Continuous integration means that new code is added by the end of each iteration to the body of code. Continuous Delivery means it is available to the customer/ client/ user. This means value is coming faster. It also means that defects might be able to be addressed with a quicker turnaround.

Lean

Lean, as expressed above, has a number of elements. However, for the purposes of this discussion we will include the following:

Defining Value – Starting with the Strategic Management, there is a defining of value to be delivered. As you move through the levels, the discussion is how to move from the the desired value to the implementable value. At the same time, you are trying to eliminate steps that have little to no value.
Minimize Work – Basically, what is the shortest distance between you and the goal? How do you start delivering as soon as possible so that value can begin to be obtained?
Optimize Value of Human Resources – This is not just making sure everyone is getting as much work completed as possible. It is also getting the best value. Especially in self-managed teams, members can contribute beyond their designated specialties in identifying solutions. In other words, are we taking advantage of all of the skill sets a person may have?
Continuous Improvement – As you go through each task, make time for self-reflection. What is not working? What could be improved? What do we need to add? Actively ask these questions at regular intervals. For Agile Teams, it should be at the end of each iteration. At the implementation of the deliverable. After a review meeting. After a planning session. This should be a no-fault discussion. You are trying to make things better for everyone. But most important, after you identify items, you actually have to implement the change. Look at this as self-maintenance. If you do this continually, things get better over time. If you make the list and wait for a later date, it never gets done, or if it does get done, it will be a massive burden.