Multi-stage continuous integration takes advantage of a basic unifying pattern of software development: software moves in stages from a state of immaturity to a state of maturity, and the work is broken down into logical units performed by interdependent teams that integrate the different parts together over time. What changes from shop to shop is the number of stages, the number and size of teams, and the structure of the team interdependencies.

Multi-stage continuous integration is an expansion upon continuous integration, it presumes that you are already following those recommended practices.

The larger and/or more complex the project, the higher the chance that the project becomes unstable. Alerts and broken builds increase as the project grows. Progress decreases and the mainline gets increasingly unstable. The risk of build failure increases exponentially as the number and locations of developers grow.^[2]

Recommended practice #1 edit

Each developer works on their own task. As they make changes, continuous integration is done against that team's branch. If it does not succeed, then that developer (possibly with help from her teammates) fixes the branch. When there is a problem, only that team is affected, not the whole development effort. This is similar to how stopping the line works in a modern lean manufacturing facility. If someone on the line pulls the "stop the line" cord, it only affects a segment of the line, not the whole line.

It is note-worthy that in recent years the "topic" or "feature" branch model has gained in popularity over the team based branch model. See for example the popular Git-Flow branching model ^[3]

On a frequent basis, the team will decide to go to the second phase: integration with the mainline. In this phase, the team does the same thing that an individual would do in the case of mainline development. The team's branch must have all changes from the mainline merged in (the equivalent of a workspace update), there must be a successful build and all tests must pass. Integrating with the mainline will be easier than usual because only pre-integrated features will be in it, not features in-process. Then, the team's changes are merged into the mainline which will trigger a build and test cycle on the mainline. If that passes, then the team goes back to the first phase where individual developers work on their own tasks. Otherwise, the team works on getting the mainline working again, just as though they were an individual working on mainline.

Changes propagate as rapidly as possible, stopping only when there is a problem. Ideally, changes make it to the main integration area just as frequently as when doing mainline development. The difference is that fewer problems make it all the way to the main integration area. Multi-stage continuous integration allows for a high degree of integration to occur in parallel while vastly reducing the scope of integration problems.^[4]

Recommended practice #2 edit

For multi-stage continuous integration, each team must have its own branch.

Multi-stage continuous integration has many advantages:^{[citation needed]}

• When unit tests fail, or a bug is discovered, developers might revert the codebase back to a bug-free state, without wasting time debugging;
• Integration problems are detected and fixed continuously - no last minute hiatus before release dates;
• Early warning of broken/incompatible code;
• Early warning of conflicting changes;
• Immediate unit testing of all changes;
• Constant availability of a "current" build for testing, demo, or release purposes;
• The immediate impact of checking in incomplete or broken code acts as an incentive to developers to learn to work more incrementally with shorter feedback cycles.

Tools that support multi-stage continuous integration include:

• AccuRev^[5] - Version Control and ALM tool
• Electric Cloud^[5] — Build, test and deployment framework tool designed to automate the software production lifecycle
• AnthillPro - Build, dependency, release tool ^[6]
• Rational Team Concert^[7] ALM-Platform

• Agile software development
• Build automation
• Continuous design
• Continuous integration
• Test-driven development
• Application lifecycle management