The epiphany of integration points is that they control product development. They are the leverage points to improve the system. When timing of integration points slip, the project is in trouble.

—Dantar Oosterwal, The Lean Machine

Continuous Integration

Continuous Integration (CI) is the process of taking features from the Program Backlog and developing, testing, integrating, and validating them in a staging environment where they are ready for deployment and release.

CI is the second aspect in the four-part Continuous Delivery Pipeline of Continuous Exploration (CE), Continuous Integration (CI), Continuous Deployment, and Release on Demand (Figure 1).

Details

Continuous integration is a critical technical practice for each Agile Release Train (ART). It improves quality, reduces risk, and establishes a fast, reliable, and sustainable development pace.

With continuous integration, the “system always runs,” meaning it’s potentially deployable, even during development. CI is most easily applied to software solutions where small, tested vertical threads can deliver value independently. In larger, multi-platform software systems, the challenge is harder. Each platform has technical constructs, and the platforms must be continuously integrated to prove new functionality. In complex systems comprised of software, hardware, and components and services provided by suppliers, CI is harder still. But the fact remains: integrating and testing components together frequently is the only practical way to fully validate a solution.

As a result, teams need a balanced approach, one that allows them to build quality in and receive fast feedback on their integrated work. For purely software-based solutions, continuous integration is relatively easy to achieve with modern tools. For more complex systems composed of hardware and software, a ‘continuish integration’ approach is required (see the Enterprise Solution Delivery article) to balance the economic trade-offs between frequency, the scope of integration, and testing.

The Four Activities of Continuous Integration

As illustrated in Figure 2, SAFe describes four activities associated with continuous integration:

• Develop describes the practices necessary to implement stories and commit the code and components to version control
• Build describes the practices needed to create deployable binaries and merge development branches into the trunk
• Test end-to-end describes the practices necessary to validate the solution
• Stage describes the practices necessary to host and validate the solution in a staging environment before production

Develop the Solution

Developing the solution refers to the implementation of stories by refining features from the Program Backlog as may be needed and then coding, testing, and committing the work product into the source control system. Testing in this activity tends to focus on unit and story-level testing and most times requires test doubles (see Test-Driven Development) to replicate other components or subsystems.

Seven practices are associated with developing the solution:

• Break features into stories – Splitting features into stories enables continuous delivery via small batches and smooth integration. This may include creating user story maps to ensure that workflows are designed to meet customer needs.
• Behavior-Driven Development (BDD) –BDD is a process Product Owners and teams use to more thoroughly understand requirements and improve quality by creating acceptance criteria and acceptance tests, and often automating them, before the code is even written. BDD works with TDD and is more fully described here.
• Test-Driven Development (TDD) – TDD involves writing the unit test first, then building the minimal code needed to pass the test. This leads to better design, higher quality, and increased productivity. TDD works with BDD and is more fully described here.
• Version control – Effective version control allows teams to recover quickly from problems and to improve quality by making sure the right components are integrated together. Aggregating assets under version control is a leading indicator of continuous integration maturity.
• Built-in quality – Built-In Quality prescribes practices around flow, architecture & design quality, code quality, system quality, and release quality.
• Application telemetry – Application telemetry is the primary mechanism that acquires and then uses application data to help determine the results of relevant hypotheses.
• Threat modeling – In addition to the threat modeling done in the Architect activity of continuous exploration, attention should be given during system design to possible vulnerabilities that may be introduced into the system.

Build the Solution

During the build phase, teams continuously integrate new code. This can be accomplished by automating the build and test tools to run upon code commit. Passing versus not-yet-passing and broken automated tests are the real indicators of progress. Automating code building enables teams to fix problems quickly, before they affect larger parts of the system. Addressing a broken build should be the highest priority.  A ‘gated commit’ ensures software has passed the gate (e.g. unit tested, performance tested, and free of known defects, etc.) before being checked into the main codebase or trunk. Code that passes the gate is automatically integrated into the trunk; which removes the complications of managing multiple branches. This trunk-based development helps to ensure the code can be reliably released on demand without the need for costly code freezes or hardening iterations.

There are five practices which can help build the solution:

• Continuous code integration – Code commit should automatically trigger compilation and testing of changes. Ideally, this happens on each commit but should happen at least several times a day.
• Build and test automation – The compilation process should be automated and include unit- and story-level tests to verify the change. These tests often use test doubles to replicate other parts of the systems and enable fast builds.
• Trunk-based development – Long-lived branches must be avoided. Teams should merge back as quickly as they can, at least once per day, and all teams should work off a single trunk.
• Gated commit – Committing to a single trunk is risky, as broken changes can impact many teams. This is why only the changes that have been validated through the build and test process are merged into the trunk.
• Application security – Code analysis tools inspect the code and third-party packages for known vulnerabilities.

Test the solution end-to-end

While critical, automated local story and component testing aren’t enough. In order to thoroughly test features, system-level integration and testing are required. With the support of the System Team, the work of all teams on the ART must be frequently integrated to assure that the solution is evolving as anticipated, as Figure 3 illustrates.

 

System-level testing happens as frequently as possible during the iteration, ideally after every commit. However, whatever the circumstances, such full-system integration must be accomplished at least once per iteration. Otherwise, the late discovery of defects and issues reflects back to earlier iterations, causing substantial rework and delays.

There are five practices which can help in end-to-end system testing:

• Test and production environment congruity – Environment congruity assures that testing exercises the solution as it would behave in front of live users and decreases the probability that defects will escape into production.
• Test automation – Many types of tests need to be run: functional testing, integration testing, regression testing, etc. The Agile Testing article details a testing matrix of what can and should be automated.
• Test data management – To create stability, tests must be consistent and realistic, replicating production as much as possible, and under source control.
• Service virtualization – Different kinds of testing require different environments. Service virtualizations allow teams to simulate a production environment without the costs and effort associated with creating and managing real environments.
• Testing nonfunctional requirements (NFRs) – system attributes such as security, reliability, performance, maintainability, scalability, and usability must also be thoroughly tested.
• Continuous integration with suppliers – Suppliers bring unique contributions that can have a significant impact on lead-time and value delivery. Their work must be continuously integrated as well. It helps to adopt a shared integration cadence and establish objective evaluation milestones.

Validate on Staging

Finally, the entire solution must be validated on a staging environment, based on the following practices:

• Maintain a staging environment –A staging environment, which matches production provides the place for such validation.
• Blue/Green deployment – The blue/green pattern involves two environments–live (production) and idle (staging). Changes flow continuously to the idle environment where they are staged until ready to deploy to production. At that point, a switch is flipped (a load balancer is updated for example), and the idle environment becomes the live environment, while the previous live environment becomes the new idle environment. This enables continuous delivery, zero-downtime deployment, and fast recovery from failures.
• System demo – This is the event where stakeholders evaluate a solution’s readiness to be deployed to production.

Enabling a Culture of Continuous Integration

Continuously integrating large and complex systems is a time-consuming journey. The following section provides some suggestions for building a successful CI culture and practice.

• Integrate often – The more frequently teams integrate, the quicker they find problems. And the harder it is to do, the more often they need to do it—eliminating impediments and adding automation along the way. This results in faster learning cycles and less rework.
• Make integration results visible – When the integration process breaks, everybody should know it broke and why. When it’s fixed, new tests should be added to detect the problem earlier and prevent it from happening again.
• Fixing failed integrations is a top priority – Teams that just keep working through integration failures fall short of the values and culture associated with building a system that can be released into production at any time. To create the right sense of urgency and importance needed to fix integration problems, teams often use flashing lights or other notifications to draw attention to a broken build, and establish visible indicators displaying the percentage of the time the system remains broken.
• Establish common cadence – Integration points are more accessible when all the teams are moving at the same consistent rhythm. If full CI can’t be accomplished during the course of an iteration, teams can make near-term trade-offs on what can be integrated, while continuously improving their techniques and infrastructure toward this goal.
• Develop and maintain proper infrastructure – Effective continuous integration depends on the availability of test and staging environments. Infrastructure is, of course, an investment. But Lean-Agile Leaders take the long view and make the investments necessary today to increase velocity for the marathon ahead.
• Apply supportive software engineering practices – Continuous integration is easier when the system is designed with those concerns in mind. Test-first development and designing for testability call for more modular solutions and separation of concerns, as well as using primary interfaces and physical test points.

Enabling Continuous Integration with DevOps

Continuous integration involves crucial ‘development’ activities that originally inspired the ‘Dev’ in DevOps. These activities are focused on solution development and pipeline flow through pre-production environments. Applying DevOps thinking, practices, and tooling in this segment of the value stream enables rapid development, frequent code integration, and built-in quality and compliance.

As illustrated in Figure 4, continuous integration is enabled by SAFe’s CALMR approach to DevOps (center) as well as several practice domains (inner rings). Each of the four activities (in green) is a collaborative effort that draws upon DevOps expertise from multiple domains to maximize delivery speed and quality.

As an example, building solutions in the continuous delivery pipeline crosses multiple DevOps domains. Checking code into version control triggers the deployment pipeline to invoke automated merge, quality and security checks, then to apply configurations stored as code to build shippable, full-stack binaries. Using DevOps, this process typically turns source code into tested, deployable solutions in a matter of minutes.

All four continuous integration activities are enabled by DevOps, though with different combinations of technical practices and tooling. For more detailed guidance on DevOps and how it enables the continuous delivery pipeline, see the DevOps article series.

---

Learn More

[1] Oosterwal, Dantar P. The Lean Machine: How Harley-Davidson Drove Top-Line Growth and Profitability with Revolutionary Lean Product Development. AMACOM, 2010.

[2] Leffingwell, Dean. Scaling Software Agility: Best Practices for Large Enterprises (Agile Software Development Series). Pearson Education, 2007.

[3] Kim, Gene, Jez Humble, Patrick Debois, and John Willis. The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations. IT Revolution Press, 2016.

 

Last update: 27 September 2021