Continuous Delivery is the ability to get changes of all types—including new features, configuration changes, bug fixes and experiments—into production, or into the hands of users, safely and quickly in a sustainable way.
Our goal is to make deployments—whether of a large-scale distributed system, a complex production environment, an embedded system, or an app—predictable, routine affairs that can be performed on demand.
We achieve all this by ensuring our code is always in a deployable state, even in the face of teams of thousands of developers making changes on a daily basis. We thus completely eliminate the integration, testing and hardening phases that traditionally followed “dev complete”, as well as code freezes.
Continuous Delivery Automates the Flow to Production. Continuous Delivery (CD) is the process to build, test, configure and deploy from a build to a production environment. Multiple testing or staging environments create a Release Pipeline to automate the creation of infrastructure and deployment of a new. Successive environments support progressively longer-running activities of integration, load, and user acceptance testing. Continuous Integration starts the CD process and the pipeline stages each successive environment the next upon successful completion of tests.
Continuous Delivery may sequence multiple deployment “rings” for progressive exposure (also known as “controlling the blast radius”). Progressive exposure groups users who get to try new releases to monitor their experience in “rings.” The first deployment ring is often a “canary” used to test new versions in production before a broader rollout. CD automates deployment from one ring to the next and may optionally depend on an approval step, in which a decision maker signs off on the changes electronically. CD may create an auditable record of the approval in order to satisfy regulatory procedures or other control objectives.
Without Continuous Delivery, software release cycles were previously a bottleneck for application and operation teams. Manual processes led to unreliable releases that produced delays and errors. These teams often relied on handoffs that resulted in issues during release cycles. The automated release pipeline allows a “fail fast” approach to validation, where the tests most likely to fail quickly are run first and longer-running tests happen after the faster ones complete successfully.
Continuous Delivery is a lean practice. The goal of CD is to keep production fresh by achieving the shortest path from the availability of new code in version control or new components in package management to deployment. By automation, CD minimizes the time to deploy and time to mitigate or time to remediate production incidents (TTM and TTR). In lean terms, this optimizes process time and eliminates idle time. Continuous Delivery is helped considerably by the complementary practices of Infrastructure as Code and Monitoring.
Continuously delivering value has become a mandatory requirement for organizations. To deliver value to your end users, you must release continually and without errors.
Continuous Delivery also supports two other patterns for progressive exposure beside sequential rings. “Blue/Green deployment” relies on keeping an existing (blue) version live while a new (green) one is deployed. Typically, this uses load balancing to direct increasing amounts of traffic to the green deployment. If monitoring discovers an incident, traffic can be rerouted to the blue deployment still running. “Feature flags” (or “feature toggles”) comprise another technique used for experimentation and “dark launches.” Feature flags turn on features on or off for different end users based on their identity and group membership.
Modern release pipelines allow development teams to deploy new features fast and safely. Issues found in production can be remediated quickly by rolling forward with a new deployment. In this way, CD creates a continuous stream of customer value.
Continuous delivery rests on three foundations: comprehensive configuration management, continuous integration, and continuous testing.
On the technology side, continuous delivery leans heavily on deployment pipelines to automate the testing and deployment processes. A deployment pipeline is an automated system that runs increasingly rigorous test suites against a build as a series of sequential stages. This picks up where continuous integration leaves off, so a reliable continuous integration setup is a prerequisite to implementing continuous delivery.
At each stage, the build either fails the tests, which alerts the team, or passes the tests, which results in automatic promotion to the next stage. As the build moves through the pipeline, later stages deploy the build to environments that mirror the production environment as closely as possible. This way the build, the deployment process, and the environment can be tested in tandem. The pipeline ends with a build that can be deployed to production at any time in a single step.
The organizational aspects of continuous delivery encourage prioritization of “deployability” as a principle concern. This has an impact on the way that features are built and hooked into the rest of the codebase. Thought must be put into the design of the code so that features can be safely deployed to production at any time, even when incomplete. A number of techniques have emerged to assist in this area.
Continuous delivery is attractive because it automates the steps between checking code into the repository and deciding on whether to release well-tested, functional builds to your production infrastructure. The steps that help assert the quality and correctness of the code are automated, but the final decision about what to release is left in the hands of the organization for maximum flexibility.
Why continuous delivery?
It is often assumed that if we want to deploy software more frequently, we must accept lower levels of stability and reliability in our systems. In fact, peer-reviewed research shows that this is not the case—high performance teams consistently deliver services faster and more reliably than their low performing competition. This is true even in highly regulated domains such as financial services and government. This capability provides an incredible competitive advantage for organizations that are willing to invest the effort to pursue it.
The practices at the heart of continuous delivery help us achieve several important benefits:
- Low risk releases. The primary goal of continuous delivery is to make software deployments painless, low-risk events that can be performed at any time, on demand. By applying patterns such as blue-green deployments it is relatively straightforward to achieve zero-downtime deployments that are undetectable to users.
- Faster time to market. It’s not uncommon for the integration and test/fix phase of the traditional phased software delivery lifecycle to consume weeks or even months. When teams work together to automate the build and deployment, environment provisioning, and regression testing processes, developers can incorporate integration and regression testing into their daily work and completely remove these phases. We also avoid the large amounts of re-work that plague the phased approach.
- Higher quality. When developers have automated tools that discover regressions within minutes, teams are freed to focus their effort on user research and higher level testing activities such as exploratory testing, usability testing, and performance and security testing. By building a deployment pipeline, these activities can be performed continuously throughout the delivery process, ensuring quality is built in to products and services from the beginning.
- Lower costs. Any successful software product or service will evolve significantly over the course of its lifetime. By investing in build, test, deployment and environment automation, we substantially reduce the cost of making and delivering incremental changes to software by eliminating many of the fixed costs associated with the release process.
- Better products. Continuous delivery makes it economic to work in small batches. This means we can get feedback from users throughout the delivery lifecycle based on working software. Techniques such as A/B testing enable us to take a hypothesis-driven approach to product development whereby we can test ideas with users before building out whole features. This means we can avoid the 2/3 of features we build that deliver zero or negative value to our businesses.
- Happier teams. Peer-reviewed research has shown continuous delivery makes releases less painful and reduces team burnout. Furthermore, when we release more frequently, software delivery teams can engage more actively with users, learn which ideas work and which don’t, and see first-hand the outcomes of the work they have done. By removing the low-value painful activities associated with software delivery, we can focus on what we care about most—continuously delighting our users.
Continuous Delivery vs. Continuous Deployment
With continuous delivery, every code change is built, tested, and then pushed to a non-production testing or staging environment. There can be multiple, parallel test stages before a production deployment. The difference between continuous delivery and continuous deployment is the presence of a manual approval to update to production. With continuous deployment, production happens automatically without explicit approval.
Continuous delivery automates the entire software release process. Every revision that is committed triggers an automated flow that builds, tests, and then stages the update. The final decision to deploy to a live production environment is triggered by the developer.
Continuous Delivery Benefits
- Automate the Software Release Process – Continuous delivery lets your team automatically build, test, and prepare code changes for release to production so that your software delivery is more efficient and rapid.
- Improve Developer Productivity – These practices help your team be more productive by freeing developers from manual tasks and encouraging behaviors that help reduce the number of errors and bugs deployed to customers.
- Find and Address Bugs Quicker – Your team can discover and address bugs earlier before they grow into larger problems later with more frequent and comprehensive testing. Continuous delivery lets you more easily perform additional types of tests on your code because the entire process has been automated.
- Deliver Updates Faster – Continuous delivery helps your team deliver updates to customers faster and more frequently. When continuous delivery is implemented properly, you will always have a deployment-ready build artifact that has passed through a standardized test process.
CD Principles
There are five principles at the heart of continuous delivery:
- Build quality in
- Work in small batches
- Computers perform repetitive tasks, people solve problems
- Relentlessly pursue continuous improvement
- Everyone is responsible
It’s easy to get bogged down in the details of implementing continuous delivery—tools, architecture, practices, politics—if you find yourself lost, try revisiting these principles and you may find it helps you refocus on what’s important.
Continuous delivery treats the commonplace notion of a deployment pipeline as a lean Poka-Yoke: a set of validations through which a piece of software must pass on its way to release. Code is compiled if necessary and then packaged by a build server every time a change is committed to a source control repository, then tested by a number of different techniques (possibly including manual testing) before it can be marked as releasable.
Developers used to a long cycle time may need to change their mindset when working in a CD environment. It is important to understand that any code commit may be released to customers at any point. Patterns such as feature toggles can be very useful for committing code early which is not yet ready for use by end users. Using NoSQL can eliminate the step of data migrations and schema changes, often manual steps or exceptions to a continuous delivery workflow. Other useful techniques for developing code in isolation such as code branching are not obsolete in a CD world, but must be adapted to fit the principles of CD – for example, running multiple long-lived code branches can prove impractical, as a releasable artifact must be built early in the CD process from a single code branch if it is to pass through all phases of the pipeline.
Deployment Pipeline
One of the challenges of an automated build and test environment is you want your build to be fast, so that you can get fast feedback, but comprehensive tests take a long time to run. A deployment pipeline is a way to deal with this by breaking up your build into stages. Each stage provides increasing confidence, usually at the cost of extra time. Early stages can find most problems yielding faster feedback, while later stages provide slower and more through probing. Deployment pipelines are a central part of Continuous Delivery.
Usually the first stage of a deployment pipeline will do any compilation and provide binaries for later stages. Later stages may include manual checks, such as any tests that can’t be automated. Stages can be automatic, or require human authorization to proceed, they may be parallelized over many machines to speed up the build. Deploying into production is usually the final stage in a pipeline.
More broadly the deployment pipeline’s job is to detect any changes that will lead to problems in production. These can include performance, security, or usability issues. A deployment pipeline should enable collaboration between the various groups involved in delivering software and provide everyone visibility about the flow of changes in the system, together with a thorough audit trail.
A good way to introduce continuous delivery is to model your current delivery process as a deployment pipeline, then examine this for bottlenecks, opportunities for automation, and collaboration points.
When to do Continuous Delivery
You’re doing continuous delivery when:
- Your software is deployable throughout its lifecycle
- Your team prioritizes keeping the software deployable over working on new features
- Anybody can get fast, automated feedback on the production readiness of their systems any time somebody makes a change to them
- You can perform push-button deployments of any version of the software to any environment on demand
You achieve continuous delivery by continuously integrating the software done by the development team, building executables, and running automated tests on those executables to detect problems. Furthermore you push the executables into increasingly production-like environments to ensure the software will work in production. To do this you use a Deployment Pipeline.
The key test is that a business sponsor could request that the current development version of the software can be deployed into production at a moment’s notice – and nobody would bat an eyelid, let alone panic.
The CI/CD Pipeline
As more software development teams aim to meet the growing demand for faster release cycles and increased software quality, many are looking to implement a continuous development pipeline to streamline their process. Adopting CI/ CD practices enable teams to adapt their software on-demand to meet user feedback, market shifts, and any adjustments to the overarching business strategy (i.e., changes to the requirements).
While there is no definitive CI/CD pipeline structure, it is typically broken down into the following stages:
- Commit & Build: When developers are finished making changes to an application, they commit their code to a shared repository which will then integrate their snippet with the central code base. The piece of software, or the new feature, is then built from the extracted code and unit tested. CI plays an essential role in streamlining this process by automating each step after the code is written.
- Automated Testing: Once the new piece of software is developed, it needs to be built and then thoroughly tested to ensure it meets all the initial requirements. There are various testing methodologies that can be used to ensure an application looks and behaves as expected and covers everything from functional tests to performance tests. During this stage, it’s essential to test the entire system in environments that are similar to the production environment because the success of the software depends on it working in the environments your end users are accessing it on.
- Deployment: In the final stage, the built piece of software is rolled out into production. CD requires this process to be automated, which ensures a reliable delivery to users.
Continuous integration and delivery means continuous quality. With products such as TestLeft, you can create and run automated tests right from your IDE, like Visual Studio, IntelliJ IDEA or Eclipse, enabling a seamless workflow with your continuous integration and continuous delivery process. Once tests are executed, you can then track changes, merge modifications and restore previous project versions or schedule, provision, and frequently deploy builds for continuous delivery.
Manual Testing and CI/CD
The CI/CD pipeline requires development, test, and operations teams to work together to create a cohesive release process and the key is automation.
The problem is that manual testing isn’t dead. It is a pervasive process that is still used throughout the industry across every stage of the development cycle. While it does serve a distinct purpose and will always be needed for specific types of testing, such as exploratory testing, it can hinder an iterative delivery process. Time is of the essence in a continuous environment and manual testing can be painfully sluggish. The slower the feedback loops, the longer it will take to make adjustments and release software. Automation accelerates the build and deploys cycle by checking code and running tests continuously and providing feedback within minutes. This is why as much of the process as possible should be automated, from unit tests to system tests and even environment provisioning.
For example, if you don’t commit your code continuously, you’ll have longer periods between integrations, making it harder to find and fix bugs. Waiting days or even weeks between builds can easily derail a project before it even reaches the testing stage.
The testing stage can be broken down into multiple parts based on the objective of each of your test ‘suites,’ each of which is designed to ensure the software meets the original business requirements. Whether you’re testing the application’s functionality, security, or performance, each suite can and should be automated. With the right tool, you can run regression tests to re-test scenarios each time a change is made to ensure the application’s existing functionality hasn’t been broken. The right tool will also allow you to conduct parallel testing, which is the process of running multiple tests, or tests cases, simultaneously on different browsers or systems. Running tests concurrently improves test coverage and will cut down on testing times, which is key to optimizing an iterative test process.
