CI/CD Pipelines: Why, Where and How

Written by Yifat Perry, Technical Content Manager | Oct 24, 2021 2:51:29 PM

What Is a CI/CD Pipeline?

A continuous integration/continuous delivery (CI/CD) pipeline is a set of processes used to deliver new versions of software. It is an approach to software development and delivery that focuses on continuous improvement, involving site reliability engineering (SRE) or DevOps practices.

In a CI/CD pipeline, monitoring and automation are introduced early in the software development process, most notably during testing and integration, but also throughout the delivery and deployment phases. Technically, each part of the CI/CD pipeline can be performed manually, but the real value of this approach is best achieved with automation.

CI/CD pipelines are comprised of several stages, each consisting of a distinct subset of tasks. Typically, the pipeline begins with the build stage, when the software code is developed. The new code is then tested, often using automated tools and techniques to reduce the time and effort required.

Once the code has been tested, it can be released (the application or software is delivered to the repository). The new code is deployed to production and evaluated based on the organization’s specific needs. This stage also involves comparing the application to security and vulnerability requirements to ensure compliance.

This is part of an extensive series of guides about software development.

In this article:

What Is a CI/CD Pipeline?
A Brief Introduction to Continuous Delivery
What are the Main Benefits of a CI/CD Pipeline?
Challenges of CI/CD Pipelines
Types of CI/CD Pipelines
CI/CD and the Cloud
7 Stages of a CI/CD Pipeline
How to Choose CI/CD Tools
CI/CD Pipelines with Cloud Volumes ONTAP

A Brief Introduction to Continuous Delivery

Continuous delivery (CD) is a software delivery approach in which teams develop and test code in short, continuous cycles. CD usually requires a high degree of automation, to ensure quality standards are not impacted by high delivery speeds. Teams implementing the CD process use incremental updates to quickly build, test, and deploy software.

CD is a popular software delivery approach, especially for DevOps teams. It is usually paired with continuous integration (CI), which helps create a chain of processes, including software development and deployment, as well as feedback loops. The complete process is called a CI/CD pipeline.

During CD processes, code is delivered on a regular basis to a staging environment or to user acceptance testing. To reduce unexpected performance problems during production, code is tested for functionality. By shifting testing to the left, the CD process can prevent issues later on during production.

Components that pass automated testing are valid candidates for release. The CD process then triggers a final human check and finally pushes to deployment. You can also add continuous deployment, to automatically deploy builds.

Related content: Read our blog Why You Need Infrastructure as Code to do DevOps Properly

What are the Main Benefits of a CI/CD Pipeline?

CI/CD pipelines are designed so everything happens simultaneously. While one software iteration is being coded, another iteration is being tested and another deployed. Benefits of CI/CD pipelines include:

Efficient development—breaking the software development process into small steps, or iterations, makes it easier to test and fix issues. It also allows you to evaluate the usefulness of each feature so you can adjust or remove less valuable features.
Improved maintenance—the pace and flow of CI/CD pipelines allow you to fix bugs much faster than in a traditional development approach.
Competitive products—a CI/CD development approach yields products much faster compared to traditional development approaches and can be more easily adapted to changing needs. This means that products can reach the market faster.
Operations support—Ops teams can address the needs of the software with regular releases, and administrators can better deploy updates or roll back software with minimal errors. Automation tools can also help reduce configuration errors while accelerating deployments.

Challenges of CI/CD Pipelines

CI/CD has become enormously popular and is practiced by a majority of modern development organizations. However, as with every innovation, it comes with its own set of challenges:

Over-reliance on automation—automation must be properly implemented and managed, with automation tools often having a steep learning curve. Changes to the toolset or development process can be challenging.
Complexity of planning—new builds must constantly be developed, tested and deployed, which requires complex planning and management. Deciding on the scope of automation vs manual tasks can take time and cause delays.
Complexity of collaboration—automation alone cannot make the development process more efficient. Project stakeholders and developers must maintain strong communication and collaboration for effective CI/CD.
Employee conduct—developers must follow established guidelines to meet standards for quality and style. Employees can easily forget or overlook such guidelines.

Related content: Read our blog Enhancing CI/CD Pipelines with Cloud Volumes ONTAP

Types of CI/CD Pipelines

There are many different ways to configure a CI/CD pipeline. Examples of pipeline types include:

Basic pipeline—all steps in each pipeline stage run concurrently, while the stages themselves are executed in order.
Directed Acyclic Graph (DAG) pipeline—this is often faster than a basic pipeline, as it is based on the relationships between jobs.
Multi-project pipeline—the pipelines for multiple projects are combined.
Parent-child pipeline—a complex pipeline is broken down into a parent pipeline that triggers sub-pipelines (or child pipelines). These all run with the same Secure Hash Algorithms (SHA) in the same project, making it useful for monorepos.
Merge request pipeline—this only runs for merge requests, not for each commit.
Merged results pipeline—this is a type of merge request pipeline that treats changes from the source as though they are already merged into the target.
Merge train—merged results are formed using pipelines that queue merges sequentially.

CI/CD and the Cloud

Cloud applications are subject to constant changes and may be scaled up or down at any time. This is why the majority of DevOps teams use CI/CD pipelines for cloud development and maintenance. In addition to enabling teams to quickly meet traffic demands, CI/CD also helps ensure that cloud applications are kept up-to-date with the latest security threats.

Cloud and CI/CD are made highly flexible and efficient through containerization and microservices. Here is how:

Quick updates - a microservices architecture enables DevOps teams to update applications in increments, rather than taking the entire application down. Microservices consist of loosely-bundled individual services and can operate independently. This means that an application can still run as a while even when a service is rendered inoperable or is removed and replaced.
Automated deployment - containers are isolated environments that include everything an application needs to run. A container can also run independently, allowing teams to update each application individually without affecting other containers. Additionally, containers have a consistent deployment procedure, which unifies all applications running in a container. This allows developers to automate the deployment process.

7 Stages of a CI/CD Pipeline

An effective CI/CD pipeline should incorporate the following stages:

Triggering the pipeline—the pipeline should be triggered automatically with a new commit. CI/CD tools can be configured to respond to changes to the repository, allowing the pipeline to run automatically. Relying on a manual trigger is risky, as people can be slow or even forget to run the pipeline.
Checking out the code—the CI server checks out the code from a source code repository like Bitbucket or GitHub. A webhook tells the CI/CD tool which commit triggers the pipeline, which checks out the source code at the appropriate commit point.
Compiling the code—when using a compiled language (e.g. Java), you need to compile your program. The CI tool must have access to the required build tools for compiling an application (e.g. Gradle or Maven). Containerization is useful for this stage, which is best run in a clean environment.
Unit testing—this involves configuring the CI/CD tool to execute tests in your codebase (you can use Jest in JavaScript or Gradle or Maven in Java). This stage verifies that unit tests all pass and are properly maintained with the growth of the code base. As applications grow, so should the number of tests.
Packaging the code—when all unit tests have passed, the code can be packaged. The target environment and programming language determine how you package an application (for example, you could build a JAR file for Java, or build a Docker image if using containers). You should build a single binary and avoid using different binaries for different environments.
Acceptance testing—acceptance tests help ensure your application functions as intended. Manual testing is time-consuming, so you should automate acceptance tests with tools like Selenium or Cucumber.
Delivery/deployment—once the application is fully tested, it can continue to the CD stage. This could be the delivery of a prepared artifact (continuous delivery) or the automated deployment of your software (continuous deployment). Continuous deployment requires a production environment you can deploy to (e.g. Kubernetes or a public cloud).

How to Choose CI/CD Tools

Here are several considerations that can help you choose the most suitable CI/CD tools for your use case:

Hosting requirements
A CI/CD tool can be self-hosted, hosted in a private cloud, or hosted within a public cloud environment. Determine your hosting needs before choosing a CI/CD tool.

A cloud-hosted tool handles infrastructure, maintenance, support, and security, but it usually requires access to sensitive information. Since the cloud vendor is in charge of many aspects, you do not get full control over the environment.

An on-premises tool provides complete control over all processes and data and processes and can potentially offer greater customization. However, it requires handling hosting, security, and updates in house.

Budget
The majority of CI/CD tools offer a wide range of versions, each offering a different set of resources and pricing models. There are also discounted versions and free open-source projects that may meet a leaner budget. The downside is that not all free or discounted offerings provide support or all necessary features.

When determining your budget, consider the following aspects:

The number as well as frequency of iterations you are releasing
The number of concurrent builds as well as timing expectations for builds
The number of environments and users
Data retention requirements

Features
A CI/CD tool should reduce pipeline complexity and maintenance. This is why any tool must provide scaling, customization, and automation capabilities while supporting all environments and languages relevant to your use case. In addition, the you may also consider tools that provide any the following functionality:

Support for Kubernetes and Docker
Easy integration with cloud vendors
Built-in testing and automation
A package repository
Metrics for collection and reporting
Real-time logging, monitoring, and alerting
Concurrent processing for builds and testing
SSH support
Secrets management

Community support
Strong community support is critical for proprietary and open source tools, because it fosters better security and documentation, as well as more frequent feature releases. It can also help prevent investment in tools that quickly become deprecated. Additionally, tools with strong support are more likely to be supported for integration with other tools.

CI/CD Pipelines with Cloud Volumes ONTAP

Cloud Volumes ONTAP ticks all the boxes when it comes to storage capabilities required for your CI/CD workflows—be it high availability, data protection, quick provisioning, or integration with Kubernetes automation.

In particular, Cloud Volumes ONTAP provides data cloning, letting you create writable clone volumes instantly and with zero capacity penalty. Cloud Volumes ONTAP and NetApp FlexClone® help organizations to speed up dev/test environments creation processes to accelerate the solution delivery and reduce time to market (TTM).

To learn more, check out this article on the CI/CD storage challenges and how Cloud Volumes ONTAP help to address them and this article on Cloud Volumes ONTAP Data Cloning Case Studies.

Learn More About CI/CD Pipelines

5 GitHub Actions CI/CD Best Practices

Software development is inherently a complex activity. Not too long ago, organizations only released software once or twice per year. Today, tools like GitHub Actions make it possible to release code into production dozens and even hundreds of times in a single day.

GitHub Actions CI/CD workflows integrate code versioning and management for fast-paced releases. Here are the basics on GitHub Actions and five best practices.

GitHub Actions Runners: Setting Up GitHub Actions CI/CD Using Self-Hosted AWS Runners and Cloud Volumes ONTAP

As a newcomer to the tooling market, “Actions” by GitHub quickly became a popular option to develop CI/CD workflows. GitHub was already one of the top code management platforms, thus these additional features were a natural segway to enable end-to-end—from code to production release—capabilities.

GitHub Actions runners can enhance your CI/CD workflows. This post shows you how to set up self-hosted runners on AWS with Cloud Volumes ONTAP.

Enhancing CI/CD Pipelines with Cloud Volumes ONTAP

In cloud-scale application development and deployment, DevOps and the CI/CD workflows integrated into DevOps pipelines play critical roles. CI/CD workflows deliver the true value of agile, as code developed by different teams can be integrated and deployed in an incremental manner.

The storage layer has a huge impact on how quickly CI/CD pipelines can release new products to market. Find out how Cloud Volumes ONTAP optimizes CI/CD storage.

CI/CD Pipelines with Cloud Volumes ONTAP Case Studies

In continuous integration and continuous delivery (CI/CD) pipelines, the storage layer is not something to take for granted. Cloud Volumes ONTAP helps enterprise CI/CD pipelines deliver by ensuring agility and data efficiency across the cloud and data center.

This blog will show you how by looking at three customer success stories—a semiconductor chip designer, an electronics manufacturing giant, and a data visualization SaaS—to reduce CI/CD costs, increase agility, and get products to market faster.

Integrating Cloud Volumes ONTAP into DevOps Kubernetes Pipelines with GitHub

CI/CD pipelines make development faster and more agile. However, when using Kubernetes-based environments, there are still costs for persistent storage and test copies to keep in mind. NetApp has an easy way to reduce those costs and speed up your pipelines by integrating with Cloud Volumes ONTAP and two of its best features: NetApp FlexClone and Astra Trident.

How can you do it? This blog will show you step by step how to instantly clone PVs in Kubernetes environments using Cloud Volumes ONTAP.

See Additional Guides on Key Software Development Topics

Together with our content partners, we have authored in-depth guides on several other topics that can also be useful as you explore the world of Software Development.