In a cloud-based environment, the continuous delivery pipeline handles continuous integration, delivery, and deployment of code and applications and is often referred to as a CI/CD Pipeline.
CI/CD pipelines are crucial to any team developing cloud-native applications because they are used to manage the software lifecycle end-to-end.
The CI/CD pipeline is a subset of a larger workflow that includes automated testing and version control. In a nutshell, continuous delivery pipelines move application code from its source and deliver it to end users as an application or online service in the cloud.
The CI/CD pipeline helps organizations achieve fast and efficient application delivery in the new cloud ecosystem, however, the speed and lack of manual oversight associated with CI/CD processes can also create new security risks. I am going to cover some of the most common security risks in this article.
Developers and risk management stakeholders that are involved in identifying cyber-related risk in their cloud-based applications.
HOW TO USE THIS INFORMATION
When performing risk analysis and risk management activities within your CI/CD Pipeline, use the checklist to ensure you are not overlooking or forgetting any of the most common threats and risks.
Please refer to the risk terms and definitions section at the bottom of this article if you need help defining common security and risk management terms.
Get My Free Cloud Security Journal
CI/CD PIPELINE SECURITY
A successful implementation of the CI/CD pipeline cannot be delivered without a comprehensive understanding of how to manage relevant threats and risks to the CI/CD pipeline.
In this section, I list some of the most common risks associated with a typical continuous delivery process along with some supporting information to help you think about the risks and potential challenges in your environment.
The risks associated with CI/CI pipeline can be managed, but only if you place security front and center. A DevOps approach is common for most organizations, however, DevSecOps is a much better approach.
The unfortunate truth is CI/CD pipeline wasn’t probably conceptualized with security as a foremost consideration and this shouldn’t be too much of a shocker for any developer or risk management professional.
CI/CD pipelines are an attractive target for threat actors because of the potential for data leaks and disruption of services.
CI/CD RISK LANDSCAPE
CI/CD pipelines are made up of a combination of various components that work with each other to provide efficient integration and deployment. These components consist of code and image repositories, build servers, containers and third-party tools. These components depend on trust relationships to communicate with each other.
Vulnerabilities to a CI/CD platform are easy to miss because the system has a variety of dependencies and configurations that attackers can exploit.
Containers, Kubernetes, and microservices are the leading drivers of digital transformation for many enterprise organizations. Developers and their organizations have quickly embraced these cloud-centric development technologies because of their advantages in application development and deployment, quicker bug fixes and patches which lead to faster feature delivery that drives competitive differentiation.
These technologies create an infrastructure that has a lot of inherent security capabilities, but they are not enabled by default because containers and Kubernetes are application development tools.
For example, Kubernetes does release security enhancements, but they are disabled by default so the upgrade will not break application functionality.
It is highly unlikely that a developer will review the latest security features and go through the rigorous testing process to implement them unless this is driven by a formal risk management process by the organization. It is easy to see why new security improvements are frequently not realized in production. This is why a DevSecOps approach to cloud applications is so important to avoiding breaches and security incidents.
Containers seem like a secure option for running applications because of their short lifespan and their isolated nature. However, attackers have been known to successfully breach container images in the past.
Cloud containers are lightweight and lower overhead virtual machines (VMs) that are increasingly used to replace traditional virtual machines. The ease of container deployment can lead to security lapses with old container images that may include known vulnerabilities. These old containers with vulnerabilities can be quickly replicated and deployed through the cloud environment by a malicious actor.
To learn more about some of the basic container vulnerabilities review CVE-2019–5736 https://nvd.nist.gov/vuln/detail/CVE-2019-5736
This CVE is useful to understand because a malicious actor was able to create rouge container images and gain administrative privileges to take over the physical server. This type of scenario applies to the most popular containers such as Docker, Kubernetes, and runC.
Containers are walled off from each other, but they are usually deployed on the same IP space. This means that if an attacker gains access to even a single container, they can then access any number of other containers in the cluster. Keep this and CVE-2019–5736 in mind when designing your containers.
Containers typically have a short life span of about 15–20 minutes, which in theory sounds like a great way to keep attackers from attacking any other component in a CI/CD pipeline. However, attackers can use that short time to access other vulnerabilities and download packages that can be used for more elaborate attacks.
There are a variety of container security vendors to include Qualys, Alert Logic, Anchore, Aporeto, Aqua Security, and many others.
When running more than one container, there is a need to coordinate and manage the deployment, which is commonly referred to as orchestration.
Get My Free Cloud Security Journal
Attackers can exploit the trust relationship between code repositories and servers to make unauthorized changes in the code and commit them to the master, which can be extremely detrimental and cause significant damage to the organization.
Developers tend to use credentials to various tools and resources through environment variables. Even though this is a better choice than hard-coding these credentials into the code, attackers can dump these variables to get all the information they need to exploit other resources.
Keep in mind that malicious actors don’t have to launch a full-scale attack to be successful. Oftentimes seasoned threat actors play a game of chess and simply access one of many available services in an application and use it to exploit resources and launch attacks on another application.
Recycled code is convenient for developers, but attackers look to compromise open source code with vulnerabilities in order to leave applications that rely on them open to attack. This is why automated code scanning should be mandatory.
Last and certainly not least is configuration errors account for too many security breaches. Along with the basic misconfiguration challenges is the lack of enabling new security features as they are released.
CONTROLS & COUNTERMEASURES FOR THE CI/CD PIPELINE
As part of the normal process, DevOps and preferably DevSecOps teams should study their pipeline and work through threat modeling exercises, and document the attack surface. These exercises should be led by a qualified cloud security subject matter expert and participants should include developers, architects, and data owners at a minimum. This should all be part of the normal operating procedures within the application development and deployment lifecycle.
The controls and countermeasures listed below are not a lot of fun or very sexy, but they are foundational to keeping your CI/CD pipeline secure and operating as designed.
Harden Your Pipeline: start with the basics and lock down all relevant systems that host repositories, build servers, and configuration managers. Do not overlook this obvious and important control.
Guard & Monitor Immutable Logs: a process should be in place to regularly trace back changes to their origin, making sure no unauthorized changes have occurred.
Audit Your Tools: the tools used in your pipelines should be independently audited and updated regularly. Access to source code repositories should also be audited and limited with tight access controls to avoid both internal and external attacks.
Monitor Entire Pipeline: your entire CI/CD pipeline should be actively monitored for anomalies and be automated to a level that alerts relevant and qualified personnel about any anomalies.
Script Management: credentials, secrets, and keys should be removed from scripts and be protected by trusted secrets managers. The managers must be routinely audited as well.
No Shared Access: access control should be implemented across the entire CI/CD pipeline and ensure there are no shared or anonymous access allowed to any service.
Container Security: do not think that containers are your silver bullet to CI/CD pipeline security. Containers increase the attack surface of any application and there is a history of successful container breaches.
Code Scanning: it should go without saying, but code analysis tools should be implemented to ensure code is written using best practices and no backdoors are open for attackers.
Tim Layton specializes in demystifying the complexities and technical jargon associated with cloud computing security and risk management for business stakeholders across the enterprise. Tim is a cloud security thought leader defining actionable and defensible strategies to help enterprise stakeholders make risk-based decisions and prioritize investments in the new digital frontier.
Stay Connected With Tim Layton
Get My Free Cloud Security Journal
COMMON CYBERSECURITY RISK TERMS DEFINED
Threat: Any circumstance or event with the potential to adversely impact organizational operations (including mission, functions, image, or reputation), organizational assets, individuals, other organizations, or the Nation through an information system via unauthorized access, destruction, disclosure, or modification of information, and/or denial of service. (NIST 800–30)
Threat: potential cause of an unwanted incident, which can result in harm to a system or organization. (ISO 27001)
Vulnerability: Weakness in an information system, system security procedures, internal controls, or implementation that could be exploited by a threat source. (NIST 800–30)
Vulnerability: weakness of an asset or control that can be exploited by one or more threats. (ISO 27001)
Likelihood: A weighted factor based on a subjective analysis of the probability that a given threat is capable of exploiting a given vulnerability or a set of vulnerabilities. (NIST 800–30)
Likelihood: chance of something happening. (ISO 27001)
Risk: A measure of the extent to which an entity is threatened by a potential circumstance or event, and typically a function of (i) the adverse impacts that would arise if the circumstance or event occurs; and (ii) the likelihood of occurrence. (NIST 800–30)
Risk: effect of uncertainty on objectives. (ISO 27001)
Security Controls: The management, operational, and technical controls (i.e., safeguards or countermeasures) prescribed for an information system to protect the confidentiality, integrity, and availability of the system and its information. (NIST 800–30)
Compensating Security Control: A management, operational, and/or technical control (i.e., safeguard or countermeasure) employed by an organization in lieu of a recommended security control in the low, moderate, or high baselines that provides equivalent or comparable protection for an information system. (NIST 800–30)
Impact Level: The magnitude of harm that can be expected to result from the consequences of unauthorized disclosure of information, unauthorized modification of information, unauthorized destruction of information, or loss of information or information system availability. (NIST 800–30)
Residual Risk: Portion of risk remaining after security measures have been applied. (NIST 800–30)
Security Posture: The security status of an enterprise’s networks, information, and systems based on information assurance resources (e.g., people, hardware, software, policies) and capabilities in place to manage the defense of the enterprise and to react as the situation changes. (NIST 800–30)
Get My Free Cloud Security Journal