CAVAS: Neutralizing Application and Container ...

CAVAS: Neutralizing Application and Container Security Vulnerabilities in the Cloud Native Era Kennedy A. Torkura, Muhammad I.H. Sukmana, Feng Cheng, and Christoph Meinel Hasso-Plattner-Institute for Digital Engineering, University of Potsdam, Potsdam,Germany {kennedy.torkura,muhammad.sukmana,feng.cheng,christoph.meinel}@hpi.de

Abstract. The security challenges of container technologies such as Docker and Kubernetes are key issues in software development and other industries. Hence, several techniques for detecting image vulnerabilities have been recently introduced, in particular for images retained at DockerHub. However, investigations on vulnerabilities affecting the application-layer of Microservice Architectures (MSA) and Cloud Native Environments (CNE) is lacking. Also, there are no efforts to gain insights into the relationships between vulnerabilities at the application and image layers of MSA. In this paper, we investigate vulnerabilities impacting these layers and apply vulnerability correlation to establish according dependence relationships between the vulnerabilities. The outcome of this analysis offers interesting insights for risk management and security hardening of microservices e.g. deployment of vulnerability correlation-based security policies critical for vulnerability detection, risk prioritization and resource allocation. Our prototype implementation extends a previously introduced security system: Cloud Aware Vulnerability Assessment System(CAVAS), which employs the Security Gateway concept for security policy enforcement. The Security Gateway leverages the client side discovery and registry cloud pattern for discovering microservices and the notion of dynamic document stores for exploring and testing MSA. Our experimental evaluation shows that the security gateway’s vulnerability detection rate out-performs that of traditional testing approaches with 31.4 %. Also, we discover that about 26.2 % of severity metrics for vulnerabilities detected by image vulnerability scanners is in-correct. Hence, correcting this information is a prerequisite step to vulnerability correlation. Our proposal can therefore be employed for efficient continuous security and risk assessments in CNE. Keywords: Cloud-Security, Vulnerability Assessment, Vulnerability Correlation, Cloud Native Environments, Application Container Security, Microservices Security

1

Introduction

Container technologies like Docker and Kubernetes are enabling rapid application development and deployment. These technologies enhance productivity

2

Kennedy A. Torkura et al.

when combined with cloud infrastructure and DevOps [1] and are key components of Microservice Architectures (MSA) and Cloud Native Environment (CNE). However, container-based infrastructure is challenged by several security concerns, requiring novel security paradigms as effective counter-measures. An emerging security concept is the shift-left approach which proposes continuous security by wiring security tests early and throughout the Continuous Integration (CI) and Continuous Development (CD) pipeline [2]. The set of activities that implement these practices are generally referred to as SecDevOps (also called RuggedOps, DevSecOps, SecOps) [3]. However there are no set guidelines for SecDevOps and most research efforts focus on image-level vulnerabilities ignoring application-level vulnerabilities especially those affecting MSA. Most MSA adopt REST [4] for inter-service communication [5], however automated REST security testing is challenging 1 . Traditional security assessment techniques fail to explore REST applications, yet the exploration phase is a prerequisite for vulnerability detection through identification of entry/exit points. This exploration difficulty emerges since REST applications are not implemented with well defined interfaces as web applications. Contribution This work proposes a cloud native, continuous security methodology that employs security policies for detecting vulnerabilities in Docker container images and microservice applications. Our approach employs the shift left concept, achieved via our previously introduced concept- Security Gateway [6]. The security gateway, which is a core component of the Cloud Aware Vulnerability Assessment System (CAVAS), serves as a control mechanism for enforcing security policies. The notion of the security gateway here refers to security assessments, hence different from how the term is used in other contexts e.g. application firewalls [7] and network routers [8]. Two innovative concepts support the security gateway: dynamic document store and security health endpoints. The dynamic document store overcomes the aforementioned challenge of detecting vulnerabilities in REST microservices by leveraging OpenAPI 2 (formerly Swagger) documents for vulnerability detection. Similarly, security health endpoints provide security observability by presenting security information of deployed microservices. The client side discovery and registry cloud pattern is leveraged for microservices discovery. Furthermore, we validate the accuracy of the vulnerability information contained in image vulnerability scan results. Figure 1 illustrates our analysis of inaccuracies in vulnerability information returned by container image vulnerability scanners. We develop a technique for rectifying these anomalies and thereafter correlate the vulnerabilities detected at both image and application levels, within and across containers of the same application. The knowledge gained via vulnerability correlation [9] provides effective risk management techniques, e.g. we employ this insights for adding correlation-based rules in security policies. These rules prioritize detection of correlated vulnerabilities given the greater security risks posed by correlated vulnerabilities. To the best of our knowledge, this work is the first that applies the principle of 1 2

https://www.owasp.org/index.php/REST Assessment Cheat Sheet https://github.com/OAI/OpenAPI-Specification

Neutralizing Application and Container Security Vulnerabilities

vulnerability information validation and correlation in the context of MSA and CNE.

Fig. 1: Incorrect Vulnerability Security Metrics in 12 Analyzed Microservices

The rest of this paper is structured as follows, the next section presents related works, followed by a background of application container image vulnerabilities, challenges to security assessments in CNE and problem definition. In Section 3, the design and system model of CAVAS is described based on specified requirements. The implementation details of CAVAS is presented in Section 5. In Section 6, we evaluate our work and Section 7 concludes the paper.

2

Related Work

Gummaraju et al’s [10] security analysis of DockerHub revealed that over 30% and 40% of official and community of Docker images respectively, have severe vulnerabilities. In [11], the authors conducted a vulnerability oriented security analysis of the Docker ecosystem and highlighted several security implications for deploying Docker-based applications. Bila et al [12] presented a continuous security architecture for cloud-based container clusters that leverages OpenWhisk 3 serverless architecture for instant vulnerability alert notification. Harbor [13] is an open-source, enterprise grade registry similar to CAVAS, however it does not integrate with software supply chains and cloud container orchestration. Tak et.al [14] demonstrated the existence of drift in containers i.e. a situation where declared containers differ from deployed containers owing to un-tracked changes. The authors recommended continuous security testing for containers since one-time static analysis tests do not detect image drifts. Some of these recommendations are considered in this work. Zhnag et. al [15] used Harbormaster for policy enforcement in Docker environments, but the policies are for access control while our policies are for vulnerability detection. Antunes et al 3

https://openwhisk.apache.org/

3

4

Kennedy A. Torkura et al.

[16] investigated on vulnerability testing tools for web services and analyzed state-of-the-art approaches, tools and architectures. Their investigation tackles traditional monolithic web services and is therefore not fitted for CNE. In [17] the challenges of deploying microservices to cloud platforms were highlighted including the security issues, however the authors offered no practical solutions to the raised issues. Thanh et al. [18] proposed approaches for integrating security and privacy in cloud application development, focusing on on development pipelines, we tackle both development and production environments. Savchenko et al. [19] introduced a methodology for validating microservice cloud applications. Two shortcomings are identified in this work, first it is not clear if the proposed framework is evaluated, secondly, the work is limited to non-security tests in development environments e.g. unit tests and integration tests, our work complements this gap. To the best of our knowledge, this work is the first applying vulnerability information validation and correlation to microservices and CNE.

3 3.1

Background and Problem Application Container Vulnerabilties

The NIST Application Container Guide [20] outlines major risks to the core components of container technologies. We outline subsets of these risks most relevant to this paper. Insecure Container Runtime Configurations Container runtimes provide options for customization which can expose the security of the system if improperly configured. For example container executed with root permissions naturally has access to the OS including devices, kernel and other containers. This privilege might be abused by compromised containers. Image Vulnerabilities Images are essentially static, archive files composed of several package definitions. Images might be free of vulnerabilities at creation time but become vulnerable after some time if vulnerabilities are discovered in one or more of its components. Containers derived from images are not automatically updated like traditional software packages, thus, a common risk in is existence of vulnerable containers whose parent image versions are outdated i.e stale images. These stale images pose serious security risks upon execution. Image Configuration-based Vulnerabilities Similar to normal software, images are prone to configuration-based vulnerabilities which might either be intentional or un-intentional. For example, an image might include an SSH daemon, running with default credentials. Attackers might exploit this vulnerability to successfully attack containerized environments.

Neutralizing Application and Container Security Vulnerabilities

Images from Untrusted Sources This is one of the most high-risk image vulnerabilities. Due to publicly available images on DockerHub and other public image registries, images are easily pulled and executed without due validation. Application Vulnerabilities Applications are commonly built into images essentially providing those applications access to the host containers internals upon execution. Vulnerable applications use containers for conducting side-channel attacks [21]. 3.2

Security Assessments in Cloud-Native Environments

Several security issues are introduced when applications are deployed to CNEs [5]. Security assessments are imperatives for timely detection and mitigation of vulnerabilities associated with these security issues. However, CNEs present several challenges for security assessments, some of these challenges are highlighted in the following subsections: Microservices Discovery Microservices are subject to constantly changing environmental parameters such as ip addresses, port numbers and service endpoints. This dynamism presents an overhead for security e.g. security assessments which are traditionally configured for static network resources, hosts and applications. Essentially, security tools are challenged with discovering microservice endpoints e.g. after scaling operations. This discoverability challenge is similar to that of virtualized environments, but occurs at the application layer hence virtualization-based approaches do not suffice. Security Testing for MSA REST is a favoured architectural style for implementing MSAs [5]. REST exposes resources using endpoints easily accessible for a wide range of applications e.g. mobile and IoT devices. However, unlike with web applications, automated security assessments for RESTful applications is difficult[16]. Security scanners detect vulnerabilities in web applications by iteratively fetching and crawling through web-page links i.e. to discover entry and exit points. This is possible since web applications have well defined interfaces. Following the crawling phase, security scanners send random requests and analyze responses for security vulnerabilities. On the other hand, REST services do not have well defined interfaces as web applications, hence the testing difficulty. Technological Diversity Microservices are built with different business capabilities by different development teams, which may use different technologies i.e. different programming languages and frameworks [19]. The motivation for this approach is to use the best tools for tackling specific problems. While this aids in productivity, it complicates security e.g. the Spring PetClinic application 4 typically used for demonstrating MSAs consists of four microservices, three 4

https://github.com/spring-petclinic/spring-petclinic-microservices

5

6

Kennedy A. Torkura et al.

developed in Java and one developed in JavaScript (UI-Service). Effective vulnerability detection for this application requires different test configurations per microservice. This is imperative given the uniqueness of vulnerabilities per technology. Furthermore, developers integrate several open source components which could be laden with vulnerabilities [22]. 3.3

Problem Definition

Challenges to Continuous Security Assessments and Vulnerability Management Continuous security assessments are useful for identifying vulnerabilities in applications and networks. Identification of vulnerabilities and timely patching reduces attack surfaces thereby thwarting cyber-attacks. Security assessments for traditional applications/environments involve statically deployed applications/systems, but CNEs are ephemeral. Microservices are dynamically launched and de-registered, owing to scaling requirements and complexities in distributed systems. Hence, a discoverability challenge emerges for security assessment i.e. the capacity to constantly locate deployed microservices. Furthermore, the diversity of technologies in MSAs increases the possibilities for security vulnerabilities. The desire for fast-paced development cycles in MSAs (to meet time-to-market) further complicate these challenges by overlooking comprehensive security tests in CD pipelines. Consequently, vulnerable microservices could be launched to production environments. Novel security assessments techniques specifically adapted and integrated to CNEs are therefore required to tackle these challenges. Limitations of Prior Research Current research tackling CNE security focuses on aspects such as authentication and authorization, monitoring and anomaly detection. These are critical challenges affecting CNEs, but vulnerability detection and security assessments are not yet considered. Effective security evaluation of microservices aids timely detection of compromised instances and reduces attack surfaces. Moreover, security evaluations are key regulatory and compliance requirements [23], e.g. the Centre for Internet Security recommends continuous vulnerability assessments for identification and mitigation of vulnerabilities.5 To the best of our knowledge, there are no research efforts focused in this direction. Traditional security assessment methods do not handle the challenge of effectively conducting security assessments in CNEs. We therefore, aim at helping security teams implement robust vulnerability management systems that are native to the cloud i.e. suited for CNE. Though several research works have analyzed the challenge of severe security vulnerabilities found in container images, practical remediation approaches are lacking. We fill this gap with our approach which is consistent with SecDevOps and emphasizes integration of security into cloud-native software supply chains and production environments. 5 https://www.cisecurity.org/controls/continuous-vulnerability-assessment-andremediation/

Neutralizing Application and Container Security Vulnerabilities

4

Design and System Model

Ensuring continuous security in CNEs requires innovative approaches which can be identified by outlining security requirements. These requirements are discussed in this section, followed by a description of how we fulfill these requirements. 4.1

Requirements for Security Assessments in Cloud Native Environments

We identified five requirements necessary for continuous security assessments in MSA and CNE. Firstly, the security assessment solution should automatically discover deployed microservice instances. This requirement has several security benefits such as inserting security control for security policy enforcement. Next is support for security policies, given their usefulness for security automation and control. Fine grained security policies provide for security efficiency. Accordingly, the solution should support a wide range of security policies and also enforce these policies. Thirdly, the solution must be tamper-proof i.e. isolated from possible attacks. It should not be discoverable by other microservices except the core services e.g. service registry and discovery and API Gateway. We apply the concept security VMs [24] to achieve this, though VMs are replaced with containers i.e. security containers. Security provisions of Cloud networks e.g. security groups are used to isolates security containers from application containers. Fourthly, the solution must effectively resolve the technologies used in developing microservices. Microservices instances for an application may be developed using diverse technologies, employing generic security testing policies might not be as effective as policies that specifically tackle the development technology. For example, a Java microservice should be tested with a Java-biased policy. Hence the solution should be able to automatically identify the development technology. We satisfy this requirement by implementing the dynamic document stores concept, details are at Section 4.4. The last requirement is the need to validate the vulnerability information returned by image vulnerability scanners. From our investigations, we detected several anomalies in the security metrics assigned to detected vulnerabilities. More details are provided in Section 4.6. 4.2

Security Automation with the Security Gateway

The notion of a Security Gateway adopts the concept of Security Enforcement Points (SEP), which are commonly used to enforce security policies at run-time. For example in [25], Almorsy et al leveraged SEP to enforce security policies by intercepting and validating requests sent against critical components. The security gateway, differs from the microservices gateway pattern 6 operationally. The security gateway enforces defined security requirements in development and production environments. Conversely, API-Gateways are concerned with efficient 6

http://microservices.io/patterns/apigateway.html

7

8

Kennedy A. Torkura et al.

Fig. 2: CAVAS Workflow: SecDevOps in the Continuous Development/Integration Environment and Continuous Security in Production Environment (AWS ECS)

routing of incoming and outgoing traffic, the security gateway serves as a SEP for the policies described in Section 6.3. Our definition also differs from the use of the term security gateway in the context of application firewalls [7] and network routers [8]. By leveraging client-side service discovery and registry, the security gateway automatically overcomes two challenges: discoverabilty and inventory of microservices. Client-side discovery is an approach that forces incoming microservices to first contact the discovery server in order to gain situational awareness of other microservices. 7 Similarly, the embedded registry component maintains living record of deployed microservices. The CAVAS workflow (illustrated in Figure 2) describes automated steps within the development environment (steps 1 11 ) and the cloud-based production environment (steps 12 - 16 ). The Security Gateway is positioned in the staging phase i.e. steps 9 - 11. 4.3

Support for Security Policies

Security policies are best practices for automating security [26] through the definition of risk levels and appropriate actions to be taken when thresholds are breached. The Security Gateway acts as a SEP for enforcing the following policies: – Container Image Security Policies - Automated vulnerability analysis of container images is enforced through container security policies. These policies define the permissible security risk levels with rules as advised in the NIST Application Container Guide [20]. Container-based microservices are vulnerable to several security issues e.g. over 30 % of official images in Docker hub were infected with severe vulnerabilities [10]. Policies also enforce scheduled security assessments in production environments. – Microservice-Specific Security Policies - Ideally, every microservice would have a specific policy used for continuous security assessment. These policies are defined based on the implementation details of each microservice. 7

http://microservices.io/patterns/client-side-discovery.html

Neutralizing Application and Container Security Vulnerabilities

This approach aims at improving efficiency of security testing by targeting specific microservice implementation technologies. For example, a Ruby microservice will be tested with security policies specifying vulnerabilities published at the Ruby security advisories 8 or OWASP Ruby Cheat-sheet 9 . Furthermore, standard application security rating systems e.g. WASC and CWE are expressed as fine-grained rules. – Baseline Security Policies - Detailed security scanning for vulnerabilities is typically time-consuming, 30-60 minutes for an average web application and 6-12 hours for medium-scale web applications. Given the speedy turnout rate for microservices, lengthy tests are no long feasible at deploy time. Therefore, to strike the balance between speed and security, we propose baseline security policies for conducting pre-deployment security tests. These tests are timebased i.e. 2-3 minutes and produce either PASS or FAIL results. 4.4

Using OpenAPI Documents for Microservice Security Testing

The challenges of conducting security tests against MSA were highlighted in Section 3. An approach for overcoming this challenge consists in leveraging web service description documents [16]. These documents contain machine readable information e.g. internal architecture of applications. Vulnerability scanners can ingest these documents and extract information requisite for security testing. Due to non-standardization of formal REST resources, the Security Gateway adopts the OpenAPI (formerly called Swagger) standard, which formats documents in JSON and YAML. We propose all microservices contain similar documents, achieveable through two approaches: on-demand generation and filebased generation. We support both approaches, documents are retained in policy stores(config server) in accordance with externalized configuration cloud native design pattern10 . Furthermore, OpenAPI documents aid discovery of first and second order vulnerabilities in REST APIs [27]. Security policies earlier described in Section 4.3 can be also retained in the dynamic document stores. The store can be protected with token-based authentication e.g. JSON Web Token (JWT) or other automated authentication methods. 4.5

Security Observability

The Health Endpoint Monitoring Pattern provides resiliency by enforcing periodic heartbeats against microservices [28]. These checks aid timely detection of failures, 11 thus, a similar technique could be applied for security resiliency by providing security information of microservices. The distributed nature of microservices already complicates security monitoring, therefore approaches for easily verifying security states of microservices is imperative. Consequently, we 8

https://www.ruby-lang.org/en/security/ https://www.owasp.org/index.php/Ruby on Rails Cheatsheet 10 http://microservices.io/patterns/ 11 http://microservices.io/ 9

9

10

Kennedy A. Torkura et al.

Listing 1.1: Example Image Scan Result with ”Negligible” Severity Metric { ” f i x ” : ”None ” , ” package ” : ”dpkg − 1 . 1 7 . 2 7 ” , ” severity ” : ” Negligible ” , ” u r l ” : ” h t t p s : / / s e c u r i t y −t r a c k e r . d e b i a n . o r g / t r a c k e r /CVE−2017 −8283” , ” vuln ” : ”CVE−2017−8283” }

proposed the concept of Security Health Endpoints in [6]. Scan reports are made available via predefined URLs e.g. http://localhost:8090/security-health whereas the health checks are accessed at http://localhost:8090/health. The information can be directly extracted for security tasks e.g. automated configuration of AWS Security group rules and integration of vulnerability information into SIEMs [29]. 4.6

Vulnerability Information Correlation and Validation

The accuracy of vulnerability information is critical for proper risk assessment hence the need to validate information retrieved from vulnerability information sources and vulnerability scanners results. For example, comprehensive and accurate security metrics are critical in probabilistic threat models where CVSS scores are used for calculating probabilities. Though vulnerability information is retrivable from several sources (e.g. software vendors, security software vendors and independent researchers), relying on single sources is risky. Such information could either be incomplete or wrong. Listing 1.1 is an example result where CVE-2017-8283 is assigned CVE severity ”Negligible” whereas the NVD entry has a CVSSv2 score of 7.5 (HIGH).12 Therefore, correlating vulnerabilities from several sources improves accuracy and creates better understanding of security risks and vulnerabilities [30]. This feature is integrated in CAVAS by using HPIVDB 13 , a publicly available vulnerability database that correlates vulnerability information from numerous sources. Currently the database contains about 96410 vulnerabilities, originating from over 225420 different applications and over 17430 software vendors. Details of our methodology is provided in Section 5.5.

5

Implementation

This section provides implementation details of our implementation : CAVAS. Several components of CAVAS introduced in our earlier works [6, 31, 32] are extended to fit into acCNE security assessment. The most obvious extension is 12 13

https://nvd.nist.gov/vuln/detail/CVE-2017-8283 https://hpi-vdb.de/vulndb

Neutralizing Application and Container Security Vulnerabilities

Fig. 3: CAVAS Architecture

the addition of container image vulnerability testing, Figure 3 illustrates the microservices-based architecture of CAVAS, which is implemented with cloud native design patterns. Detailed description of the components is provided in the following subsections:

5.1

Security Gateway

The Security Gateway concept leverages cloud design patterns for security testing by adapting the service discovery and registry server to fulfill our design goals (Section 4). This approach enables integration of SEPs for security policies enforcement. The Security Gateway consists of a customized Eureka server deployed in the staging environment with a Config Server, these are core components of CNE. In [6], we described our adaptation of the Netflix Eureka server to serve as the Security Gateway. The dynamic document stores supports automated security testing for REST applications using OpenAPI documents. These OpenAPI documents are retained in the Config Server (policy store), with other configuration files as specified in the microservice externalized configuration tenet. The Security Gateway performs Pre-deployment Security Tests using a Algorithm 1, and can be easily adapted to enforce enterprise security policies e.g. where a PASS mandatory for deploying microservices. Listing 1.2 is a baseline security policy for detecting SQL Injection, XSS and CSRF vulnerabilities.

5.2

Image Vulnerability Analyzer

The Image Vulnerability Analyzer (IVA) interacts with Anchore and Docker Security Bench for appropriate vulnerability analysis. It depends on the Registry Monitoring Service (RMS) for alerts on when to conduct a test and the image or container to be tested. IVA is implemented in Java using Spotify Docker Java client library. 14 14

https://github.com/spotify/docker-client

11

12

Kennedy A. Torkura et al.

Algorithm 1 SecurityGateway Algorithm 1: procedure PreDeploymentSecTest(M uT Inf o, SecP olicy) 2: Receive registration request from MuT 3: assign instanceStatus STARTING . testing mode 4: appSecTests (MuTInfo, SecPolicy) . app tests 5: if appSecT ests.equals F AIL then 6: denydeploymentRequest (MuTInfo) 7: end if 8: complianceTests (MuTInfo, SecPolicy) . config tests 9: if complianceT ests.equals F AIL then 10: denyDeploymentRequest (MuTInfo) 11: if currentT imeStamp − latestT imeStamp > 24hours then 12: imageSecTest (MuTInfo, SecPolicy) . conditional image tests 13: if imageSecT est.equals F AIL then 14: denyDeploymentRequest (MuTInfo) 15: end if 16: if imageLastSecT est.equals F AIL then . most current image tests 17: denyDeploymentRequest (MuTInfo) 18: end if 19: end if 20: end if 21: assign instanceStatus UP . deploy to prod 22: end procedure

5.3

Registry Monitoring Service

The RMS is a notification endpoint for receiving webhook event notifications from our private image registry. Our development environment includes a private image registry based on Docker Registry 2.0. The private registry enables control for image entry and image exit activities via the Docker Registry API 15 . All events in the registry such as image pull and image push commands as illustrated in steps 4 - 7 of Figure 2, are registered as events and sent to the RMS as event notifications. On receipt of these notifications, the RMS sends scanning requests to the IVA. Images not available at the private registry are pulled from DockerHub, in this case IVA scans these in-coming images for security vulnerabilities, a PASS is required for entry. 5.4

Vulnerability Scanning Engines

Rather than building custom vulnerability scanning engines, we leverage existing engines and extend their capabilities. Hence, Anchore, OWASP ZAP and Docker Security Bench are integrated as scanning engines for CAVAS. Anchore is an open-source tool for conducting static vulnerability analysis of application containers.16 Anchore retrieves vulnerability information from several sources 15 16

https://docs.docker.com/registry/spec/api/ https://github.com/anchore/anchore-engine

Neutralizing Application and Container Security Vulnerabilities

and has an API for 3rd party integration. For application vulnerability analysis, we leverage OWASP ZAP17 , a popular open-source dynamic vulnerability scanner. OWASP ZAP’s API enables automation and integration with CAVAS, it also supports OpenAPI documents ingestion. The community edition of Docker Bench Security 18 is integrated for detecting security mis-configurations. Docker Bench security implements the Centre for Internet Security (CIS) benchmark 19 which specifies best practices for establishing secure configuration baselines in Docker environments. 5.5

Vulnerability Information Validation and Correlation

In order to correlate vulnerabilities, several steps are required. This approach was prompted when we discovered negligible and unknown values returned as vulnerability severity metrics. Since the NVD does not use these security metrics, we carefully inspected the results and discovered several anomalies. Thus, a technique for validating and correcting these anomalies was devised. Essentially, vulnerabilities with either unknown or negligible severity metrics were filtered and crosschecked against the HPI-VDB using API calls. Anomalies are thereafter replaced with the correct metrics: CVSSv2 scores e.g. 10.0, 3.5, CVE severity e.g. HIGH, LOW or CVE status e.g. AWAITING ANALYSIS, DEFERRED 20 . The next step was vulnerability correlation, done by identifying commonalities in image and application vulnerabilities. We used Common Weakness Enumeration (CWE) since it is commonly used for classifying application vulnerabilities as well as an additional classification for software packages, hence some vulnerabilities are assigned both CVEs and CWEs. Our technique queries our internal vulnerability analysis results database i.e persisted results of application and image scans, for matching CWEs/CVEs within microservices that consists an application. The knowledge gained can be used in several security scenarios e.g. policy formulation, risk assessment and vulnerability prioritization. See Section 6.2 for details. 5.6

Cloud Deployment Manager

To satisfy the need for continuous security in-production environments, the Cloud Deployment Manager (CDM) manages microservices deployed in the production AWS environment. The CDM can be deployed either in development or production environments as illustrated in Figure 2. CDM integrates with several AWS services e.g. Elastic Container Service (ECS), Elastic Container Registry (ECR) and Elastic Computing Cloud (EC2). It conducts in production security testing e.g fuzzing by spawning identical instances of microservices scheduled for security tests. These tests are conducted in the staging environment, thereby protecting deployed microservices from negative effects that might 17

https://www.owasp.org/index.php/OWASP Zed Attack Proxy Project https://github.com/docker/docker-bench-security 19 https://www.cisecurity.org/benchmark/docker/ 20 https://nvd.nist.gov/vuln 18

13

14

Kennedy A. Torkura et al.

Listing 1.2: Example Baseline Security Policy { ” p o l i c y name” : ” v e r s i o n 2 t e s t s ” , ” p o l i c y type ” : ” B a s e l i n e S e c u r i t y P o l i c y ” , ” a t t a c k s t r e n g t h ” : ”medium ” , ”max p e r m i s s i b l e r i s k ” : ” low ” , ” scope ” : { ” depth limit ” : 5 }, ” checks ” : [ ” s q l i n j e c t i o n ” ,” xss ” ,” c s r f ” ] , ” p l u g i n s ” : {” d i s c o v e r y c h e c k s ” : ” mild ” , ” c r o s s s i t e s c r i p t i n g ” : ”medium ” , ” c r o s s s i t e r e q u e s t f o r g e r y ” : ”medium ” , ” s q l i n j e c t i o n ” : ”medium”} }

emerge consequent of tests e.g. fuzzing tests. This is possible since containers are immutable hence testing microservices with the same source image is sufficient to identify vulnerabilities without testing the actual production microservice. The CDM continuously inspects processes running in deployed microservices to identify malicious activities (steps 15 and 16 of Figure 2). The concept of security health endpoints presented in Section 4.5 is implemented in each microservice. The endpoints retrieve and present the most current security assessment results in json format when queried e.g on receipt of GET requests.

Table 1: Summary of Test Environment Environment Variable Local Host Operating System Local Host RAM/HDD Automation Server/Git Cloud Container Orchestration

6

Value Ubuntu 14.04 Trusty Tahr 32GB/500GB Jenkins/Github AWS ECS, AWS EC2

Evaluation

We evaluated CAVAS through practical experiments in a private CI/CD pipeline, details of our testbed is in Table 1. Three types of experiments were conducted, first the vulnerability detection rate [33] for CAVAS is determined and evaluated. Next, we show the efficiency of our proposed technique for vulnerability information validation and correlation. Lastly, the effectiveness of our security policies is demonstrated.

Neutralizing Application and Container Security Vulnerabilities

Fig. 4: CAVAS Vulnerability Detection Rate

6.1

Vulnerability Detection Rate

Three container-based microservice applications were used for this experiment: Spring PetClinic 21 , Movie recommendation Service 22 and an eShop Application from Eberhard Wolff’s book [34],also available on GitHub 23 . We will thereafter refer to these applications as PetClinic,kbastani and eWollf respectively. The applications were cloned from their GitHub repositories and adapted to fit our test criteria i.e support for OpenAPI documents generation was added. The applications were selected based on their popularity as reference microservice architectures as evidenced through their high GitHub ratings. Each application consists of several microservice instances, e.g. PetClinic consists of four microservices: petclinic-vets,petclininc-api-gateway, petclininc-customers and petclinincvisits. The core microservices components e.g. service registry and discovery of the test applications are not tested. To measure the image vulnerabilities, the corresponding images for each microservice is built and analyzed for security vulnerabilities using the integrated Anchore scanning engine. The results are thereafter collected and persisted in CAVAS reports database. In order to measure the application vulnerabilities, the images are executed i.e. containerized and tested using dynamic security testing techniques. This step is necessary for detecting the vulnerabilities existing in the application code. In this test phase, two Eureka servers are deployed : with traditional OWASP ZAP vulnerabil21

https://github.com/spring-petclinic/spring-petclinic-microservices https://github.com/kbastani/spring-cloud-microservice-example 23 https://github.com/ewolff/microservice 22

15

16

Kennedy A. Torkura et al.

ity scanner and with the Security Gateway, we refer to them as Eureka A and Eureka B respectively. The baseline security policy is used for testing. Eureka A conducts the test using conventional web application testing methods hence several vulnerabilities are not detected. Conversely, Eureka B leverages the dynamic document stores to retrieve and employ the OpenAPI documents for each Microservice Under Test (MuT). Since this approach overcomes the challenge of testing REST services (see Section 3.2), more vulnerabilities are detected. The details of the tests are illustrated in Figure 4. Ultimately, Eureka B outperforms Eureka A with about 31.4 %. Figure 5, also shows the detected images vulnerabilities, over 5,600 vulnerabilities were detected for the 12 images tested.

Fig. 5: Anomalies in Images Vulnerability Severity Metrics

6.2

Vulnerability Information Validation and Correlation

In order to verify the efficiency of our vulnerability correlation and validation approach, we conducted two tests. In the first test, we investigated the correctness of vulnerability information returned from the image vulnerability tests. To conduct a broader test, a fourth application- PiggyMetrics 24 (thereafter called piggymetrics) was introduced. We discovered that about 20.8 % of detected vulnerabilities returned negligible as severity values, while about 5.4 % reflected unknown, also as severity values. This error-rate is unhealthy for risk assessment hence CAVAS automatically detects and corrects vulnerability information. Additionally, we wanted to investigate the correlation between image and 24

https://github.com/sqshq/PiggyMetrics

Neutralizing Application and Container Security Vulnerabilities

application vulnerabilities. According to [30], vulnerability correlation provides deeper visibility into vulnerability analysis, enabling effective risk management. As seen in Figures 4 and 5, the image vulnerabilities detected in microservices per application are almost identical. This is the first level of correlation, the second level is to detect the correlation between the vulnerabilities detected in the application as well as the image. To do this, a common factor is used as the base, for our experiments, we used the CWE which is used for both packages and web application vulnerability classification. Other types of classifications are not feasible e.g. few web applications have CVEs. Due to space limitations, details of the results are not shown, however Table 2, briefly shows the CVEs, corresponding CVSS, severity metrics and CWE id for the vulnerabilities correlated in the kbastani application. All the vulnerabilities correlate with CWE-200, described as: An information exposure is the intentional or unintentional disclosure of information to an actor that is not explicitly authorized to have access to that information. 25 The correlation results can be applied to different scenarios, e.g. the vulnerabilities in Figure 2 could be prioritized for patching because the correlation reflects high security risks. These correlated vulnerabilities introduce shared-code vulnerabilities which have severe security implications if exploited [35]. A successful attack might easily transverse across applications and images that share such vulnerabilities. Hence more efforts could be implemented towards rectifying correlated vulnerabilities.

Table 2: Vulnerabilities Correlated in the kbastani Application CVE Id CVE-2017-9526 CVE-2017-1000100 CVE-2017-7407 CVE-2015-5276 CVE-2018-1000007

6.3

kbastani Application CVSS Score Severity 4.3 Medium 4.3 Medium 2.1 low 5.0 Medium 5.0 Medium

CWE Id CWE-200 CWE-200 CWE-200 CWE-200 CWE-200

Security Policy Enforcement

In order to demonstrate the efficiency of the security gateway in enforcing security policies, we commit a code to the petclinic-customers code repository and generally follow steps 1-10 of the CAVAS workflow (Figure 2). At the staging area, the baseline security policy is implemented against the MuT as a process of the pre-deployment test. Since the MuT does not satisfy the rules expressed in the policy, it fails the test and cannot be pushed to production. We also evaluate a policy based on the knowledge gained from the vulnerability correlation effort. 25

https://cwe.mitre.org/data/definitions/200.html

17

18

REFERENCES

kbastani-movies-ui is the MuT and a rule is added to the baseline security policy to check for correlated vulnerabilities i.e. CWE-200 and CVE-2018-1000007. Again, following steps 1-10 of CAVAS workflow, the MuT fails the test and the deployment request is canceled.

7

Conclusion and Future Work

This paper presents an innovative shift-left and continuous security approach suited for detecting application and container image vulnerabilities in development and production microservice-based environments. Our prototype implementation extends our previous work: Cloud Aware Vulnerability Assessment System(CAVAS), which employs the Security Gateway concept for security policy enforcement by acting like a SEP. The Security Gateway uses the dynamic document stores and takes advantage of client side discovery and registry cloud pattern for identifying and testing microservices for security vulnerabilities. Ultimately, our evaluations show that the security gateway’s vulnerability detection rate out-performs traditional testing techniques by 31.4 %. Additionally, we propose techniques for validating vulnerability information returned from image vulnerability scans given that over 26.2 % of this information is incorrect. After correcting these anomalies, we employ vulnerability correlation for identifying dependence relationships between image and application vulnerabilities. The result of this analysis is interesting and prospective since microservices of an application are composed of virtually identical packages, thus very similar vulnerabilities i.e. shared-code vulnerabilities. This vulnerability commonality can be exploited by attackers e.g. a successful attack against one microservice might expose other microservices to the same attack vector. To counter this, we propose vulnerability correlation derived policies that identify and prevent deployment of vulnerable containers. In the future, we intend to conduct more complex experiments for vulnerability correlation to measure its efficiency for risk assessment. Therefore, supporting schemes like vulnerability correlation matrix for CNE might be relevant.

References [1]

B. Fitzgerald and K.-J. Stol, “Continuous software engineering: A roadmap and agenda”, Journal of Systems and Software, vol. 123, pp. 176–189, 2017. [2] J. Bird, DevOpsSec Securing Software through Continuous Delivery. O’ Relliy Media, Inc, 1005 Gravenstein Highway North, Sebastopol, CA, 2016. [3] A. A. Ur Rahman and L. Williams, “Software security in devops: Synthesizing practitioners’ perceptions and practices”, in Proceedings of the International Workshop on Continuous Software Evolution and Delivery, 2016. [4] R. T. Fielding and R. N. Taylor, “Architectural styles and the design of network-based software architectures”, PhD thesis, 2000.

REFERENCES

[5]

[6]

[7]

[8] [9]

[10]

[11] [12]

[13] [14]

[15]

[16]

[17] [18]

[19]

19

N. Dragoni, S. Giallorenzo, A. L. Lafuente, M. Mazzara, F. Montesi, R. Mustafin, and L. Safina, “Microservices: Yesterday, today, and tomorrow”, in Present and Ulterior Software Engineering, Springer, 2017, pp. 195–216. K. A. Torkura, M. I. Sukmana, and C. Meinel, “Integrating continuous security assessments in microservices and cloud native applications”, in Proceedings of the10th International Conference on Utility and Cloud Computing, 2017. D. Scott and R. Sharp, “Abstracting application-level web security”, in Proceedings of the 11th international conference on World Wide Web, ACM, 2002, pp. 396–407. R. Oppliger, “Security at the internet layer”, Computer, vol. 31, no. 9, pp. 43–47, 1998. P.-Y. Chen, G. Kataria, and R. Krishnan, “Correlated failures, diversification, and information security risk management”, MIS quarterly, pp. 397– 422, 2011. J. Gummaraju, T. Desikan, and Y. Turner, “Over 30% of official images in docker hub contain high priority security vulnerabilities”, Technical report, BanyanOps, Tech. Rep., 2015. T. Combe, A. Martin, and R. Di Pietro, “Containers: Vulnerability analysis”, tech. report, Nokia Bell Labs, Tech. Rep. N. Bila, P. Dettori, A. Kanso, Y. Watanabe, and A. Youssef, “Leveraging the serverless architecture for securing linux containers”, in Distributed Computing Systems Workshops (ICDCSW), 2017 IEEE 37th International Conference on, 2017. VMWare, Harbor. [Online]. Available: http : / / vmware . github . io / harbor/. B. Tak, C. Isci, S. Duri, N. Bila, S. Nadgowda, and J. Doran, “Understanding security implications of using containers in the cloud”, in USENIX Annual Technical Conference (USENIX ATC’17), 2017. M. Zhang, D. Marino, and P. Efstathopoulos, “Harbormaster: Policy enforcement for containers”, in Cloud Computing Technology and Science (CloudCom), 2015 IEEE 7th International Conference on. N. Antunes and M. Vieira, “Designing vulnerability testing tools for web services: Approach, components, and tools”, International Journal of Information Security, pp. 1–23, 2016. C. Esposito, A. Castiglione, and K.-K. R. Choo, “Challenges in delivering software in the cloud as microservices”, IEEE Cloud Computing, 2016. T. Q. Thanh, S. Covaci, T. Magedanz, P. Gouvas, and A. Zafeiropoulos, “Embedding security and privacy into the development and operation of cloud applications and services”, in Telecommunications Network Strategy and Planning Symposium (Networks), 2016 17th International, IEEE. D. I. Savchenko, G. I. Radchenko, and O. Taipale, “Microservices validation: Mjolnirr platform case study”, in Information and Communication Technology, Electronics and Microelectronics (MIPRO), 2015 38th International Convention on.

20

[20]

[21]

[22] [23]

[24]

[25]

[26]

[27]

[28] [29]

[30]

[31]

[32]

[33]

[34] [35]

REFERENCES

M. Souppaya, J. Morello, and K. Scarfone, “Application container security guide”, 2017. [Online]. Available: https://doi.org/10.6028/NIST.SP. 800-190. M. Schwarz, S. Weiser, D. Gruss, C. Maurice, and S. Mangard, “Malware guard extension: Using sgx to conceal cache attacks”, in International Conference on Detection of Intrusions and Malware, and Vulnerability Assessment, Springer. D. Wichers, “Owasp top-10 2013”, OWASP Foundation, February, 2013. C. S. Alliance, “Domain 4: Complaince and audit management”, 2011. [Online]. Available: https : / / cloudsecurityalliance . org / wp - content / uploads/2011/09/Domain-4.doc. Y. Sun, S. Nanda, and T. Jaeger, “Security-as-a-service for microservicesbased cloud applications”, in Cloud Computing Technology and Science (CloudCom), 2015 IEEE 7th International Conference on. M. Almorsy, J. Grundy, and A. S. Ibrahim, “Adaptable, model-driven security engineering for saas cloud-based applications”, Automated software engineering, 2014. S. Subashini and V. Kavitha, “A survey on security issues in service delivery models of cloud computing”, Journal of network and computer applications, vol. 34, no. 1, pp. 1–11, 2011. S. Davis, “Using the open api specification to find first and second order vulnerabilities in restful apis”, 2016. [Online]. Available: https://2016. appsec . eu / wp - content / uploads / 2016 / 07 / AppSecEU2016 - Scott Davis-Scanning-with-Swagger.pdf. A. Homer, J. Sharp, L. Brader, M. Narumoto, and T. Swanson, Cloud Design Patterns. Microsoft Press, 2014. S. Roschke, F. Cheng, R. Schuppenies, and C. Meinel, “Towards unifying vulnerability information for attack graph construction.”, in ISC, Springer, 2009. L. Wang, R. Ma, H. Gao, X. Wang, and C. Hu, “Analysis of vulnerability correlation based on data fitting”, in Chinese Conference on Trusted Computing and Information Security, Springer, 2017, pp. 165–180. K. A. Torkura and C. Meinel, “Towards cloud-aware vulnerability assessments”, in Signal-Image Technology & Internet-Based Systems (SITIS), 2015 11th International Conference on. K. A. Torkura, M. I. Sukmana, F. Cheng, and C. Meinel, “Leveraging cloud native design patterns for security-as-a-service applications”, in Smart Cloud (SmartCloud), 2017 IEEE International Conference on. J. Bau, E. Bursztein, D. Gupta, and J. Mitchell, “State of the art: Automated black-box web application vulnerability testing”, in Security and Privacy (SP), 2010 IEEE Symposium on, IEEE, 2010, pp. 332–345. E. Wolff, Microservices: Flexible Software Architecture. Addison-Wesley Professional, 2016. A. Nappa, R. Johnson, L. Bilge, J. Caballero, and T. Dumitras, “The attack of the clones: A study of the impact of shared code on vulnerability

REFERENCES

21

patching”, in Security and Privacy (SP), 2015 IEEE Symposium on, IEEE, 2015, pp. 692–708.