Where to Integrate Security Practices on DevOps

Where to Integrate Security Practices on DevOps Platform Hasan Yasar, Software Engineering Institute |CMU [email protected] Telephone: +1 412-268-9219 4500 Fifth Avenue Pittsburgh, PA 15213 USA Kiriakos Kontostathis, Software Engineering Institute | CMU [email protected] Telephone: +1 412-268-3337 4500 Fifth Avenue Pittsburgh, PA 15213 USA

Abstract "Software security" often evokes negative feelings amongst software developers because this term is associated with additional programming effort, uncertainty and road blocker activity on rapid development and release cycles. The Secure DevOps movement attempts to combat the toxic environment surrounding software security by shifting the paradigm from following rules and guidelines to creatively determining solutions for tough security problems (Taschner, 2015). Secure software should be focused on a proactive approach that limits the attack surface and produces reliable software. Secure DevOps developers want their software to bend but not break, which means the software absorbs attacks and continues to function. The burgeoning concepts of DevOps include a number of concepts that can be applied to increase the security of developed applications. Applying these and other DevOps principles can have a big impact on creating an environment that is resilient and secure. Specifically, this paper clearly explains how to address security concerns in the early stages of the development lifecycle and leverage that knowledge throughout the SDLC. Keywords: Software Security, Secure DevOps, Software Development Life Cycle, Secure Software Development Life Cycle

[Distribution Statement A] This material has been approved for public release and unlimited distribution. Please see Copyright notice for non-US Government use and distribution

1



1. Introduction

In the past, software security focused on anticipating where and how attacks would be leveraged on an application and putting up barriers to prevent those attacks. However, most attacks, especially sophisticated attacks, can't be anticipated, which means that fixes are bolted on as new vulnerabilities are discovered. The inability to anticipate attacks and lack of implementing security early in a project lifecycle is why there are often patches in response to 0-day vulnerabilities (Kongsli, 2006). This paper discusses ways to reverse this reactive nature of secure development and help teams to respond more quickly and more effectively to cyber incidents. Developing a secure lifecycle requires the development team to focus on continuous integration, infrastructure as code, eliminating denial of service (DOS), and limiting the attack surface. These include adding automated security testing techniques such as fuzz testing, software penetration testing to the software development cycle or the system integration cycle. Other techniques include standardizing the integration cycle in order to reduce the possibility of faults and security concerns by implementing good security practices from the inception of the project, or “pushing security left” (Tomhave & Kenefick, 2014). This paper focuses on how implementing security from the inception of a project greatly improves the overall security of the software being produced. It also discusses how implementing these pieces of security on a DevOps platform allows for rapid release cycles as well as secure software. These techniques provide developers, and other team members, a deeper understanding of what makes a truly secure application. 1.1 Problem Statement It is very difficult to integrate security into the software development lifecycle because there is no time for manual penetration testing and audits. There is also no opportunity to put in control gates and perform extensive security reviews. These problems arise due to the velocity of change in the software lifecycle. The velocity of change increases due to rapid business changes (Liu, Y., L. Chengbo, and L. Wei) , growing vulnerabilities, software complexity and dependencies on third-party libraries (i.e. open source modules). Organizations like Facebook and Etsy are delivering changes into production environment 50 or more times each day. Amazon has thousands of small engineering teams working independently and continuously deploying changes into various production environments. In 2014, Amazon deployed 50 million changes which equates to more than one deployment per second (Brigham & Liguori, 2015). This paper discusses how a development team can take measures early in the DevOps software development lifecycle to ensure the security of their piece of software.

[Distribution Statement A] This material has been approved for public release and unlimited distribution. Please see Copyright notice for non-US Government use and distribution

2



2. What is DevOps?

Modern software development teams attempt to leverage engaging, semiautonomous technologies to facilitate the myriad of data management, business requirements and communications tasks necessary throughout the Software Development Life Cycle (Cois, Yankel, & Connell, 2014). This has led to a new development methodology, DevOps, which aims to ensure quick release cycles and promotes a collaborative, integrated communication platform which should be made available to all project stakeholders. This includes development, operational, compliance, tester, business analyst, project managers and end users who are sharing same business goals to maintain world class reliability, operation, and security. These features allow a development team to embed other project stakeholders at the inception of a project. Embedding different types of project stakeholders gives the project team knowledge that may be crucial in obtaining these quick release cycles. Generally, project teams practicing DevOps embed IT operational teams at the inception of their projects. However, it is possible to embed any type of team, including security teams and allow a project team to plan for security from the inception of a project, which leads to a more secure piece of software.

3. Barriers to Incorporating Security into the SDLC

There are several barriers that face companies, development teams, etc. which make incorporating security into the SDLC difficult. The following sections discuss the most common barriers, which means that this list is not exhaustive. 3.1 Silos Silos are often brought up when discussing barriers to new methodologies and technologies in the software community. Unfortunately, Secure DevOps is unable to escape that same fate. DevOps is built upon collaboration and communication between all stakeholders for a certain project. Security, which is often its own silo, is another stakeholder that needs to be part of a project from its inception. Another major issue is that a security team has strict guidelines that need to be followed when developing a secure application. This environment leads to slow development and even slower release cycles. To combat this, development teams have to treat security teams as first-class stakeholders and incorporate security from the beginning of a project instead of implementing security as an afterthought. Incorporating security early on could prove to be crucial when it comes time to release applications into a production environment (Bliesner, 2014). Both development and security teams should be using the vast amount of security monitoring and testing tools to assist in the overall transparency of the security for a specific project or set of projects.

[Distribution Statement A] This material has been approved for public release and unlimited distribution. Please see Copyright notice for non-US Government use and distribution

3



3.2 Company as a Whole

Companies as a whole can make a huge impact in how security is implemented throughout the SDLC. This is more than just the people at the top of the organizational chart, it’s the company culture and policies along with the actual company structure. The company culture and people have huge impacts on how DevOps is utilized in the workplace (Spafford & Haight, 2014). There are several opinions about the correct company structure for Secure DevOps and how to set up a company with security in mind. However, most companies are already set up and struggling to transition to a security-focused development strategy. As a result, there is not one way to overhaul your company’s culture, policy, or structure and most importantly that is not a practical solution. The key lies in how development teams can leverage their unique company and develop secure applications. One technique that is effective in almost all situations is to bring security in at the inception of the project. This may come in several different ways. For example, one company might have the resources to complete a formal security requirements gathering process while another company may not have those resources. The second company in this scenario can still discuss how they are planning to incorporate security throughout the SDLC in their general requirement gathering and systems design processes. 3.3 Toolset Differences Another major barrier to implementing security is the differences in toolsets between security, operational, and even other development teams. Everybody has their favorite tool for each development activity. Many DevOps tools allow for easy integration into other tools. Unfortunately, that is not always true and sometimes tools may integrate with each other, but it takes a lot of time to manually set-up and may have to be continually maintained over time. One could argue that a solution to this problem is to demand that everyone uses the same set of tools. The problem with this solution is that every tool has its advantages and disadvantages and in some situations one tool would be better suited than the other and vice versa in other situations (Humble, J. and D. Farley). The key to overcoming these toolset differences is that the development team, at the inception of the project, should design their system with this issue in mind. What security testing tools are necessary to produce a secure application? What build agents offer easy integration with those testing tools? What monitoring tools give us the most helpful information in terms of security? These types of questions could be used to build the overall system architecture necessary to produce a secure application.

[Distribution Statement A] This material has been approved for public release and unlimited distribution. Please see Copyright notice for non-US Government use and distribution

4



4. Secure Development Activities

These sections discuss the secure development activities that need to take place in order to produce a secure application. These sections also discuss where we can introduce these activities throughout the SDLC. 4.1 Security Requirements Gathering Generally, one of the first steps when starting a project, or a new phase of a project, is a requirements gathering process. This process takes many forms depending on the company, industry, and members of the project team. There are some common topics discussed. For instance, the features of the new project and the improvements that need to be made if it the development team is starting a new phase of a project. Unfortunately, security is often not discussed, and if so, not in great detail, when gathering requirements. However, including security as early as the requirements gathering could be crucial to the overall security of the project. Performing a formal security requirements gathering session saves a lot of time because it is extremely difficult to include security at a later stage without carefully planning for it beforehand. This is also where it is crucial to start treating the security team as a first-class citizen. Their input throughout the SDLC saves a lot of time when it is time for the security team to approve the application for production. SecReq is one method of security requirements gathering that is built upon the Common Criteria for Information Technology Security Evaluation, the HeRa Tool, and UMLSec. This method talks specifically about tracing the information gathered in the security requirement gathering to the software design process to create a secure software design (Houmb, S.H., Islam, S., Knauss, E. et al, 2010). 4.2 Threat Modeling Threat modeling is another important process that is crucial when designing the system architecture. This process gives an approach to gather information regarding the security risks that is associated with the current piece of software. This is another process that ensures the introduction of security concern at the inception of the project. Threat modeling would take place at the inception of the project and would be a major task of the security requirements gathering and system design processes. Threat modelling provides great information that can be incorporated into the design process of the software development lifecycle. Threat modelling provides the development team with a definition of security for the application, identification of potential threat and vulnerabilities, justifies security features, and results in finding architecture bugs earlier in the development lifecycle. There are several types of threat modelling techniques including the Microsoft STRIDE model and the DREAD model that can be leveraged to obtain this information (Burns, 2005). [Distribution Statement A] This material has been approved for public release and unlimited distribution. Please see Copyright notice for non-US Government use and distribution

5



4.3 Environment Configuration

Once the requirements are gathered and the design is well underway, the environments have to be setup to best support our application. Since we are discussing security on a DevOps platform, it is important to note that the environments are automatically provisioned and set up. Therefore, there are some important things that need to be addressed at this step. For instance, the development has to ensure that the third-party dependencies needed for our application do not introduce vulnerabilities into the application. The Common Vulnerabilities and Exposure (CVE) list provides information on the known vulnerabilities for third-party dependencies. Ensuring secure environments helps the security team to green light an application because they do not have to chase down the team who set up the environments and figure out what thirdparty dependencies were used. 4.4 Secure Static Analysis Generally, code should not be able to be added to the original code base until it has gone through some kind of code review to ensure the security and functionality of the new code. Static analysis provides a quick and easy way to review code and even prevent insecure code from reaching the code base. There are several static analysis tools that are built to look for insecure coding practices. The data obtained from a static analysis tool can be leveraged by the code repositories to prevent insecure coding from entering the official code base for a project. This is also leveraged during code reviews to help developers and other team members learn why the code was marked as insecure and what vulnerabilities that coding mistake could introduce to the application. Jim Bird discusses what is needed to be successful when using secure static analysis. Static analyzers must run quickly, give clear feedback to developers, integrate into existing tools, and the team must take time to configure their tools perfectly for their application (Bird, 2016). 4.5 Security-Focused Code Review In addition to using static analysis tools to block insecure code. Development teams perform regular code reviews that specifically focus on the security of the application. These code reviews provide a place where developers and other team members to better learn and understand what makes a secure application and how that maps to writing secure code. Sarah Vonnegut provides a couple of tips to conduct a better secure code review. Those tips are to create code review checklists which will make sure that all reviewers are looking for the similar issues, secure code reviews should emphasize learning, not blaming, code should be reviewed every time something substantial was changed, automated tools and human reviews should be used in conjunction, and document/monitor each code review to look for patterns (Vonnegut, 2016). All of the security activities that have been discussed are completed before any integration into the production environment(Duvall, P.M., S. Matyas, and A. [Distribution Statement A] This material has been approved for public release and unlimited distribution. Please see Copyright notice for non-US Government use and distribution

6

Glover) . However, these steps are crucial to ensuring that an application is secure before it is pushed into a production environment. Performing these steps go a long way in convincing the security team that the application is secure and will not introduce vulnerabilities into the system as a whole. 4.6 Software Penetration Testing Software Penetration testing is an important step in releasing any piece of software because it includes tests that are specific to the application (B. Arkin, S. Stender, and G. McGraw ). Penetration testing is generally done by hand and can take a lot of time to complete. However, there are several tools that perform standard and custom penetration testing after the build agent builds the application. Automating these tests greatly improves the amount of time it takes to release software into a staging or production environment. Another nice feature of automated penetration testing is the ability to fail a build if problems arise while performing penetration tests. Penetration testing offers many additional benefits including the ability to highlights the weaker parts of an application, ensures availability for a system, helps to meet necessary compliance standards, creates a more trustful environment for all stakeholders, improves standing in a specific market with better quality assurance, and it is a crucial aspect to the suite of security measures necessary for a secure application (Finjan, 2016). 4.7 Environment Testing Environment testing includes another set of tests to perform that ensures that the environments are setup correctly and free of common vulnerabilities. These tests can also be automated during the build process and, just like penetration testing, fails the build if there are critical problems while performing environment-specific tests(Marvin, R.) (Callahan, J., F. Schneider, and S. Easterbrook) . This ability to fail a build automatically is crucial because there is not the necessary down time needed to rollback our application to a previous, more secure version of the software(Feitelson, D.G., E. Frachtenberg, and K.L. Beck,). 4.8 Security Review The final security activity is the manual security review when the application is ready to be transitioned into production. This step is very difficult to automate because this is where the security team comes and gives it final approval for the release of software. Even though this step cannot be easily automated, all of the work done to this point makes it far easier for the security team to approve the application for release. It is also important to make sure that the security has some feedback loop to provide the development team information about their latest release. This is especially handy if the security team decides to write another penetration test because there is a new major feature, that test should be added to source control and is used during the next build as an automated test instead of the security having to manually run the test. As mentioned above, [Distribution Statement A] This material has been approved for public release and unlimited distribution. Please see Copyright notice for non-US Government use and distribution

7

treating the security team as a first-class citizen from the inception of a project helps expedite this process because the security team was able to give input and advice to the development team.

Figure 1: Secure DevOps Software Development Life Cycle



This image depicts the Secure DevOps SDLC. It shows how security can truly be implemented throughout the SDLC and most importantly in the early stages of a project. The project life cycle starts with the planning phase. There should be no delay from when a project gets started and when security starts getting talked about and implemented. During the inception phase, a formal security requirements gathering process and perform an extensive threat modeling process should be completed. After planning, the designing process can begin and environments need to be configured and provisioned with the necessary dependencies. This is the step where CVEs are taken into consideration and the needed third-party dependencies are added to the architecture of the system. The next steps in the SDLC are the coding steps, which is where development teams write code, commit that code to a central repository, and then perform code reviews to ensure that the code is up to secure coding standards. This section of the SDLC is where static analysis tools can be leveraged and securityfocused code reviews can be performed.

[Distribution Statement A] This material has been approved for public release and unlimited distribution. Please see Copyright notice for non-US Government use and distribution

8

After the coding portion is the most automated portion of the SDLC and where automated security tests can be performed. Penetration and environment testing are amongst some of the testing that takes place throughout this portion of the SDLC. A major benefit to the DevOps platform is the ability to fail the build if custom and/or standard tests fail. If the build fails, the new version of the software will not be deployed to the production environment. This means that there will not be any down time because there is no long and manual rollback process. After the code builds properly and passes all of the necessary security tests, the security team can give its final approval as the final step before being pushed into production. The vast amount of security that has been carefully designed and implemented makes this step extremely easy because the security team will have knowledge of the project from its inception and be more comfortable with its security.

5. Benefits

There are several benefits to incorporating security throughout the SDLC. The major benefits include increased team efficiency, traceability, and quick incident response to cyber incidents. 5.1 Team efficiency High team efficiency is a highly sought after attribute by managers and developers alike. The DevOps platform supports the ability to have high team efficiency because so many pieces of the SDLC are automated. Incorporating security into multiple checkpoints on a DevOps platform only increases a team’s efficiency. The smaller, bite-size tasks make implementing security much easier and far more effective. Also, including important stakeholder, i.e. the security team, at early stages of a project allows them to give input and carefully monitor the project throughout the SDLC. This transparency allows them to quickly assess an application when it comes time to release a push into production. 5.2 Traceability Traceability is also another extremely important attribute to project teams. If something goes wrong, having a traceable process makes it much easier to find and resolve the issue. This is another major benefit to spreading out security tasks across the SDLC. These smaller, easier to understand security tasks makes it much easier to spot the problem and resolve those problems. Also, since security is being implemented from the inception of the project, there is documentation available on what has been done throughout the project lifecycle. The final thing that makes traceability apparent, is the ability to easily create short, quick feedback loops which puts all team members on the same page and continually allows for communication between all project stakeholders. [Distribution Statement A] This material has been approved for public release and unlimited distribution. Please see Copyright notice for non-US Government use and distribution

9





5.3 Quick Incident Response

Another major benefit to incorporating security throughout the SDLC is the ability for quick incident response. If a cyber incident should occur, there are feedback loops and there is team integration into the entire system already established. Since these necessary communication channels are already established, the development team will get the same notification that the security team received and can get to work right away. The alternative, and current state of industry, includes the security team receiving a notification of a cyber incident and performing an assessment. Once that assessment is completed, they notify the development team which then can get to work resolving the issue.

6. Conclusion Implementing security for an application is a difficult problem which is solved in many different ways. It is also an issue that is often avoided because it is seen as extra effort without getting any new features or improvements. However, this type of thinking is flawed, security should be seen as a quality attribute and carry the same, if not more, importance as the other major quality attributes, i.e. availability, usability, etc. This paper discussed the importance of incorporating security at multiple checkpoints throughout the SDLC with a special focus on implementing security from the inception of the project. Implementing security from the start makes it the development, build, and release process much more efficient which allows for more releases into production every day. This paper also focused on how this process should be built on a DevOps platform because it allows for constant collaboration, visibility amongst other stakeholders and the ability to create quick feedback loops. The many communication channels allow project stakeholders to have access to the same, up-to-date information. The DevOps platform also allows for constant automation which means that the security tests and checks put in place can be run automatically and the results can determine whether or not the application moves to the next stage of the SDLC. Security and DevOps work beautifully together if it is implemented carefully. Once a secure SDLC is setup, there are several benefits that arise from the careful planning and constant collaboration. Amongst the major benefits are increased team efficiency, full SDLC traceability, and quick response time to cyber incidents.

[Distribution Statement A] This material has been approved for public release and unlimited distribution. Please see Copyright notice for non-US Government use and distribution

10



7. Acknowledgement

Copyright 2016 Carnegie Mellon University This material is based upon work funded and supported by the Department of Defense under Contract No. FA8721-05-C-0003 with Carnegie Mellon University for the operation of the Software Engineering Institute, a federally funded research and development center. NO WARRANTY. THIS CARNEGIE MELLON UNIVERSITY AND SOFTWARE ENGINEERING INSTITUTE MATERIAL IS FURNISHED ON AN “AS-IS” BASIS. CARNEGIE MELLON UNIVERSITY MAKES NO WARRANTIES OF ANY KIND, EITHER EXPRESSED OR IMPLIED, AS TO ANY MATTER INCLUDING, BUT NOT LIMITED TO, WARRANTY OF FITNESS FOR PURPOSE OR MERCHANTABILITY, EXCLUSIVITY, OR RESULTS OBTAINED FROM USE OF THE MATERIAL. CARNEGIE MELLON UNIVERSITY DOES NOT MAKE ANY WARRANTY OF ANY KIND WITH RESPECT TO FREEDOM FROM PATENT, TRADEMARK, OR COPYRIGHT INFRINGEMENT. [Distribution Statement A] This material has been approved for public release and unlimited distribution. Please see Copyright notice for non-US Government use and distribution. DM-0004267

References Bird, J. (2016, April 12). Static Analysis and Code Reviews in Agile and DevOps. Retrieved November 29, 2016, from https://softwaresecurity.sans.org/blog/2016/04/12/static-analysis-and-code-reviews-in-agileand-devops Bliesner, K. (2015, December 3). Bridging the Gap Between DevOps and Security. Brigham, R., & Liguori, C. (Directors). (2015, October 15). AWS re:Invent 2015 | (DVO202) DevOps at Amazon: A Look at Our Tools and Processes [Video file]. In Youtube. Retrieved from https://www.youtube.com/watch?v=esEFaY0FDKc Burns, S. F. (2005, January 5). Threat Modeling: A Process To Ensure Application Security. SANS Insitute. Retrieved from https://www.sans.org/readingroom/whitepapers/securecode/threat-modeling-process-ensure-applicationsecurity-1646.

[Distribution Statement A] This material has been approved for public release and unlimited distribution. Please see Copyright notice for non-US Government use and distribution

11

Cois, C. A., Yankel, J., & Connell, A. (2014, October). Modern DevOps: Optimizing software development through effective system interactions. 2014 IEEE International Professional Communication Conference (IPCC). doi:10.1109/ipcc.2014.7020388 Finjan. (2016, July 11). Benefits of Penetration Testing. Retrieved November 29, 2016, from https://blog.finjan.com/benefits-of-penetration-testing/ Houmb, S.H., Islam, S., Knauss, E. et al. Requirements Eng (2010) 15: 63. doi:10.1007/s00766-009-0093-9 Kongsli, V. (2006). Towards agile security in web applications. Companion to the 21st ACM SIGPLAN Conference on Object-oriented Programming Systems, Languages, and Applications - OOPSLA '06. Spafford, G., & Haight, C. (2014). Apply Gartner Research for a DevOps Perspective When Implementing a Bimodal Strategy. Retrieved from https://www.gartner.com/doc/2893418/apply-gartner-research-devopsperspective Taschner, C. (2015). Build DevOps Tough! Retrieved November 29, 2016, from https://insights.sei.cmu.edu/devops/2015/04/build-devops-tough.html Tomhave, B., & Kenefick, S. (2014). Security in a DevOps World. Retrieved from https://www.gartner.com/doc/2725217/security-devops-world B. Arkin, S. Stender, and G. McGraw, "Software Penetration Testing," IEEE Security and Privacy, vol. 3, no. 1, pp. 84 - 87, 2005 Vonnegut, S. (2016, February 05). 5 Best Practices for the Perfect Secure Code Review - Checkmarx.com. Retrieved November 29, 2016, from https://www.checkmarx.com/2016/02/05/5-best-practices-perfect-securecode-review/ Humble, J. and D. Farley, Continuous delivery: reliable software releases through build, test, and deployment automation. 2010: Pearson Education Callahan, J., F. Schneider, and S. Easterbrook. Automated software testing using modelchecking. in Proceedings 1996 SPIN workshop. Duvall, P.M., S. Matyas, and A. Glover, Continuous integration: improving software quality and reducing risk. 2007: Pearson Education. Feitelson, D.G., E. Frachtenberg, and K.L. Beck, Development and deployment at Facebook.IEEE Internet Computing, 2013: Day, G.S., The capabilities of market-driven organizations. The Journal of Marketing, 1994: [Distribution Statement A] This material has been approved for public release and unlimited distribution. Please see Copyright notice for non-US Government use and distribution

12

Liu, Y., L. Chengbo, and L. Wei, Integrated Solution for Timely Delivery of Customer Change Requests: A Case Study of Using DevOps Approach. International Journal of U-and E-ServiceScience and Technology, 2014. Marvin, R., Testing in a continuous delivery world, in Software Development Times. 2014.

[Distribution Statement A] This material has been approved for public release and unlimited distribution. Please see Copyright notice for non-US Government use and distribution

13