Recommending Trainings for Developers Based on Static Analysis of Security Vulnerabilities Siraj Muneer
Muhammad Nadeem
Edward B. Allen
Department of Computer Science, BUITEMS Quetta, Baluchistan, Pakistan
Department of CSE Mississippi State University Mississippi State, MS, USA
Department of CSE Mississippi State University Mississippi State, MS, USA
[email protected]
[email protected]
[email protected]
ABSTRACT Vulnerable code may cause security breaches in software systems resulting in loss of confidential data and financial losses for the organizations. Software developers must be given proper training to write secure code. Conventional training methods do not take the code written by the developers over the time into account, which makes these training sessions less effective. In this paper we propose a recommendation system to help identify focused and narrow areas in which developers need training. The system leverages the power of static analysis techniques to suggest the most appropriate training topics for different software developers. Preliminary empirical evaluation shows that the proposed system is promising.
be used as the basis for recommending the most appropriate training topics to the software developers who contributed in writing those modules, hence improving their skill in terms of software security.
2. LITERATURE REVIEW Apart from some of the common recommender systems discussed in previous section, there are many recommendation systems related to the field of software development. Several recommendation system have been proposed and/or built to facilitate the software developers in different ways.
developers to fix the software bugs. A developer enters the contextual information of the bug into the tool, and the tool searches through the repository of projects for bugs with similar context which have been already fixed.
Categories and Subject Descriptors D.4.6 [Security and Protection]: Information flow
General Terms
Keywords Recommender system, security, training, static code analysis, software vulnerabilities Recommender or recommendation systems are special type of information filtering system which seek to predict the 'rating' or 'preference' that a user would give to an item [1]. Recommender systems have been commonly used in recent years for various purposes. Some examples recommend movies [2] to viewers, music to users based on their geographical location [3], news articles through websites or news forums [4], magazines or books to the readers [5], social tags [6] and personalized social updates [8], research articles, and products [7] or services in general. In addition to the common categories, there are also recommender systems for experts, restaurants, financial services, insurances, and twitter followers. The recommendation system proposed in this research is based on the static analysis of the code written by software developers over time. Static analysis tools e.g., FindBugs [9] and others [10] report suspected vulnerabilities present in source code. These results may
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. ACMSE'14, March 28–29, 2014, Kennesaw, GA, USA. Copyright 2014 ACM 978-1-4503-1203-5/12/03...$15.00.
Context based recommendations for software developers [12] help software developers to use the elements of the source code in the integrated development environment (IDE) in smarter ways by identifying the elements which are commonly related. The system provide an efficient way to reach the desired elements.
Security, Experimentation
1. INTRODUCTION
Debug Adviser [18] is a recommender system, which helps
Hipikat [15] is a recommender tool, which scans project's archives and recommends artifacts from the archive, which are related to the software developer's task at hand.
Recommender systems to support requirements elicitation [19]: Requirements elicitation includes complex tasks for which recommender systems can be helpful, examples of such tasks are identification of potential subject matter experts, recommending possible features for stakeholders to consider, and keeping stakeholders informed of relevant issues.
Strathcona [16] is a recommendation tool to assist developers in finding relevant fragments of code, or examples, of an API's use. These examples can be used by developers to provide insight on how they are supposed to interact with the API.
RASCAL [17] is a recommender agent tracks usage histories of a group of developers to recommend to an individual developer components that are expected to be needed by that developer.
Static code analysis [11] is an emerging technique for secure software development that analyzes large software code bases, without execution, to reveal potential vulnerabilities present in the code. Static analysis can evaluate the system or a component of a system based on its form, structure, content, or associated documentation. It is analogous to the analysis of development products that rely on human examination.
modules, the output of the module is the recommendation reports.
3. PROPOSED SYSTEM Our proposed system introduces the strengths of static analysis tools in a recommendation system for identifying the most suitable training topics for software developers.
3.1. Architecture of proposed system As shown in Figure 1, the proposed system has five major components.
Training Delivery Module: Accepts the recommendation reports from the recommendation module and delivers the training modules to the developers using multiple channels including video streaming, tips delivered via an IDE plugin, and email messages containing helpful literature.
4. EMPIRICAL STUDY We conducted an empirical study to explore the feasibility of the Static Code Analysis, Developers’ Performance Assessment, and Recommendations module.
Software Code Repository
Metadata
Static Code Analysis Module
Developers’ Performance Assessment Module
The experiment simulates the core of the proposed automated system. We used FindBugs [9] for static analysis of the code repository. FindBugs analyzes Java code bases and finds potential vulnerabilities in the code. In the first phase of our architecture, selections from the code repository are passed to the Static Analysis module as shown in the Figure 1. The results of the static analysis are then sent to the Developers’ Performance Assessment module which fetches the metadata from the code repository and combines it with the results from the Static Analysis module to profile the performance of each developer.
Recommendation Module
Recommendation reports
4.1. Method
Training database
Training Delivery Module
4.2. Data collection The code being tested is a custom built web application using JSP and Java Servlets technology and deployed on an Apache Tomcat Server. The application was built by a team of professional Java developers for a university to store students’ records. A summary of the case study system is shown in Table 1. Table 1. Summary of system studied
Video streaming
IDE plugin
Email and others
Developer Figure 1. Architecture of the proposed system
Software Code Repository: Contains code and metadata which is analyzed by the recommendation system.
Static Code Analysis Module: Contains one or more static analysis tools which scan the given code repository to find vulnerabilities. The output of this phase is an XML based report containing details about detected vulnerabilities (e.g., Filename, location, vulnerability description).
Developers’ Performance Assessment Module: Takes the results of the static analysis tool(s) and metadata of the software repository as input and creates a profile for each developer. A profile primarily contains the vulnerabilities induced by each developer in the code over time. The profiles are fed into the recommendation module.
Recommendation Module: Takes the profiles of developers as input and queries the database based on the information in the developer’s profile to find most appropriate training
Number of modules
101 (Java source files)
Lines of code
32 KLOC
Total Vulnerabilities detected
302
Vulnerabilities / KLOC
9.4
In keeping with the small scale of this exploratory study, we selected three developers, and three modules written by each. The developers are labeled here as A, B and C, and the modules written by them are Module A1, A2, A3, B1, and so on. Table 2 shows the lines of code for each module, and the count of vulnerabilities found in each module by FindBugs. Vulnerabilities are categorized by FindBugs. The Developers’ Performance Assessment module fetches metadata from the code repository and links it with results of Static Analysis module. In addition, it also adds risk information associated with each category. The results of the developers’ Performance Assessment Module are summarized in Table 3. Based on the output of the Developers’ Performance Assessment module, the Recommender module queries the training database and finds training module associated with each category. There could be multiple training modules associated with each category of vulnerabilities. The training Recommendation module also assigns a priority to each training for each developer.
Table 2. Details of selected modules
Table 4. Training recommendation
Developer
Modules
LOC
Vulnerabilities reported
Total
Developer A
A1
89
1
4
A2
48
2
A3
92
1
B1
63
5
B2
1018
24
B3
243
13
C1
164
2
C2
303
2
C3
571
5
Developer B
Developer C
A
B 42
9 C
In a fully functional system, the output of the training Recommendation module is directly sent to the Training Delivery module, the Table 4 gives an idea how the output could be structured. Table 3. Output of developers’ assessment module Developer A B
C
Vulnerabilities
Category
Risk
#
Total
Security: SQL
High
2
4
Performance
Low
2
Security: Malicious Code
High
36
Performance
Low
5
Dodgy Code
Low
1
Security: SQL
High
3
Dodgy Code1
Low
3
Bad Coding Practice
Medium
3
42
9
The list of training topics was manually taken from websites that specialize in delivery of security related training. The Training Delivery module would take reports generated by the Recommendation module and is responsible for delivering training to the individual developers. As shown in Figure 1. Training Delivery module delivers training in multiple possible ways, for instance, a tip to the developer with the help of a plug-in integrated to the developer’s IDE, or as a video streamed to the developer’s computer. However in our exploratory experiment, the actual training was not delivered.
4.3. Analysis Based on the results of manual evaluation of the proposed system, we concluded that the system is feasible to work in an operational environment. In software organizations with hundreds of developers and fairly large projects, the training need assessment becomes challenging, we anticipate that the system will be more effective than the conventional training given to developers as it takes into account the source code written by the developers prior to recommending trainings.
1
Developer
“Dodgy code” means an error prone style of code
Training Recommended
Priority
Avoiding SQL injection
High
Writing secure code
High
Writing efficient code
Low
How to avoid Malicious Code
High
Writing secure code
High
Writing code that does not break
High
Writing efficient Java code
Low
Dodgy Code
Low
Avoiding SQL injection
High
Dodgy Code
Low
Best coding practices in Java
Medium
4.4. Threats to validity Though the results of our exploratory feasibility study are promising, but there are some generic limitations of the static analysis tools which may introduce some inaccuracies or limitations in our system. A false positive is a detection of a vulnerability by a static analysis tool in the given code which actually does not exist. Software development teams can spend many man-hours to track and fix the discovered vulnerabilities [20]. Usually a high percentage of discovered vulnerabilities are actually false positives [13]. The results generated by such tools can cause unproductive consumption of time and resources [14]. The occurrence of false positive detections implies out system will make training recommendations to the software developers which are not actually needed. A false negative detection refers to a vulnerability that actually exists in a software code repository but the static analysis tools is unable to detect. False negative detections means our system will not recommend training to the software developers which should have been recommended. Our experiment had a very limited scale which may not be realistic of industry projects. In the future we plan to test our approach on open source large scale systems.
5. CONCLUSION AND FUTURE WORK In this work we have proposed a training recommendation system which utilizes the power of static analysis tools to scan the code repositories. The proposed system combines the results of static analysis tool with the metadata of the software repository to profile software developers, and finally recommends training for each developer based on their profiles. Small scale experiment showed that the system has potential. Finally we have highlighted some of the limitations of the system, most important of which is the high number of false positive results generated by the static analysis tools. In the future we will make use of publically available software vulnerability repositories and the countermeasures proposed to fix these vulnerabilities as additional input to recommendations.
[12] Antunes, B., Cordeiro, J., and Gomes, P. 2012. An approach to
6. ACKNOWLEDGMENT Authors acknowledge the valuable input of Byron J. Williams and R. Wesley McGrew in finalizing the architecture of the proposed recommender system.
7. REFERENCES [1]
Recommender system, Recommender_system
[2]
Azaria, A., Hassidim, A., Kraus, S., Eshkol, A., Weintraub, O., and Netanely, I. 2013. Movie recommender system for profit maximization. In Proceedings of the 7th ACM conference on Recommender systems (Hong Kong, Oct. 12-16, 2013). RecSys '13. ACM, New York, NY, 121-128. DOI= http://dx.doi.org/10.1145/2507157.2507162
[3]
Kaminskas, M., Ricci, F., and Schedl, M. 2013. Locationaware music recommendation using auto-tagging and hybrid matching. In Proceedings of the 7th ACM conference on Recommender systems (Hong Kong, Oct. 12-16, 2013). RecSys '13. ACM, New York, NY, 17-24. DOI= http://dx.doi.org/10.1145/2507157.2507180
[15] Cubranic,
[4]
Garcin, F., Dimitrakakis, C., and Faltings, B. 2013. Personalized News Recommendation with Context Trees. In Proceedings of the 7th ACM conference on Recommender systems, (Hong Kong, Oct. 12-16, 2013). RecSys '13. ACM, New York, NY, 105-112. DOI= http://dx.doi.org/10.1145/2507157.2507166
[16] Holmes, R., Walker, R. J., and Murphy, G. C. 2005. Strathcona
[5]
Pera, M. S., and Ng, Y. K. 2013. What to read next? Making personalized book recommendations for K-12 users. In Proceedings of the 7th ACM conference on Recommender systems, (Hong Kong, Oct. 12-16, 2013). RecSys '13. ACM, New York, NY, 113-120. DOI= http://dx.doi.org/10.1145/2507157.2507181
[6]
http://en.wikipedia.org/wiki/
context-based recommendation in software development. In Proceedings of the sixth ACM conference on Recommender systems, (The Dublin, The Ireland, Sep. 9-13, 2012). RecSys '12. ACM New York, NY, 171-178. DOI= http://dx.doi.org/10.1145/2365952.2365986
Belém, F., Santos, R., Almeida, J., and Gonçalves, M. 2013. Topic diversity in tag recommendation. In Proceedings of the 7th ACM conference on Recommender systems (Hong Kong, Oct. 12-16, 2013). RecSys '13. ACM, New York, NY, 141-148. DOI= http://dx.doi.org/10.1145/2507157.2507184
[7]
Zhang, Y., and Pennacchiotti, M. 2013. Recommending branded products from social media. In Proceedings of the 7th ACM conference on Recommender systems, (Hong Kong, Oct. 12-16, 2013). RecSys '13. ACM, New York, NY, 77-84. DOI= http://dx.doi.org/10.1145/2507157.2507170
[8]
Pan, Y., Cong, F., Chen, K., and Yu, Y. 2013. Diffusion-aware personalized social update recommendation. In Proceedings of the 7th ACM conference on Recommender systems, (Hong Kong, Oct. 12-16, 2013). RecSys '13. ACM, New York, NY, 69-76. DOI= http://dx.doi.org/10.1145/2507157.2507177
[9]
FindBugs TM, Find bugs http://findbugs.sourceforge.net/
in
Java
Programs,
[10] List of tools for static code analysis, http://en.wikipedia.org/
wiki/List_of_tools_for_static_code_analysis [11] J. Zheng, L. Williams, N. Nagappan, W. Snipes, J. P.
Hudepohl, and M. A. Vouk, On the Value of Static Analysis for Fault Detection in Software, IEEE Transactions on Software Engineering, 32, 4, (Apr. 2006), 240-253. DOI= http://dx.doi.org/10.1109/TSE.2006.38
[13] Austin, A. and Williams, L. 2011. One Technique is Not
Enough: A Comparison of Vulnerability Discovery Techniques, In Proceedings of the 5th International Symposium on Empirical Software Engineering and Measurement (Banff, Canada, September 22-23, 2011). ESEM’11. CPS, Los Alamitos, CA, 97-106. DOI=http://dx.doi.org/10.1109/ESEM.2011.18 [14] Bessey, A., Block, K., Chelf, B., Chou, A., Fulton, B., Hallem,
S., Henri-Gros, C., Kamsky, A., McPeak, S., and Engller, D. 2010. A few billion lines of code later: using static Analysis to find Bugs in the Real World, Communications of the ACM, 53, 2, (Feb. 2010), 66-75. DOI= http://dx.doi.org/10.1145/1646353.1646374 D., and Murphy, G. C. 2003. Hipikat: Recommending pertinent software development artifacts. In Proceedings of the 25th International Conference on Software Engineering, (San Francisco, CA, May 18-26, 2003). ICSE ’13. IEEE Computer Society, Washington, DC, 408-418. DOI= http://dx.doi.org/10.1109/ICSE.2003.1201219 example recommendation tool. In ACM SIGSOFT Software Engineering Notes, 30, 5, (Sep. 2005). ACM, New York, NY, 237-240. DOI= http://dx.doi.org/10.1145/1095430.1081744 [17] McCarey, F., Cinnéide, M. Ó., and Kushmerick, N. 2005.
RASCAL: A recommender agent for agile reuse. Artificial Intelligence Review, 24, 3-4, (Nov. 2005). Norwell, MA, 253276. DOI= http://dx.doi.org/10.1007/s10462-005-9012-8 [18] Ashok, B., Joy, J., Liang, H., Rajamani, S. K., Srinivasa, G.,
and Vangala, V. 2009. DebugAdvisor: a recommender system for debugging. In Proceedings of the 7th joint meeting of the European software engineering conference and the ACM SIGSOFT symposium on The foundations of software engineering, (The Amsterdam, The Netherlands, Aug. 24-28, 2009). ESEC/FSE '09. ACM, New York, NY, 373-382. DOI= http://dx.doi.org/10.1145/1595696.1595766 [19] Castro-Herrera, C., and Cleland-Huang, J. (2010, May).
Utilizing recommender systems to support software requirements elicitation. In Proceedings of the 2nd International Workshop on Recommendation Systems for Software Engineering, (Cape Town, South Africa, May 2-10, 2010). RSSE '10. ACM, New York, NY, 6-10. DOI= http://dx.doi.org/10.1145/1808920.1808922 [20] Nadeem, M., Williams, B. J., and Allen, E. B. 2012. High false
positive detection of security vulnerabilities: a case study. In Proceedings of the 50th Annual ACM Southeast Regional Conference, (Tuscaloosa, AL, Mar. 29-31, 2012). ACM-SE '12. ACM New York, NY, 359-360. DOI=http://dx.doi.org/10.1145/2184512.2184604
