Study The Impact of Improving Source Code on Software Metrics Qosai Zoubi CIS Department Computer science and IT Faculty Yarmouk University Irbid, Jordan
[email protected]
Izzat Alsmadi CIS Department Computer science and IT Faculty Yarmouk University Irbid, Jordan
[email protected]
Abstract— The process of improving the quality of the software products is a continuous process where software developers learn from their previous experience and from previous software releases to improve the future products or release. In this paper, we evaluate the ability of Software source code analysis process and tools to predict possible defects, errors or problem in the software products. More specifically, we evaluate the effect of improving the code according to recommendations from source code tools on software metrics. Several open source software projects are selected for the case study. The output of applying source code analysis tools on those projects result in several types of warning. After performing manual correction of those warning, we compare the metrics of the evaluated projects before and after applying the corrections. Results showed that the size and structural complexity in most cases are increases. On the other hand, some of the complexities related to coupling and maintainability are decreases1. Keywords: Source code analysis, software maintainability, software quality, and software testing.
I.
metrics,
INTRODUCTION
The size and complexity of software products are continuously increasing. This is since those programs become more complex as a result of increasing their functionalities, requirements, improvements, modifications, etc. Software quality is always a concern as problems related to software quality may consume a large amount of time and resources of the software projects during the development stages or even after the release of the software. Software testing includes several processes and activities that are conducted all over the software development stages. Testing occurs for the requirements, design, code, user interface, etc. As testing is continuous and needs a large amount of long and tedious activities, the majority of goals in the testing stage focus on automating the testing activities. Source Code Analysis (SCA) tools are different in concept from software testing tools and software metrics. Software testing tools are programs that are trying to find errors, defects, bugs, failures, etc in the software. Those different terms are, sometime, different based on the level and the nature of the 1
This paper is support by Scientific Research Faculty at Yarmouk University
Bilal Abul-Huda CIS Department Computer science and IT Faculty Yarmouk University Irbid, Jordan
[email protected]
error. The output of each test case in a testing process is one of two: pass or fail. The designer of the test cases, define the inputs for each test case along with expected outputs. In execution, test cases are executed and actual results are compared with expected results. For those failed test cases (i.e. expected result is different from the actual result), a debugging process further starts to see why those test cases produce incorrect outputs or results. Errors can be syntax, semantic, functional, non-functional, etc. Error may stop the compilation process or may not and only cause different or unexpected behavior from those defined by users. On the other hand, software metrics are programs or tools that collect information about the software regarding its characteristics. Examples of such metrics include: Lines Of Code (LOC), size, complexity. Number of functions per class, etc. While the high level goal of software metrics is similar to software testing in improving the overall quality of the software, there are several differences between software testing and metric tools. Software metrics are usually collected after testing and after making sure that program is free from errors (at least syntactic errors). The level of quality attributes that metrics try to measure is also different from those that testing measures. For example, there are many metrics (e.g. complexity, fan in, fan out, depth of inheritance) that try to measure design quality attributes. Such attributes may not be direct evaluated through testing. One more difference is that the goal of collecting software metrics is usually for the improvement of the future releases of the software and not the current one. As a third category of software quality tools, SCA are programs that collect information about other aspects of the software. While SCA tools may produce information about (errors), however, first the type of errors SCA tools focus on are different of those that software testing tools focus on. On the other hand, the majority of the SCA functions focus on findings early for, possible, not confirmed errors. Those, possible, errors, or programming mistakes that may make future problems are classified as Warning. The other category of issues that SCA tools collect are classified as Information where such information is not likely going to produce possible errors in future. In the next paragraphs, we will introduce two
examples of SCA tools that we used in this paper and evaluation. A.
StyleCop
StyleCop is an open source SCA tool from Microsoft that checks .NET code for conformance of several design guidelines defined based on Microsoft's .NET Framework [6]. In StyleCop the rules are classified in to several categories [6]: Naming, maintainability, documentation, ordering, readability, spacing, and layout. Table 1 shows a sample of some warnings and their classification. Table 1. A sample SCA warning classification.
Warnings The spacing around an operator symbol is incorrect. The call to Channel should only use the 'base.' prefix if the item is declared virtual in the base class and an override is defined in the local class. Otherwise, prefix the call with this rather than base. All using directives must be placed inside of the namespace Method names begin with an uppercase letter. The class must have an access modifier A statement containing curly brackets must not be placed on a single line. The opening and closing curly brackets must each be placed on their own line. The constructor must have a documentation header.
B.
categories Spacing
Readability
Ordering Naming Rules Maintainability Layout Documentation
JustCode
JustCode is another example of SCA tools. There are some JustCode features that include [7]: On-the-fly Code analysis, code navigation and search, refactoring, quick fixes, coding assistant and hints. JustCode executes its code analysis by applying custom inspections. There are several inspects that can be performed by JustCode. Examples include [7]: Identical if and else clauses, obsolete casts, empty statements, assignments with no effect, unused private members, unused parameters, variables, namespaces, or statements. Figure 1 shows a sample of SCA output from JustCode.
Errors - by default JustCode underlines errors with a red curly line
Warnings - JustCode underlines warnings with a green curly line Figure 1. A sample of SCA output.
C.
Refactoring
This is a process of improving the software cosmetically without adding or changing functionalities. The majorities of the tasks performed by SCA tools can be classified under refactoring activities. Examples of those can fall under the categories: spacing, readability, ordering, naming rules, layout, and documentation (shown in Table 1). The exception is those warning that are classified under the maintainability category. Those can have deeper impact on the code. II.
RELATED WORK
Automatic source code analysis tools consist of four main parts: model construction, analysis and pattern recognition algorithms, patterns knowledge and result representation. Model is constructed from the source code or the binary file of the program under analysis, the analysis and pattern recognition algorithms are looking at the model in order to find some probable problems. The patterns are stored in a separate block called Patterns Knowledge. Design patterns describe some generic solutions and the best practices to recurring software problems including both structural and behavior aspects of the program (Kirkov and Agre , 2010 [3]). The static analysis approach is meant to review the source code, checking the compliance of specific rules, usage of arguments and so forth; the promise of static analysis is to identify many common coding problems automatically before a program is released. Static analysis aims to examine the text of a program statically, without attempting to execute it. The dynamic approach is essentially executing the code, running the program and dynamically checking for inconsistencies of the given results. This means that testing and reviewing code are separate and distinguishable things, but it is unadvised that one should occur without the other (Gomes, Morgado, Gomes, and Moreira, 2008 [5]). Static analysis for software defect detection has become a popular topic; there are many contexts for using static analysis for defect detection. One context is when performing code review of a newly written module. Another context is when you wish to look for defects in a large existing code base. FindBugs is an open source static analysis tool that analyzes Java Class files looking for programming defects. The analysis engine reports nearly 300 different bug patterns; each bug pattern is grouped into a category (e.g., correctness, bad practice, performance and internationalization), and each report of a bug pattern is assigned a priority, high, medium or low (Ayewah et al, 2007 [1]). One such fault-detection technique is static analysis, the process of evaluating a system or component based on its form, structure, content, or documentation, which does not require program execution, SCA can be used as an added faultdetection filter in the software development process, SCA tools automate the identification of certain types of anomalies, as discussed above, by scanning and parsing the source text of a program to look for a fixed set of patterns in the code. An important benefit of ASA tools is that they do not necessitate
Priority 1 2 3 4 5 6 7
Table 3 shows a sample from JustCode SCA tool of Warnings on three evaluated open source projects. Table 3. Sample warnings from JustCode
Warning The namespace doesn't match the project default namespace & directory This using directive is not required Type name does not match file name Unused Field Unused Method Unused Parameter
Chatters
Draw √
Splitting
√ √ √ √ √ √
√
√
√
√
Comparing JustCode and StyleCop SCA tools shows that StyleCop issues are more on the issues related to refactoring while JustCode evaluation issues are deeper toward design quality and maintainability issues. Table 4 shows a summary of comparison between the two SCA tools.
√ √
Increase Documentation
Table 2. SCA Categories’ priority
Category Maintainability Rules Naming Rules Readability Rules Spacing Rules Ordering Rules Layout Rules Documentation Rules
√ √
Increase Maintainability
We run the SCA tools on the three source codes downloaded from the SourceForge.NET to do experiment and find metrics result on it. The programs selected are Chatters, Draw_with_Mouse_src, and Splitting. In comparing the StyleCop tool results, we have found that the warnings in the results are divided into several categories: Naming, Maintainability, Documentation, Ordering, Readability, Spacing, and Layout. After analyzing these categories, we gave each category a priority number based on its importance and its impact on the program. Table 2 shows the priority number given to categories based on their importance.
√
Increase Usability
EXPERIMNT AND ANALYSIS
StyleCop JustCode
Increase Performance
III.
Table 4. SCA tools comparison Tool
Increase Complexity
execution of the subject program yet infer results applicable to all possible executions (Zheng and Nagappan, 2006 [5]). Testing and static analysis complement each other. Static analysis is less feasible without source code. It may be impossible, say, in testing embedded systems or remote testing of Internet services. On the other hand, one cannot test an incomplete program, while static analysis might be feasible. More importantly, testing is unlikely to uncover very special cases (Black, 2007 [2]).
√
As mentioned earlier, the major goal in this paper is to evaluate the correlation between SCA tools and software metrics. In order to do this, several open source code projects are selected for this case study. Table 5 shows a summary of those projects. As it can be seen, the four projects vary in terms of size. Table 5. A summary of case study under test
Name Chatters Splitting Draw Sol_AIEasy
LOC 2506 291 709 4060
Files 15 2 7 20
LocMetrics and SLOC Metrics 3.0 metrics tools are used to evaluate the difference in the evaluated software project before and after applying the modifications recommended by SCA tools. In this experiment we only focus on evaluating the impact of SCA modifications on software structural metrics. This is why metrics selected are basics (e.g. Lines of Code LOC, Comments, BLOC, etc). All those metrics focus on the count of the lines of code. The difference is that some of them count all lines of code no matter of their purpose. Others exclude comments, headers, etc. Table 6 shows the difference in metrics before and after applying the SCA modifications. Values are absolute which means that we didn’t differentiate whether the metric value is increased or decreased. The results in Table 6 show a large variation between the evaluated metrics. While some metrics showed no difference at all as a result from applying SCA modifications, some other metrics (e.g. LOC, and CWORD) showed relatively large variations as a result of SCA modifications.
Maintainability metrics
Table 6. SCA impact analysis of the first metric tool Proj. SLOC SLOC LOC MVG Physical Logical Chatters
BLOC
61
39
12
4
3
2
25
12
0
36
10
9
4
0
19
84
21
6
0
8
Splitting Draw SolAIEasy
C&SLOC
CLOC
CWORD
HCLOC
HCWORD Table 8. Impact of SCA modifications on Selected maintainability metrics
Chatters Splitting Draw
0
20
126
0
0
1
55
314
0
0
Table 7 shows the results from the second software metric tool. Results also confirmed that some metrics are impacted much more significantly than others. Variations from one tool to another are also different based on the size of the project. Table 7. SCA impact analysis of the second metric tool
SLOC
Comments
Chatters
971
25
7
1,003
Splitting
501
13
30
484
20
19
10
55
8
84
Draw Sol-AIEasy
9 21
BlankLines
Total
Modifications of the recommendations from SCA tools were implemented automatically and manually. Some of the recommendations can be implemented by the SCA tools themselves as they have options to automatically fix or apply those modifications. Some through recommendations need more complex steps and hence should be implemented manually. In many cases, specially for comments related to documentation, naming, ordering, we tried to implement quick trivial modifications to mainly get rid of their warnings. As we mentioned earlier, the focus in this paper was on studying the impact of SCA modifications on structural metrics. We believe that applying modifications of more through and complex warning such as those related to the category “maintainability” mentioned earlier, require more through solutions and is expected to impact several metrics related to complexity, coupling, etc.
Difference between the metrics before and after applying SCA
WMC
0
NIV
0
NIM
92
RFC
13
NOC
0
CBO
0
FANIN
0
DIT
235
LCoM
25
Project
0
SolAIEasy
Proj.
We also measured the impact of SCA modifications on other types of metrics, particularly, coupling or maintainability metrics. Those are metrics that showed relations with future easiness to fix, update or modify the software code. Table 8 shows a sample of those metrics. More details and references about those metrics can be found in related papers. Results in Table 8 shows that very few of those complexity or maintainability metrics were affected by SCA modifications. Other similar metrics are evaluated and showed somewhat similar results.
Chatters
0
0
0
1
0
1
0
0
0
Draw
0
0
0
0
0
0
0
0
0
Splitter
0
0
0
3
0
0
0
0
0
AI-Easy
0
0
0
0
0
0
0
0
0
IV. CONCLUSION Software testing consumes a large percentage of software projects’ time and resources. Several alternative methods are used to improve the quality of software products. In this paper, we evaluated the impact of applying modifications recommended by software code analysis tools (SCA) on software metrics. A case study of several open source projects is assembled. Two software metric tools are also selected to compare the characteristics of the evaluated software products before and after applying SCA modifications. Some modifications recommended by SCA tools are implemented automatically by the tools themselves, while others are implemented manually in a generic format with no through analysis. Results showed that there is a variation in the impact of SCA modifications on software metrics. Some metrics were significantly affected by those modifications, while other metrics show small or no sign of impact. The focus in this paper was on evaluating the impact on structural metrics. In future, through evaluation of the impact on maintainability metrics will be conducted as those metrics are more important in terms of software code and design quality.
REFERENCES [1] Ayewah, N., Pugh, W., Morgenthaler, J., Penix, J., & Zhou, Y. (2007). Evaluating Static Analysis Defect Warnings On Production Software. ACM. [2] Black, P. (September 2007). SAMATE and Evaluating Static Analysis Tools. Ada User Journal, 28. [3] Kirkov, R., & Agre, G. (2010). Source Code Analysis – An Overview. Bulgarian Academy of Sciences, 10. [4] Gomes, I., Morgado, P., Gomes, T., and Moreira, R. (2008). An overview on the Static Code Analysis approach in Software Development. [5] Zheng, J., & Nagappan, N. (2006). On the Value of Static Analysis for Fault Detection in Software. IEEE Transactions on Software Engineering, 4. [6] StyleCop.Codeplex. Web site Available at:http://stylecop.codeplex.com/ Accessed October 8, 2011. [7] JustCode.Web site Available at:http://www.telerik.com/products/ justcode.aspx. Accessed October 15, 2011. [8] Source Forge. Web site Available at:http://sourceforge.com. Accessed October 23, 2011.
