PIVoT: Project Insights and Visualization Toolkit

PIVoT: Project Insights and Visualization Toolkit Vibhu Saujanya Sharma Accenture Technology Labs Bangalore, India [email protected]

Abstract—An in-process view into a software development project’s health is critical for its success. However, in services organizations, a typical software development team employs a heterogeneous set of tools based on client requirements through the different phases of the software project. The use of disparate tools with non-compatible outputs makes it very difficult to extract one coherent picture of the project’s health and status. Existing project management tools either work at the process layer and rely on manually entered information, or are activity centric, without a holistic view. In this paper, we present PIVoT, a metric-based framework for automated, non-invasive, and in-process data collection and analysis in heterogeneous software project environments, that provides rich, multi-dimensional insights into the project’s health and trajectory. Here, we introduce the different analyses, insights and metrics, and discuss their usage in typical software projects. Keywords-Software development management; Project management; Software metrics;

I. I NTRODUCTION Software development consists of a complex set of activities, with different tools and processes to aid the development team. Ideally, one should be able to gauge a software development project’s current health as well as its future trajectory in terms of the development quality, team dynamics and process adherence. Lack of early insights into such health and quality trends can lead to significant schedule and cost over-runs later. Note that though each step in the software development process generates a set of project data and events, the task of gathering, consolidating and analyzing data and events from software development environment in near real time and ascertaining actionable insights from the same is not quite as simple. While in a few cases, the project environment can be homogeneous, i.e., a family of related tools forming an integrated platform like IBM RTC or Microsoft VSTS [1], [2], typical environments utilize a heterogeneous toolset with many discrete and disparate tools. The latter is much more prevalent in software services organizations wherein each project team is usually mandated to use a different set of tools based on client needs, unlike a more standardized toolset used in product development companies. The tools used in such heterogeneous software development environments are usually not designed to be used with each other

Vikrant Kaulgud Accenture Technology Labs Bangalore, India [email protected]

and have different ways and formats in which they output their events and metrics. This poses a challenge in terms of collecting in-process project data and gather useful insights from the same. Interestingly while most organizations have a set of mature processes, methods and metrics, there are no tools to objectively link these with different fine grained activities in the software development environment. Currently this is manual and subjective, based on human input. While homogeneous collaborative development environments like IBM RTC [2] and Microsoft VSTS [1] integrate different tool outputs to provide a cross project view, they provide very limited support for composite metrics and team analysis. Moreover they are not applicable in a majority of projects that utilize a heterogeneous set of tools. Other open source tools like Hackystat[3] and Sonar [4] act as platforms to execute a set of supported tools and collect their outputs, but have very limited ability to output composite insights and metrics and do not provide in-built support for studying project team dynamics. In this paper, we introduce PIVoT - Project Insights and Visualization Toolkit, which allows for in-process, automated and non-invasive collection and analysis of project environment data and presents a set of actionable insights and metrics to visualize the current and future trajectory of the project. PIVoT utilizes a standard project schema to classify in-process data coming from different tools which helps to overlay a set of relationships between the disparate data elements. PIVoT has a rich set of analyses which can provide composite metrics and insights in the dimensions of the process and activities, development quality, and team dynamics. PIVoT has been piloted in more than 50 projects since the first version was released recently in our organization, and the projects have reported significant benefits in terms of quality and efficiency improvements as well as cost savings by using PIVoT. In the next section, we describe the usage and internals of PIVoT in greater detail. We then describe the set of inprocess insights and analyses provided by PIVoT. In Section 4, we discuss, compare, and contrast some of the related work in this area. Finally, we conclude this paper in Section 5.

Figure 1.

PIVoT uses adapters in the project environment for in-process insights

II. PIVOT A PPROACH PIVoT is based on collecting project data using noninvasive adapters in a heterogeneous software development environment. In this section we provide more details on its internals. 1) PIVoT Internals: PIVoT consists of a extensible set of adapters, an internal database (repository), an analysis engine and a reporting module and can consume diverse set of tool outputs and logs typically used in custom development projects (see Figure 1). PIVoT facilitates single installation of the tool to be used in a delivery unit wherein multiple projects can be configured in PIVoT. Each project gets a unique name and is associated with a unique source control server / trunk. This allows data collection and analysis to be performed on multiple projects and subsequently benchmarking projects on critical metrics and insights. PIVoT relies on high fidelity data collected from project tools using non-invasive data adapters for analysis and reporting and does not require the project to change the development process.The data adapters hook into the project tools and access their output upon execution. E.g. PIVoT accesses the output produced by PMD, a popular Java quality tool, whenever PMD is executed by a developer or a build server. In the current implementation, PIVoT executes different tools like code quality, code coverage etc. as part of its processing tool-chain. During configuration, for each project the PIVoT user selects the tools supported by PIVoT that need to be executed. Tools in the following categories are selected- code quality (PMD, Checkstyle, Oracle code quality), unit test coverage (Cobertura, Oracle unit test coverage) and source control repository (IBM Clearcase, Subversion, CVS). A snapshot of the Project and Source Control Configuration window is shown in Figure 2. The user then configures the project milestones - start and end dates of the build and unit testing phases. PIVoT can be

Figure 2.

PIVoT Configuration

executed either at periodic intervals or integrated with a continuous integration (CI) system. If the projects are not using CI, the former provides more control to the PIVoT user in terms of how frequently is data collected and analyzed and hence is preferable. 2) PIVoT Project Model: PIVoT internally uses a project model which embodies the relationships between the different project roles, events, artifacts, and metrics generated by different tools. The model was built over time by studying the usage of different events and tools typically associated with delivery projects. We gathered typical project events and metrics used in academia as well as industry and tried to put down the relationships between them in form of a concept map. Note that with each event, metric, or artifact there maybe a human involvement and thus we need to include project roles also in this model. The model links thus events such as a build to other events and data like build violations, modifications, metrics, etc., which though

Figure 3.

PIVoT Project Model: Subset of the Concept Map and a Formal DTD

are generated by disparate tools, correspond to that build. A modification is then linked to an owner who could be a developer in this case. This concept map was then formalized to create a schema for the internal database of PIVoT a part of which is shown in Figure 3. Each first level element in this model is associated with a set of attributes which define an instance of that element. In the equivalent database representation, these elements form the tables and their columns respectively. For example, a Modification (Figure 4) has attributes like modificationID, buildID, modificationType, className, module, package, projectName, timeStamp, owner, changeNumber, linesAdded, linesDeleted and linesModified. Out of these, the modificationID is a unique number for each build, and thus modeificationID forms the primary key of the modification table, in the database representation of the same. As a set of modifications are related with each build, the buildID forms the foreign key for this table. At run time, the database corresponding to this model is populated by gathering data and consolidating different ids (like buildid, projectid, etc) and timestamps from the corresponding tools. Note that the assumption here is that the timestamps on different systems are reasonably accurate which is true in most cases, since the tools typically either run in a chained fashion on a single build machine, or run on a shared well-managed project infrastructure. Once the data from different tools is populated, PIVoT can generate the set of insights explained next. III. PIVOT A NALYSES AND I NSIGHTS PIVoT generates 30 metrics and insights categorized in following 6 categories. These are outputted in form of a set of HTML reports corresponding to each category. 1) Code Quality: Code quality is an important facet of project’s quality tracking parameters. PIVoT enables the project manager to track code quality using the following insights.

Figure 4.

PIVoT Project Model: Modification Schema

a) Code Quality Index (CQI): CQI is a score given to the code quality on a scale of 1 (lowest) to 10 (highest). CQI is calculated using a notion of ”code quality violation density”, i.e. more violations per KLOC, poorer is the code quality. To calculate the violation density, the violations reported by a code quality tool are aggregated based on their severity, e.g. critical, warning, information. Further weights are attached to each aggregate to penalize critical severity aggregate and reduce the effect of the warning aggregate. This needs to be done to handle the typical severity configured for different rules in a code quality tool and prevent a high number of ”informational violations” from reducing the code quality score. This decision was also based on the observation that project managers tend to look at ”actionable” insights, and if one were to inform them that the code quality index is poor due to a very large number of informational violations, they would lose confidence on the score. In the current implementation of PIVoT we assign a weightage of 0.2 to informational violations aggregate and 0.8 to the warning violations aggregate. The

critical violations aggregate is weighted with a factor 1.0. Further the weighted aggregates are summed together to calculate the total violations in the code and using the non commenting lines of code metric, the violation density is computed. Using a mapping table, the violation density is converted into a CQI score. CQI is calculated for the entire source code the project as well as for modified code, the latter reflecting the quality of the code produced in period since last build / PIVoT execution. b) Top code quality issues: As further actionable data, PIVoT identifies the top code quality issues (by frequency of occurrence) and also the top code quality issue categories in the code. Further, PIVoT also ranks packages / modules on basis of the code quality issues in them. This information allows the project manager to create a code quality improvement plan. c) Code quality patterns: Correlating code churn information, PIVoT identifies code quality issues that continue to persist in the code over multiple builds. These represent the violations that persist in the code in spite of multiple builds and perhaps code remediations exercises done during these builds. If the persisting code quality violations are of critical severity, the project manager can take proactive actions to conduct code review and refactoring specifically for these set of violations. Further, if the persisting violations are of warning severity, the project manager can bump up their severity to critical so that the developers are forced to take cognizance of these repeating violations. Importantly, PIVoT measures rate of increase of quality issues. This is a trend graph showing the slope of increase of quality issues. A high slope means that developers are introducing code quality violations faster than they can refactor and remove. The interesting correlation we provide in PIVoT is of code churn and this rate. If PIVoT observes that the rate of increase of quality issues is positively correlated to code churn, then it is definitely a cause of worry for the project manager. It means that the developers are modifying code or creating new code, but are not concerned about the code quality. Moreover, if this correlations exists close to project milestones such as start of testing cycle, it becomes a red flag for the project manager and triggers an intervention in terms of changing the project plan and / or increasing effort on code review and refactoring.

2) Quality of Component Testing Effort (QCTE): QCTE is a metric specifically designed to evaluate the effectiveness of unit testing (sometimes referred to as component testing). We defined is as [5]: QCTE = ElapsedEffort (1 - UnhealthyComponents / TotalComponents), where the total components is the number of components (classes) in the code and unHealthyComponents are the components (classes) showing poor code coverage or deteriorating code coverage. Loading the metric with ElapsedEffort gives an indication of whether the unit testing effort has resulted in well-tested components (classes). PIVoT also applies a forecasting model to this metric to predict if the project will manage to achieve high code coverage and effective unit testing in the allocated unit testing schedule. Apart from this metric, PIVoT also shows key unit testing insights on untested components and poorly tested components categorized as per their complexity. 3) Development Efficiency: In typical IT services organizations, one of the key tasks of the project manager is control the team costs by having a suitable mix of senior and junior resources. It has been observed that while costs can be controlled by managing this co-called pyramid, the project manager typically loses sight of the ’quality of artifacts’ produced. Therefore, a project may be excellent in terms of controlled costs, but poor in terms in quality of work. This skew leads to additional rework effort, often negating the controlled costs! For providing the project managers an insight in to this, we use a new metric called Cost of Development Index (CDI). CDI is based on the principle that a development team needs to work towards high quality code in terms of good code quality and adequate unit testing. For objectively measuring this, PIVoT establishes the baseline code quality and unit testing coverage for the code. The code quality is measured using CQI. From this baseline, the improvements made in CQI and test coverage are calculated periodically. For every period, using the improvements, a “goodput” factor is calculated. The factor is calculated as ((CQI improvement + T est Coverage Improvement)/20). Both improvement factors are scaled on a scale of 1 to 10, with 10 being the highest improvement. Thus, the goodput factor ranges from 0 (on improvement) to 1 (highest improvement). The number of noncommenting lines of code produced by the team in a period (”throughput”) is weighed by the goodput factor, effectively weighing down the throughput. This weighted throughput is called the ”goodput” since it is the amount of code exhibiting good code quality and good unit testing. Next, using the cost

incurred on the team, the cost of development index (CDI) is calculated as the ratio of cost to the goodput. Lower the index, better is the efficiency. We have observed in different pilots that, indeed projects having controlled costs, have lower efficiency as well, thus proving to the project managers that both costs and development efficiency needs to balanced. 4) Code churn (coding activity): Code churn trends as measured from the commit activities in the source control repository are important to track as code churn is a good defect predictor. A typical healthy code churn pattern is a slow churn in the beginning of the build stage, ramping up during the middle, and gradually ramping down towards the end of the stage. So, if the code churn is high close to the build stage period end, it shows that the project is not ready to move into functional testing and might end up with schedule and cost slippages with higher defects. PIVoT measures and reports the following code churn insight reports - code churn, per package code churn, top code contributors and per package code contributors. Using these, one can get early warnings and quickly identify where the churn is happening and by whom, and can then take steps to mitigate schedule and cost slippage by re-prioritizing testing. 5) Team Analysis: PIVoT also analyzes the project data from a “team” perspective. In our field studies [6], we found that even in small teams of about 30 developers, project managers are not fully aware of the team structure and nature of contributions from developers - parts of the code contributed, amount contributed, type of contribution (fresh code, modification to old code etc.). It has been shown in various other studies [7], [8], [9] as well that the team organization and structure has important bearing on the project metrics and outcomes. PIVoT helps to bridge the gap between a manager’s intuitive understanding of the team and the actual on-ground reality. Project managers can choose to restructure teams or reallocate tasks for better delivery. PIVoT social network analysis to project data and provides insights in following categories:

a) Resource connectedness: PIVoT identifies highly connected resources, poorly connected resources and disconnected resources. Highly connected resources represent subset of developers collaborating extensively while disconnected resources represent developers working in isolation. Disconnected resources are important to identify since they represent a “knowledge transition” hole as there is no other developer who has knowledge of the code written by these resources.

Figure 5.

Team Analysis: Programmer Fragmentation Insight

b) Programmer fragmentation: It is a well known fact that the quality of output suffers if an individual works on multiple disparate activities. PIVoT measures the contributions of programmers to different modules and represents the relative fragmentation in a pie-chart. An example output is shown in Figure 5. c) Code contribution: This are the same insights as reported in the code churn reports. PIVoT currently utilizes the source control repository to calculate these insights, so intuitively, team members who have worked on same set of packages in the coding phase will emerge as highly connected to each other. Note that thus team members who exclusively worked on a package will emerge as being disconnected. This may be planned or unplanned and so the results of these insights have to be interpreted in the light of the what was initially planned. In future versions, as PIVoT assimilates more data from other phases of software development, these insights will be based on a richer, cross-phase team data to present a more holistic picture of the team dynamics. In future versions, we also intend to add a team evolution perspective [10] which shows how the team evolved over time in terms of centrality of individuals, and roles being performed by individuals vis-a-vis the project plan. 6) Composite graphs: PIVoT provides composite graphs of key insights against code churn to indicate how the insights are performing as the coding activity proceeds. As shown in Figure 6 the following graphs are provided - Code churn and rate of increase of quality issues, Code churn and QCTE, Code churn and Modified CQI, Code churn and Modified CDI,

IV. R ELATED W ORK AND D ISCUSSION

Figure 6.

PIVoT Composite Graphs Report

Intersection between quality issues produced by code churn and top quality issues. Code churn is used as a common factor for the first four correlations. The code commits resulting the code churn are a culmination of a developer’s coding activities and hence provides the ”truth of development activities”. While it is intuitively clear that while code churn is desirable, (indeed expected as per the trend pattern discussed earlier,) the code churn needs to be accompanied by good coding and testing practices. This is the basis of the first four correlations, which measure effect of code churn on code quality, unit testing and development efficiency. The correlation between code churn to rate of increase of quality issues is discussed earlier. For the other correlations, a positive correlation indicates a healthy project, since if with the increasing code churn, the code quality (modified CQI), unit testing effectiveness (QCTE) and development efficiency (modified CDI) also increase, its a good sign! The last composite insight provides a drill down into the nature of code quality violations produced by code churn. Using these composite insights, the project manager gets a quick and thorough understanding of the project’s progress and its impact on key quality parameters. Once the composite insights are obtained, detailed explorations in to other insights and metrics to understand potential root causes can be made using rest of PIVoT insights.

Hackystat[3] tool is a platform for consolidating data from different project tools. Hackystat uses the concept of ’software telemetry’ where special instrumentation needs to be introduced in each tool that needs to be monitored. Analysis of this data shows different project insights. Sonar[4] is another prominent open source tool that uses Maven[11] (a build management tool) to execute a set of code quality and unit testing tools on an application’s code. Later, Sonar consolidates data from different tools’ outputs and performs analysis on this data for generating reports. Sonar offers a plug-in environment for executing custom tools and generating custom reports. Blue Optima[12] is a commercial tool that allows enterprises to analysis their application development portfolios. BlueOptima offers detailed analysis useful for benchmarking the development trends across different projects. While these tools are similar in nature with PIVoT, there are some key differences that make PIVoT better suited to deployment in typical IT services organizations. Unlike Hackystat, PIVoT does not need instrumentation of tools to gather data. PIVoT, instead runs the tools from which it needs data and consolidates the data using the project model described earlier. This approach allows PIVoT to be deployed seamlessly in a project with no changes required to the tools used in the project. While this approach is similar to the one used in Sonar, PIVoT does not rely on Maven. In case a project is not Maven-based, one needs to migrate a non-Maven project to Maven project type for using Sonar. With PIVoT, this is not required, making deploying PIVoT easier and faster. BlueOptima requires the enterprise to upload project data to BlueOptima servers where the analysis happens. Since IT services organization handle client code, this approach raises data confidentiality concerns and may limit BlueOptima’s usage in IT services organizations. The approach taken by PIVoT for data collection, data consolidation and persistence and data analysis makes it very suitable for use in IT services organizations. Another important difference between these tools and PIVoT is the underlying project model used in PIVoT. It encompasses tools, activities and users. Hence, while we are focusing metric based insights, it is quite easy to add specialized analysis like Micro Process Adherence[13] in to PIVoT. Other tools focus more on tool data and hence are not able to generate process related insights. Yet another key difference is in the type of insights shown in such tools. During the development of PIVoT, we interviewed practitioners to identify the most important insights project managers and team leads need. This ensures proper breadth of insights while remaining focused. Finally, PIVoT provides team analysis insights that other tools don’t provide out of the box. Using this well-defined breadth, project managers are able to obtain insights into three critical

facets of a project’s progress - quality, efficiency and team using PIVoT.

[2] IBM, “Ibm rational team concert,” www.ibm.com/software/rational/products/rtc/, 2011.

V. C ONCLUSION Typical software development project environments in services organizations consist of heterogeneous tools and activities due to client mandated tooling and infrastructure. This creates challenges in creating an holistic picture of a project’s health and its future trajectory. Project Insights and Visualization Toolkit (PIVoT), is a framework for automated, non-invasive, and in-process data collection and analysis of software development project environment, and helps teams understand their project’s health and drive continuous improvement programs. PIVoT is different from other project management tools in that it automatically captures in-process data from disparate project tools, consolidates and links it using a rich project model and provides in-process insights into code quality and churn, cost of development, component testing effectiveness and team dynamics and evolution. The reports provide a rich visual representation of various inprocess metrics in these dimensions. We have put a special emphasis on keeping PIVoT as non-invasive as possible. This is important to make sure that there is little change to the well-established SDLC processes in our organization and thus the impact of adopting PIVoT by any project is minimal (confined to a one time configuration). It also has the second benefit of guaranteeing objective insights without the subjectivity of human input. This makes the reports factual rather than based on any human evaluation of the project status - the latter being very common in today’s process management tools and metrics. Note that PIVoT comes with an obligation of interpreting it’s reports in the context of the project plan, phase and complexity. For example a disconnected team member may be so by plan, or a consistent high code churn is expected and well-accounted for in the project plan, and thus whether a project action is required or not vis-a-vis an alert is the responsibility of the project manager. However note that PIVoT provides this visibility in an in-process manner which otherwise would have been hidden or significantly difficult to arrive at otherwise! Thus we believe it is a valuable tool in the armory of the project and team managers. Through different pilots, we have found that PIVoT has facilitated predictable project delivery by giving early warnings on deteriorating component test effectiveness, poor code quality trends and low development efficiency. Research is underway in expanding the scope of PIVoT to diverse platforms as well as in adding more composite and team metrics [10] and forecasting abilities. With newer releases of PIVoT in the pipeline, we are confident that significantly more projects will benefit from its features in the near future.

[3] P. M. Johnson, H. Kou, J. M. Agustin, Q. Zhang, A. Kagawa, and T. Yamashita, “Practical automated process and product metric collection and analysis in a classroom setting: Lessons learned from hackystat-uh,” in Proceedings of the 2004 International Symposium on Empirical Software Engineering. Washington, DC, USA: IEEE Computer Society, 2004, pp. 136–144.

R EFERENCES [1] Microsoft, “Visual studio team system,” crosoft.com/india/vsts/index.aspx, 2011.

www.mi-

[4] SonarSource, “Sonar,” http://www.sonarsource.org/effectivecode-review-with-sonar/, 2011. [5] V. Kaulgud and V. S. Sharma, “Insights into component testing process,” in Proceeding of the 2nd international workshop on Emerging trends in software metrics, ser. WETSoM ’11. New York, NY, USA: ACM, 2011, pp. 27–30. [6] S. Datta, V. Kaulgud, V. S. Sharma, and N. Kumar, “A social network based study of software team dynamics,” in Proceedings of the 3rd India software engineering conference, ser. ISEC ’10. New York, NY, USA: ACM, 2010, pp. 33–42. [7] M. Cataldo, J. D. Herbsleb, and K. M. Carley, “Socio-technical congruence: a framework for assessing the impact of technical and work dependencies on software development productivity,” in Proceedings of the Second ACM-IEEE international symposium on Empirical software engineering and measurement. Kaiserslautern, Germany: ACM, 2008, pp. 2–11. [Online]. Available: http://portal.acm.org/citation.cfm?id=1414008 [8] C. Bird, N. Nagappan, P. Devanbu, H. Gall, and B. Murphy, “Does distributed development affect software quality?: an empirical case study of windows vista,” Commun. ACM, vol. 52, pp. 85–93, August 2009. [Online]. Available: http://doi.acm.org/10.1145/1536616.1536639 [9] N. Nagappan, B. Murphy, and V. Basili, “The influence of organizational structure on software quality: an empirical case study,” in Proceedings of the 30th international conference on Software engineering, ser. ICSE ’08. New York, NY, USA: ACM, 2008, pp. 521–530. [Online]. Available: http://doi.acm.org/10.1145/1368088.1368160 [10] V. S. Sharma and V. Kaulgud, “Studying team evolution during software testing,” in Proceeding of the 4th international workshop on Cooperative and human aspects of software engineering, ser. CHASE ’11. New York, NY, USA: ACM, 2011, pp. 72–75. [11] Maven, “Maven,” http://www.maven.org/, 2011. [12] BlueOptima, 2011.

“Blueoptima,”

http://www.blueoptima.com/,

[13] V. S. Sharma and V. Kaulgud, “Micro process adherence for delivering reliable software,” in Supplementary Proceedings of the 20th Annual IEEE International Symposium on Software Reliability Engineering, ser. ISSRE ’09, 2009.