Feb 25, 2016 - Best Paper Award at the. 3rd IEEE Working Conference on Software Visualization (VISSOFT 2015). Florian Fittkau .... Elastic application-level monitoring for large software landscapes in the cloud. In Proceedings of.
Hierarchical Software Landscape Visualization for System Comprehension: A Controlled Experiment Best Paper Award at the 3rd IEEE Working Conference on Software Visualization (VISSOFT 2015)
Florian Fittkau, Alexander Krause, and Wilhelm Hasselbring
Fittkau, Krause, and Hasselbring
Hierarchical Landscape Visualization
2016-02-25
1
Introduction Introduction
Usually flat visualizations in application performance management (APM) tools to comprehend large software landscapes For example: AppDynamics, Dynatrace
Could be ineffective for large software landscapes → Idea: Hierarchical landscape visualization
Controlled experiment to evaluate its effectiveness and efficiency → Own implementation of various landscape visualizations, for internal validity Fittkau, Krause, and Hasselbring
Hierarchical Landscape Visualization
2016-02-25
2
Flat Visualization in ExplorViz Visualization in ExplorViz
Fittkau, Krause, and Hasselbring
Hierarchical Landscape Visualization
2016-02-25
3
Hierarchical Visualization in ExplorViz Visualization in ExplorViz
Fittkau, Krause, and Hasselbring
Hierarchical Landscape Visualization
2016-02-25
4
Hypotheses Controlled Experiment
H1: Flat Group and Hierarchical Group require different times for completing typical system comprehension tasks. H2: The correctness of solutions to typical system comprehension tasks differs between Flat Group and Hierarchical Group. The null hypotheses H10 and H20 follow accordingly.
Fittkau, Krause, and Hasselbring
Hierarchical Landscape Visualization
2016-02-25
5
Experiment Design Controlled Experiment
Between-subjects design with random assignment Object landscape: Modeled technical IT infrastructure of the Kiel University landscape (140 applications) 29 students (M.Sc.) from the master course “Software Engineering for Parallel and Distributed Systems” 5 system comprehension tasks Pilot study
Fittkau, Krause, and Hasselbring
Hierarchical Landscape Visualization
2016-02-25
6
System Comprehension Tasks Controlled Experiment
ID
Description
T1
Context: Identification of Critical Dependencies Name three applications that have a high fan-in (at least two incoming communication lines).
T2
Context: Potential Bottleneck Detection Name the Top 3 communications with the highest request count in descending order. Write down the start application and the end application.
T3
Context: Scalability Evaluation Which applications are duplicated on multiple nodes? The answer should contain all 8 duplicated applications which are all named differently.
T4
Context: Service Analysis What is the purpose of the WWWPRINT application in your opinion? How does the process might work to achieve the functionality for the user?
T5
Context: Risk Management What are the consequences of a failure of the LDAP application? Name all affected applications and briefly describe their purposes. Hint: Remember the received paper about the introduction to the university landscape.
Fittkau, Krause, and Hasselbring
Hierarchical Landscape Visualization
2016-02-25
7
Experiment Operation Controlled Experiment
Automated, interactive tutorial
Online questionnaire Screen recording
Fittkau, Krause, and Hasselbring
Hierarchical Landscape Visualization
2016-02-25
8
Experiment Results Overall time spent and correctness Controlled Experiment
Fittkau, Krause, and Hasselbring
Hierarchical Landscape Visualization
2016-02-25
9
Descriptive Statistics of the Results Extract Controlled Experiment
Time Spent Flat Hierarchical mean difference p-value
Fittkau, Krause, and Hasselbring
23.49
23.45 -0.17 %
Correctness Flat Hierarchical 17.07
0.9802
Hierarchical Landscape Visualization
19.5 +14.24 % 0.02303
2016-02-25
10
Qualitative Feedback Controlled Experiment
The participants were asked to give suggestions to improve the visualization they used for solving the tasks. Flat Visualization Group: Some labels representing the request count overlap, such that they were forced to get the count by hovering over the communication line. (5 users) Tabular representation for some tasks could be useful (2 users) Hierarchical Visualization Group: Animations for opening and closing nodes were suggested (3 users) Highlighting nodes or connections was suggested (2 users) Feedback mainly related to the tooling. Fittkau, Krause, and Hasselbring
Hierarchical Landscape Visualization
2016-02-25
11
Threats to Validity Controlled Experiment
Internal Validity Subjects (competence, experience, motivation) Tasks (review, difficulty, bias) Tutorials Time constraints External Validity Single object landscape Task selection Students
Fittkau, Krause, and Hasselbring
Hierarchical Landscape Visualization
2016-02-25
12
Content of Evaluated Artifact Artifact
VISSOFT 2015 offered an artifact submission process for papers that were accepted for the main program. Our evaluated artifact is available at http://dx.doi.org/10.5281/zenodo.18853 Source code and binaries Input files Tutorial material 29 screen recordings Raw results R scripts
Fittkau, Krause, and Hasselbring
Hierarchical Landscape Visualization
2016-02-25
13
More at VISSOFT 2015 More at VISSOFT 2015
In addition to Fittkau et al. [2015d] which addresses the landscape , the following papers on the application perspective of perspective have been presented at VISSOFT 2015:
3D Print [Fittkau et al. 2015b] Virtual Reality [Fittkau et al. 2015c]
Fittkau, Krause, and Hasselbring
Hierarchical Landscape Visualization
2016-02-25
14
On-Screen 3D Visualization of EPrints [Fittkau et al. 2013] More at VISSOFT 2015
Fittkau, Krause, and Hasselbring
Hierarchical Landscape Visualization
2016-02-25
15
3D Print of EPrints [Zirkelbach et al. 2015a;b] More at VISSOFT 2015
Fittkau, Krause, and Hasselbring
Hierarchical Landscape Visualization
2016-02-25
16
3D Print of PMD [Fittkau et al. 2015b] More at VISSOFT 2015
Fittkau, Krause, and Hasselbring
Hierarchical Landscape Visualization
2016-02-25
17
Virtual Reality with ExplorViz [Fittkau et al. 2015c] More at VISSOFT 2015
Fittkau, Krause, and Hasselbring
Hierarchical Landscape Visualization
2016-02-25
18
Virtual Reality with ExplorViz [Fittkau et al. 2015c] More at VISSOFT 2015
Fittkau, Krause, and Hasselbring
Hierarchical Landscape Visualization
2016-02-25
18
Virtual Reality with ExplorViz [Fittkau et al. 2015c] More at VISSOFT 2015
Fittkau, Krause, and Hasselbring
Hierarchical Landscape Visualization
2016-02-25
18
Conclusions Conclusions
Hierarchical Visualization achieves significant improvement in correctness, not in time is open source (http://www.explorviz.net) and a replication package is provided (http://dx.doi.org/ 10.5281/zenodo.18853)
http://www.dynatrace.com/
Meanwhile, some APM vendors offer hierarchical landscape visualizations Perspectives on 3D print and virtual reality Fittkau, Krause, and Hasselbring
http://www.instana.com/
Hierarchical Landscape Visualization
2016-02-25
19
References References F. Fittkau and W. Hasselbring. Elastic application-level monitoring for large software landscapes in the cloud. In Proceedings of the 4th European Conference on Service-Oriented and Cloud Computing. Springer, Sept. 2015. F. Fittkau, J. Waller, C. Wulf, and W. Hasselbring. Live trace visualization for comprehending large software landscapes: The ExplorViz approach. In Proceedings of the 1st International Working Conference on Software Visualization (VISSOFT 2013), Sept. 2013. doi: 10.1109/VISSOFT.2013.6650536. F. Fittkau, P. Stelzer, and W. Hasselbring. Live visualization of large software landscapes for ensuring architecture conformance. In Proceedings of the 2014 European Conference on Software Architecture Workshops (ECSAW 2014). ACM, Aug. 2014a. ISBN 978-1-4503-2778-7. doi: 10.1145/2642803.2642831. F. Fittkau, A. van Hoorn, and W. Hasselbring. Towards a dependability control center for large software landscapes. In Proceedings of the 10th European Dependable Computing Conference (EDCC 2014). IEEE, May 2014b. F. Fittkau, S. Finke, W. Hasselbring, and J. Waller. Comparing trace visualizations for program comprehension through controlled experiments. In Proceedings of the 23rd IEEE International Conference on Program Comprehension (ICPC 2015), pages 266–276. IEEE, May 2015a. doi: 10.1109/ICPC.2015.37. F. Fittkau, E. Koppenhagen, and W. Hasselbring. Research Perspective on Supporting Software Engineering via Physical 3D Models. In Proceedings of the 3rd IEEE Working Conference on Software Visualization (VISSOFT 2015), pages 125–129. IEEE, Sept. 2015b. F. Fittkau, A. Krause, and W. Hasselbring. Exploring software cities in virtual reality. In Proceedings of the 3rd IEEE Working Conference on Software Visualization (VISSOFT 2015), pages 130–134. IEEE, Sept. 2015c. F. Fittkau, A. Krause, and W. Hasselbring. Hierarchical software landscape visualization for system comprehension: A controlled experiment. In Proceedings of the 3rd IEEE Working Conference on Software Visualization (VISSOFT 2015), pages 36–45. IEEE, Sept. 2015d. F. Fittkau, S. Roth, and W. Hasselbring. ExplorViz: Visual runtime behavior analysis of enterprise application landscapes. In Proceedings of the 23rd European Conference on Information Systems (ECIS 2015). AIS, 2015e. doi: 10.18151/7217313. C. Zirkelbach, W. Hasselbring, and L. Carr. Combining Kieker with Gephi for Performance Analysis and Interactive Trace Visualization. In Symposium on Software Performance 2015, 2015a. C. Zirkelbach, W. Hasselbring, F. Fittkau, and L. Carr. Performance Analysis of Legacy Perl Software via Batch and Interactive Trace Visualization. TR 1509, Kiel University, Aug. 2015b. URL http://eprints.uni-kiel.de/29430/. Fittkau, Krause, and Hasselbring
Hierarchical Landscape Visualization
2016-02-25
20
