GEM: Graphical Explorer of MPI Programs
∗
Alan Humphrey
Christopher Derrick
Ganesh Gopalakrishnan
University of Utah Salt Lake City, Utah 84112
University of Utah Salt Lake City, Utah 84112
University of Utah Salt Lake City, Utah 84112
[email protected]
[email protected] Beth R. Tibbitts
[email protected]
IBM Corporation Eclipse Parallel Tools Platform Lexington, KY 40511
[email protected] ABSTRACT
1.
Formal dynamic verification complements MPI program testing by detecting hard-to-find concurrency bugs. These bugs are elusive and difficult to reproduce and track down using conventional methods due to the difficulty in determining the state of a program when the bug occurs. In previous work, we described our dynamic verifier called In-situ Partial Order (ISP [5]) that can efficiently search the execution space of an MPI program while detecting important classes of bugs. ISP does this by executing processes under the control of its own scheduler. One major limitation of ISP, when used by itself, is the lack of a powerful and widely usable graphical front-end. We now present a new tool called Graphical Explorer of MPI Programs (GEM [2]) that overcomes this limitation. GEM is designed as an Eclipse plugin within the Parallel Tools Platform (PTP [1]), a rapidly evolving tool integration framework for parallel program development and analysis. GEM serves to bring ISP within reach of a wide array of programmers with its original release as part of PTP version 3.0 in December, 2009. Our poster illustrates the motivation for GEM, an overview of its features and our future plans.
The Message Passing Interface (MPI [3]) library continues to enjoy a dominant position in HPC, and has been ported to run on virtually every parallel machine available today. Given the extensive presence of MPI, it is imperative that highly effective, debugging tools with intuitive graphical front-ends be created for MPI programs. These debugging tools must strive to meet certain goals: (i) reliably localize errors and display these errors intuitively, (ii) display only relevant information about a particular execution, avoiding redundancy, (iii) depict items of interest so that users gather deeper insights about their programs and the APIs/libraries they use, (iv) help locate root causes of bugs found, (v) provide facilities to step through and visualize process executions. These goals are even more important to meet for concurrent program debuggers, as concurrent program executions are often less intuitive than sequential program executions. Today, there is an impressive array of tools available for debugging MPI programs, unfortunately, these tools only provide ad hoc techniques for process interleaving (schedule) generation, and as a result, many interleavings are not considered. In practice, these omitted interleavings often harbor elusive bugs. Considering all interleavings is not an option, as even a simple MPI program with five processes, each making five calls has potentially 10 billion interleavings (schedules). It can be easily seen though, that large groups of interleavings are totally equivalent. ISP has the ability to determine relevant interleavings (possible schedules with runtime behavior different from those already observed), and the ability to enforce interleavings. While ISP was a significant step forward in the arena of dynamic verification tools for MPI, its usage was hindered by the absence of a widely usable and intuitive graphical user interface. This poster illustrates our contribution in this regard through a new tool called Graphical Explorer of MPI Programs (GEM).
Categories and Subject Descriptors D.0 [Software]: General; I.6.4 [Simulation and Modeling]: Model Validation and Analysis
General Terms Design, Theory, Verification
Keywords Dynamic Verification, Graphical User Interfaces, Dynamic Interleaving Reduction, Message Passing, MPI, Multi-core, Eclipse Parallel Tools Platform.
INTRODUCTION
∗
Supported in part by Microsoft, NSF CNS-0509379, CCF0811429, CCF-0903408. Portions of this material are supported by or based upon work supported by the Defense Advanced Research Projects Agency (DARPA) under its Agreement No. HR0011-07-9-0002. Copyright is held by the author/owner(s). SOFTVIS’10, October 25–26, 2010, Salt Lake City, Utah, USA. ACM 978-1-4503-0028-5/10/10.
2.
GEM FEATURES
GEM makes ISP truly a push-button dynamic formal verification tool for MPI C/C++ codes with its tight integration within the Eclipse IDE (Figure 1). GEM offers support for both Eclipse C/C++ Development Tooling (CDT) Managed Build and Makefile projects.
The test machine used for the ParMETIS case study was an HP Pavilion laptop running Ubuntu 8.10, with 4GB RAM and an Intel Core2Duo T-9300 CPU running at 2.5 GHz. The following are results from our GEM verifications: • 2-procs: 1:58 min, 13,432 transitions, 1 interleaving • 5-procs: 3:56 min, 71,633 transitions, 1 interleaving • 10-procs: 9:36 min, 221,057 transitions, 1 interleaving
Figure 1: Eclipse Integration of GEM
In all cases we found that over 95% of the total verification time was spent by ISP and the remainder was spent by GEM loading its data structures and displaying results. Through this case study we were better able to determine an upper bound on the scalability of our tool. Using ISP, GEM was able to scale to tens of processes.
4. The Analyzer view is the primary Eclipse view available through GEM. This view gives a user-friendly debugger look and feel and has the ability to present verification results as if the matches happened according to program order as well as the ability to view instruction execution by internal execution order (the order in which ISP’s scheduler issues calls to the MPI runtime). Other facilities are provided by GEM that allow a developer to quickly localize the source of deadlocks, MPI resource leaks and local assertion violations (Figure 2). A developer can then visit the offending lines of code within the Eclipse editor with a single mouse click.
5.
Figure 2: GEM’s Resource Leak Browser With the integration of GEM into PTP, an MPI developer can subsequently write, debug, formally verify and launch their MPI code on a cluster, all from within the Eclipse IDE. PTP also offers a scalable, parallel debugger.
3.
CASE STUDY: VERIFYING PARMETIS
It was our goal not only to make GEM useful as an academic tool, but also that it be viable for larger scale codes. GEM proved highly capable with all of our smaller projects and most medium scale Makefile projects. This included a Life simulation verified with up to 20 processes in under three minutes. Of particular interest though, as to the real viability and scalability of the GEM/ISP combination, is the ParMETIS case study. ParMETIS 3.1 ( [4]) is a widely used parallel graph partitioning and sparse matrix ordering library. Verifying ParMETIS makes for an excellent study due to the inherent complexities involved with verifying a project of such size. Some ParMETIS routines have more than 12,000 lines of code between themselves and their helper functions, and involve an enormous number of MPI calls.
CONCLUSIONS AND FUTURE PLANS
The limitations of GEM and ISP have necessitated work on a decentralized scheduling algorithm and in recent work, we have developed a scalable analysis tool called Distributed Analyzer of MPI Programs (DAMPI [6]) that conducts analysis similar to ISP, but with many scalability oriented heuristics that permit analysis to scale up to thousands of processes while running on a cluster. As part of our ongoing work we are in the process of seamlessly integrating DAMPI within GEM ([7]) to provide a framework for initial verification using ISP followed by subsequent verification of scaled up models using DAMPI. A hybrid scheme is also under consideration where known omissions of DAMPI (due to its scalability heuristics) are later analyzed through subsequent runs of ISP.
REFERENCES
[1] The Eclipse Parallel Tools Platform. http://www.eclipse.org/ptp. [2] A. Humphrey, C. Derrick, G. Gopalakrishnan, and B. Tibbitts. GEM: Graphical Explorer of MPI Programs. In International Conference on Parallel Processing ICPP - PSTI Workshop, 2010. Accepted in PSTI10. [3] MPI 2.1 Standard. http://www.mpi-forum.org/docs/. [4] ParMETIS - Parallel Graph Partitioning and Fill-reducing Matrix Ordering. http://glaros.dtc. umn.edu/gkhome/metis/parmetis/overview. [5] S. Vakkalanka, G. Gopalakrishnan, and R. M. Kirby. Dynamic Verification of MPI Programs with Reductions in Presence of Split Operations and Relaxed Orderings. In Computer Aided Verification (CAV), pages 66–79, 2008. [6] A. Vo, S. Aananthakrishnan, G. Gopalakrishnan, B. R. de Supinski, M. Schulz, and G. Bronevetsky. A Scalable and Distributed Dynamic Formal Verifier for MPI Programs. In International Conference on High Performance Computing (SC), 2010. Accepted in SC10. http://www.cs.utah.edu/fv/DAMPI/sc10.pdf. [7] A. Vo, G. Gopalakrishnan, S. Vakkalanka, A. Humphrey, and C. Derrick. Seamless Integration of Two Approaches to Dynamic Formal Verification of MPI Programs, Programming Language Design and Implementation (PLDI) - AMP Workshop 2010.
