V ISUA LIZ ATION C ORNER Editors: Cláudio T. Silva,
[email protected] Joel E. Tohline,
[email protected]
AN EXPERIMENTAL DISTRIBUTED VISUALIZATION SYSTEM FOR PETASCALE COMPUTING By Jinghua Ge, Andrei Hutanu, Cornelius Toole, Robert Kooima, Imtiaz Hossain, and Gabrielle Allen The coming era of petascale computing and heterogeneous platforms calls for fundamental changes in perspective on how we design distributed visualization software.
I
nteractive, photorealistic visualization is an important tool in scientific research because it lets scientists make visual discoveries in real-time. Modern computational science is increasingly producing largescale data sets using high-performance computing (HPC) resources that are remote from the user, making interactive visualization and analysis of such data a challenging task. One option to visualize this data is to replicate it on local resources for local interactive analysis. However, local data replication is cumbersome and in many cases the processing power of desktop visualization systems is inadequate. Existing parallel visualization systems—such as ParaView1 and VisIt2—were designed to support large dataset visualization. However, existing systems typically impose long wait times during data loading and preprocessing, and require lengthy noninteractive processes for highquality visualization. When the data is located remotely and is tens of gigabytes and larger, there’s a pressing need for satisfactory interactivity and performance. In contrast to ex ist ing systems, our approach uses powerful remote resources connected by highspeed networks to directly visualize remote data without replication.
We also combine distributed resources in a single visualization application. Our distributed visualization system, eaviv, adopts cutting-edge technologies such as parallel rendering on graphics processing unit (GPU) clusters, progressive visualization, high-speed network I/O, and scalable video streaming. Our driving goal in designing the eaviv system is to keep pace with the increasing scale of simulations and data growth. The system supports distributed collaboration, letting multiple users in physically distributed locations interact with the same visualization to communicate their ideas and explore the datasets cooperatively. We’ve demonstrated and tested eaviv across the Louisiana Optical Network Initiative (LONI) and TeraGrid. Here, we describe eaviv’s design principles as well as its hardware and software components.
78
1521-9615/10/$26.00 © 2010 IEEE
Copublished by the IEEE CS and the AIP
&,6(9L]&RUQLQGG
Design Principles
Developed by a small group of researchers at the Louisiana State University (LSU) Center for Computation & Technology (CCT), the eaviv distributed visualization system is a prototype petascale data analysis tool that addresses the root problems of next-generation visualization systems.
We designed eaviv to support experimentation, and it’s constantly undergoing optimization. Our work is guided by several key design principles; in brief, eaviv must be • lightweight, so system optimization research can be continuously applied to implementation refinement; • user-oriented and developed in collaboration with specific users within computational science communities; • modular, so it’s easy to configure for different use scenarios; • progressive in adopting emerging infrastructure and technology in HPC, networking, GPU computing, and mobile devices; • scalable, such that its overall scalability results from attentiveness to each individual component’s scalability; and • asynchronous, emphasizing interactivity by providing a progressive visualization pipeline so that software components couple asynchronously without blocking the pipelined data flow. These design principles represent the key factors that distinguish the eaviv system architecture from other visualization software. They are used as guidelines throughout our software development.
COMPUTING IN SCIENCE & ENGINEERING
30
The eaviv System
We’re actively developing eaviv as an open source research project, and the US National Science Foundation’s Early-concept Grants for Exploratory Research (EAGER) program supports its experimental network testbed. We demonstrated eaviv at the 2008 International Conference for HPC, Networking, Storage and Analysis (SC08) and won first prize at the IEEE International Scalable Computing Challenge (SCALE2009). We presented eaviv at SC09 and at the Spring 2010 Internet2 Member Meeting. For these demonstrations, we used data from scientific user communities including chemical tomography,3 numerical relativity (Cactus; http:// cactuscode.org) and astrophysics (Enzo; http://lca.ucsd.edu/projects/enzo). Figure 1 shows the visualization pipeline’s data flow from • the supercomputer that runs simulation and stores the data to • the data servers that transfer the data to the renderer cluster, which streams images to the displays. Hardware components
The system’s hardware components— including HPC clusters, highresolution displays, and interaction devices—are distributed, specialized resources connected by high-speed networks. Network testbed. Our work addresses
fundamental issues in distributed visualization design and implementation, where network services represent a first-class resource. To support this, we built a testbed connecting resources at CCT, LSU, the National Center for Supercomputing Applications (NCSA) at the University of Illinois at Urbana-Champaign (UIUC), and the
SEPTEMBER/OCTOBER 2010
&,6(9L]&RUQLQGG
Data server
Input device Tile display Rendering
Data server
Figure 1. The eaviv visualization pipeline connects data servers, GPU clusters, and displays. The data servers and rendering GPU clusters are physically distributed resources remote to the local user. They’re connected to the local displays by highspeed networks. Large-scale datasets are thus processed by remote resources and the resulting image streamed to local users.
Laboratory of Advanced Networking Technologies (SITOLA) at Masaryk University in the Czech Republic. The Internet2 Interoperable Ondemand Network (ION) provides wide-area connectivity to support dynamic network circuit services. Applications request and reserve point-topoint circuits between sites as needed using automated control software. Computation and display. The primary
resources connected by our network testbed include contributions from CCT, LONI, NCSA, TeraGrid, and SITOLA. • GPU and rendering. Spider, LSU’s distributed computing testbed, consists of eight rendering nodes and four I/O nodes. Rendering nodes are connected pair-wise to four Nvidia Tesla S1070 graphics units. The NCSA’s accelerator cluster (http:// iacat.uiuc.edu/resources/cluster) is a 40-node cluster that combines both GPU and field-programmable gate array technology. NCSA’s Lincoln TeraGrid cluster (www.ncsa. illinois.edu/UserInfo/Resources/ Hardware/Intel64TeslaCluster)
has 192 nodes with 96 Nvidia Tesla S1070 graphics units. • Storage. SITOLA provides a Sun X4540 server with eight AMD Opteron 2356 cores, 64 GBytes memory, and 48 Tbytes storage. Running Linux with disks configured in RAID-0, it allows up to 1.7 GBytes per second of sustained reading and 1 Gbps of sustained writing in real time. NCSA’s mass storage system (MSS; www.ncsa.illinois.edu/UserInfo/ Data/MSS) is a production system for permanent data storage and consists of a high-performance system running NCSA’s UberFS. MSS is a parallel, automatically backed-up storage system accessible via SSH, FTP, and GridFTP. • Display. LSU has a Sony SXRD 4K projector (4,096 × 2,160 pixels), and SITOLA has two tiled displays, one 6 × 4 with total resolution of 55 Megapixels and six display nodes, and the other 2 × 2 with 9 Megapixels. Software Components
Our system’s main software components are the distributed data server and parallel renderer. The eaviv system also supports specialized tangible remote
79
30
VISUALIZ ATION CORNER
Figure 2. Four rendering nodes are used to process the data in parallel. Each node renders one quarter of the data into a 256 × 256 local image and contributes partially to the final composited image following an overlap-based compositing schedule. The global assembled view has a resolution of 512 × 512. Using this method, Pcaster can produce high-resolution images scalably. Data Courtesy of Britton Smith, University of Colorado.
interaction and uses Scalable Adaptive Graphics Environment (SAGE)5 highperformance parallel image streaming software. Data server. Data servers can be ei-
ther local or remote. We use the local data server for simple scenarios when data is readily accessible and local disk speed is sufficient or when transport performance isn’t critical. We use remote data servers in two scenarios: when the data isn’t available locally, or when local I/O performance is insufficient. In this second scenario, network throughput can exceed that of the local disk system, and we can increase data transfer speed by distributing the data on the network and loading it in the remote machines’ main memory. High-performance data transmission over wide-area networks is difficult to achieve. One of the main factors affecting data transport performance is the network transport protocol; an unsuitable protocol on a wide-area network can have a poor performance. In some situations, the standard Internet TCP protocol might sustain only a few Mbps throughput on a 10 Gbps dedicated network connection. We’ve designed our system to support highspeed transport protocols, including Uniform Data Transfer,4 allowing us to achieve high network throughput
80
&,6(9L]&RUQLQGG
on long-distance and high-capacity network links. In addition, eaviv supports parallelism: each renderer node can access data from multiple data servers. This increases data throughput, as well as increasing the amount of data that the system can access. Parallel volume renderer. Our experimental data sets are large-scale, time-varying 3D volumes. This is a common data model in the computational science community and— because 3D data size grows more quickly than 2D data as a problem’s scale increases—the format provides an interesting challenge for our system. We implemented the ray-casting parallel volume renderer called Pcaster as the rendering component to demonstrate the distributed pipeline’s workflow. Compared to parallel volume renderers in existing software—such as Avizo (www.vsg3d.com), EnSight (www. ensight.com),VisIt or ParaView—Pcaster is a purely GPU-based volume renderer and image compositor supporting high-resolution rendering. Pcaster asynchronously couples with parallel data servers for network-streamed data input. We’ve tested Pcaster with data sets up to 64 Gbytes per timestep and achieved interactive frame rates (5 to 10 frames per second) using 32 Nvidia
Tesla S1070 GPUs on NCSA’s Lincoln cluster to produce render images of 1,024 × 1,024 resolution. Figure 2 shows an image with contributions from four renderers. Each node renders one quarter of the volume data set and contributes a 256 × 256 local composited image. The global assembled view has a resolution of 512 × 512. Using this method, Pcaster can produce high-resolution image scalable with the parallel rendering. For example, when we split a 40,963-voxel volume into 10,243 voxel subvolumes and render it using 64 rendering nodes, each renderer produces a 1,024 × 1,024 image, and the final composition has a resolution of 4,096 × 4,096. For multi-Gbyte datasets, data loading can take a long time. Rather than blocking on I/O, with the renderer waiting for data to load, Pcaster uses a pool-and-show mechanism. That is, rendering proceeds at interactive rates while data is continuously streamed into the renderer’s data pool. This produces a progressive visualization output for the user. Figure 3 shows an example of an intermediate rendering that demonstrates the progressive data pooling: 32 nodes are rendering the partial data in the pool. Pcaster’s progressive rendering mechanism not only gives users more prompt interaction with the data, but also provides visual performance metrics, including data preparation time and network transfer speed. Users can easily see any system bottlenecks and adapt their strategy as needed. Image streaming and remote tangible interaction. To support ultra-high-
resolution distributed visualization, eaviv integrates the SAGE image streaming software for high-performance parallel image delivery. Developed at the University of Illinois Chicago’s
COMPUTING IN SCIENCE & ENGINEERING
30
Electronic Visualization Laboratory, SAGE can stream remote application imagery of potentially hundreds of megapixels of contiguous display resolution to user’s local display. Also, SAGE bridges can be used to multicast the video stream from the application to multiple users, thus supporting collaborative scientific visualization environments. SAGE also supports the standard Internet UDP protocol; although UDP doesn’t provide data transmission reliability, it’s more suitable than TCP for high-performance video streaming. To encourage users to embrace the collaboration environment, eaviv provides a tangible interaction mechanism for more natural engagement with the visualization. In the tangible interaction approach, physical objects control and represent computational media.6 Figure 4 shows the tagged physical tokens (in this case, RFID-tagged cards) that we used in eaviv experiments. These tangible user interfaces (TUIs) are bound directly to visualization parameters within Pcaster, thus steering the interaction with the remote rendering. These devices provide immediate tactile feedback, which is useful for fine-grained control of remote objects during network latency. Using SAGE and tangible interaction devices enables collaborative visualization, letting multiple users at different locations simultaneously interact and collaborate via the visualization system.
Use Cases
When run with a local data server and a single renderer, eaviv can be used as a desktop tool for small-scale local data. When run with remote data servers, dedicated networks, and parallel rendering, it can handle large-scale distributed data. The eaviv system’s configurable architecture thus permits
SEPTEMBER/OCTOBER 2010
&,6(9L]&RUQLQGG
a variety of usage scenarios; we now describe two current examples. Classroom Visualizations
At the LSU Visualization Services Center, we’re in the process of applying eaviv to a computer lab and classroom environment. One target application will be a human anatomy class that will visualize data from the Visible Human Project (www.nlm. nih.gov/research/visible/visible_human. html). The Visible Human Project was established in 1989 to build a digital image library of volumetric data representing complete normal adult male and female anatomy (completed in 1994 and 1995, respectively, and rescanned in 2000 at a higher resolution). The project has generated two datasets of 65 and 40 Gbytes, respectively. For certain body sections, computed axial tomography scans, or CT scans, and magnetic resonance imaging (MRI) data are also available. Although dataset sections have been in use on individual lab computers, interactive full-scale visualization of complete datasets requires parallel rendering. The study of anatomy is highly visual. Learning where individual regions lie with respect to others and where they fit in the overall structure helps greatly in the learning process. In the classroom, both instructor and students will have control over separate instances of the visualization application. The instructor will use a large projection wall for teaching, and students will each have a desktop computer where they can zoom in to the dataset for in-depth exploration. LSU’s Spider will run eaviv and send the visualization image to each desktop computer in the lab classroom. This classroom use case is guiding our future development of eaviv’s user interaction modes. In addition, in the
Figure 3. Progressive parallel rendering during the receipt of data; 32 nodes are concurrently rendering the data partially available in the data pool. The continuous update of intermediate rendering images clearly demonstrates the rate at which data are being transferred through the network and offers a peek inside the dataset. Data Courtesy of Erik Schnetter, Louisiana State University (LSU) Center for Computation & Technology.
Figure 4. Parameter interaction tray. These tangible user interfaces (TUIs) are bound directly to visualization parameters within Pcaster, thus steering the interaction with the remote rendering. Instead of a single user operating with a keyboard and mouse, collaborating users can use multiple tangible devices, providing better interaction and cooperation.
collaboration mode, in which instructors and students interact with the same visualization, we’ll develop an independent steering mode that will let each student interact with the data individually. Distributed Execution on Production Resources
To demonstrate eaviv’s full distributed capability, we consider a complex black-hole data visualization scenario
81
30
VISUALIZ ATION CORNER
in which people in their individual offices at LSU use the following resources to visualize the remote data: • a SITOLA data server, which has datasets of hundreds of Gbytes or Tbytes; and • NCSA’s Lincoln, the most powerful rendering cluster available to our project.
Acknowledgments The eaviv project is funded by the US National Science Foundation awards Louisiana RIICyberTools #0701491, Eager network testbed #0947825, and Viz Tangibles #0521559. The initial eaviv research is funded by the Center for Computation & Technology at LSU.
References
Andrei Hutanu is an IT consultant at the Center for Computation & Technology, Louisiana State University. His research interests include software system design and engineering, distributed computing, and high-speed network applications with an emphasis on distributed visualization, data management, and collaborative applications. Hutanu has a PhD in computer science from Louisiana State University. Contact him at
[email protected].
1. A. Cedilnik et al., “Remote Large Data
The common practice of production HPC resource usage involves submitting a job into a scheduling queue and waiting for it to be executed. The eaviv system requires separate resource allocation of at least three components: the networks, the data servers, and the GPU rendering cluster. The three allocations must coincide in time for the application to execute. The only solution that can enable this execution is advanced resource reservation to allocate all resources simultaneously. In our testbed, the Lincoln GPU cluster supports advance reservation; the network circuit can be reserved through the LONI ION website (https:// ldcn1.sys.loni.org:8443/oscars). Experimental resources at SITOLA and CCT aren’t managed by a scheduler. As this scenario shows, running a system that enables scientific investigation of large datasets requires collaboration among multiple resource providers.
T
he eav iv system’s h ighperformance pipeline architecture supports cutting-edge emerging infrastructure and serves as a data analysis prototype for the nextgeneration of scientific investigation. It’s our hope that our work will benefit the visualization tools developer and the research community, both in using our software components and following our software design principles to improve existing tools.
82
&,6(9L]&RUQLQGG
Cornelius Toole is PhD candidate in the Department of Computer Science at Louisiana State University. His research interests include human computer interaction, scientific visualization, and distributed computing. Toole has an MS in computer science from Jackson State University. Contact him at
[email protected].
Visualization in the ParaView Framework,” Proc. Eurographics Parallel Graphics and Visualization, 2006, ACM Press, 2006, pp. 163–170. H. Childs et al., “A Contract-Based System For Large Data Visualization,” Proc. 16th IEEE Visualization, IEEE CS Press, 2005, pp. 25–33. H. Kyungmin, H.A. Harriett, and L.C. Butler, “Burning Issues in Tomography Analysis,” Computing in Science & Eng., vol. 10, no. 2, 2008, pp. 78–81. G. Yunhong and R.L. Grossman, “UDT: Udp-Based Data Transfer for High-Speed Wide Area Networks,” Computing Networks, vol. 51, no. 7, 2007, pp. 1777–1799. L. Renambot et al., “Enabling High Resolution Collaborative Visualization in Display Rich Virtual Organizations,” Future Generation Computing Systems, vol. 25, no. 2, 2009, pp. 161–168. H. Ishii and B. Ullmer, “Tangible Bits: Towards Seamless Interfaces Between People, Bits and Atoms,” Proc. SIGCHI Conf. Human Factors in Computing Systems, ACM Press, 1997, pp. 234–241.
Robert Kooima is a post-doctoral researcher at the Center for Computation & Technology and an adjunct professor in the Department of Computer Science at Louisiana State University. His research interests include real-time 3D computer graphics and parallel rendering for scientific visualization. Kooima has a PhD in computer science from University of Illinois at Chicago. Contact him at
[email protected].
Jinghua Ge is a visualization consultant at the Center for Computation & Technology, Louisiana State University. Her research interests include computer graphics, scientific visualization, parallel rendering with GPU clusters, and distributed visualization pipeline. Ge has a PhD in computer science from the University of Illinois at Chicago. Contact her at
[email protected].
Gabrielle Allen is an associate professor of computer science at Louisiana State University. Her research interests include computational frameworks, high-performance computing, high-speed networks, and computational astrophysics and coastal modeling. Allen has a PhD in computational astrophysics from Cardiff University. Contact her at gallen@cct. lsu.edu.
2.
3.
4.
5.
6.
Imtiaz Hossain is the manager of the Visualization Services Center of Information Technology Services at Louisiana State University. His research interests include image processing, pattern recognition, and data visualization. Hossain has an MS in mathematics from Louisiana State University. He is a member of the ACM, IEEE, and SPIE. Contact him at
[email protected].
COMPUTING IN SCIENCE & ENGINEERING
30
