2009 IEEE International Advance Computing Conference (IACC 2009) Patiala, India, 6–7 March 2009
MPI and PVM based HPC Setup for Multi Scale Modeling Satyadhar Joshi 1, Rohit Pathak 2, S. Ahmed1, K.K.Choudhary1, D.K. Mishra2 1
Shri Vaishnav Institute of Technology & Science, Indore 2 Acropolis Institute of Technology & Research, Indore
[email protected],
[email protected],
[email protected],
[email protected] Abstract- We have implemented multifarious aspects of multi scale modeling on various HPC (High Performance Computing) setups. Distribution of jobs from macro to nano scale has been shown. This distribution is substantiated on MPI (Message Passing Interface) and PVM (Parallel Virtual Machine) on MATLAB, Linux and WCCS (Windows Compute Cluster Server) environments. In this paper we have shown the connections and a novel way of implementing multi scale computations on an HPC setup. We have also compared MPI and PVM based HPC setup for MATLAB, Linux and WCCS environments. The selection criteria for identification and proposition of the tool, protocol and environment for an HPC setup has been corroborated. Comparison of the advantages and disadvantages of each of the methodologies being put forward. Thus depending on the need the correct choice can be made. MPI.NET was used under WCCS where C# was used. The latest versions were used for PVM Linux based setup where Open SUSE Linux was used as the operating system. The main two criteria user friendly and performance were compared and the recommendations are made for making the right balance between them.
I.
INTRODUCTION
Importance of HPC and multi scale modeling are eminent in current era. Various models for HPC have been proposed but very less has been talked about different models available for an HPC based setup for multi scale modeling. Because each model of an HPC setup has its advantages and disadvantages, so we need to take into consideration many aspects before deciding our requirements. Multi scale modeling poses some challenges which can be met by choosing the appropriate environment for HPC. Also the cost of the HPC setups may vary so it is important to select things as per our needs. Combination of continuum mechanics with quantum simulations is shown with the essence of multi scale modeling and simulation of Nanosystems [1]. Thus a basic foundation in this regard has been laid. As shown in fig.1 we have distributed our computations in various computational domains as per the HPC setup. In multi scale approach continuum mechanics to quantum mechanics linking and accurate simulation of properties Multi scale modeling for single electron device has been studied but the HPC setup complexities involved is an area to work in this regard before it can be usefully implemented [4]. Multi scale modeling can also be used for bio complexity as discussed [6]. Thus we can see the effect of developments in multi scale modeling can benefit major areas across many domains. There is a necessity of large scale computer simulation tools and numerical algorithms for the structural design and modeling of Nanorobots which required a HPC setup. A
methodology making use of multi-scalar and multi-physics modeling combined with virtual reality has been presented earlier for nanorobotic prototyping systems and the demonstration of integration of physics at various length scales and time scales has been brought to fruition. Furthur a reduction of 10-6 seconds in time for computation with respect to a very short time scale of 10-12 seconds using multiscale modeling using molecular dynamics and continuum mechanics approaches was achieved [21]. Thus this remains one of the major areas where HPC will benefit a lot in making things fast and reliable. We can see that applications in the areas of novel polymer composites for ultrahigh density capacitors for pulsed power applications and ballistic electron transport innovative molecule-onsemiconductor configuration exhibiting negative differential resistance have been presented in [22] which again shows the need for multi scale modeling. Nano-imprint technology is one of the most promising methodologies for the manufacture of nanometer-size formation. A multi-scale analysis arrangement has been proposed for the simulation of nano imprint technology and some useful results have been presented by Kim et al. and the estimation of viscoelasticity on molecular weight of polymer stamps has been carried out which can be implemented if the complexities and confusions area addressed in HPC which we have tried to address [23]. Also the usage of analysis scheme and the obtained results for the determination of suitable materials for nano imprinting stamps and process parameters such as pressure, time, and geometric ratios etc. of nano patterns has been suggested. Better thermoelectric materials designing necessitates the use of cutting-edge techniques and simulation tools as proposed [5]. Parallel Approach to the Nano thermal Numerical Analysis has been discussed in [3] where the effect of single electronics in the device thermal composition has been computed but this kind of computation can only be solved by optimizing the computations on an HPC setup. The implicated complexities levy quite a challenge on physics based multiscale modeling, far surpassing the capabilities of existing tools in this field and in the absence of awareness about the complexities and options that are there for implementing multi scale modeling on HPC these work cannot get to some reasonable result and application. Distribution of various level of computations has been demonstrated in fig.1. Thus we have presented some solutions which will help in increasing the rapidly commercialization of various devices modeled by multi scale modeling and take the research to a yield point. Each
3406
section contains the formulated codes that have been executed and the corresponding results have been shown below.
Microsoft Windows Cluster Server 2003 Environment Microsoft Visual Studio 2005 Integrated Development Environment
C# Programming Language .NET Framework Extreme Optimization Numerical Library
MPI.NET Library Figure.1 Distribution of various computations needed to be performed in an HPC setup
II.
PROPOSED ARCHITECTURE FOR HPC SETUPS
A. MPI in WCCS environment Multi scale modeling using MPI interface has been discussed earlier in [5].We need to communicate between different nodes to perform a distributed computation. MPI is an excellent library for message passing between different nodes. MPI provides us with high performance and reliability. The libraries are to work on cluster, so MPI is used to design them accordingly. Equally distributed tasks are distributed among the nodes for efficient computation, may it be reliability analysis or simulation of a device. Importance of a hybrid molecular- dynamics / finite-element (FE) methods to optimize code to perform multi scale computation using MPI has been shown [2]. Given below in Fig. 2. is a diagram illustrating the run time environment in our HPC setup in WCCS. Microsoft Windows Cluster Server 2003 Microsoft .NET Common Language Runtime Microsoft Compute Cluster
.NET Framework
Microsoft MPI
MPI.NET
Fig. 2. Proposed Runtime Environment
The development environment has been illustrated in Fig. 3. For using features of Microsoft MPI, MPI.NET API (Application Programming Interface) in C# is used. MPI.NET is an open source, high-performance, easy-to-use implementation of Message Passing Interface (MPI) for Microsoft's .NET environment [17]. Most MPI implementations provide support for writing MPI programs in C, C++, and FORTRAN [18]. MPI.NET provides support for all of the .NET languages (especially C#), and includes significant extensions (such as automatic serialization of objects) that make it far easier for us to build parallel programs that run on clusters [19].
Fig. 3. Our Proposed Development Environment
Given below is the very general C# program code for distribution of computations of nanotechnology. Implementation of MPI.NET on WCCS /*Including the namespace of the MPI.NET library*/ using MPI; namespace hpctest{ class Program { static void Main(string[] args) { using (new MPI.Environment(ref args)) { Communicator comm = Communicator.world; Console.WriteLine("Check from process " + comm.Rank); switch (comm.Rank) { case 0: /*PROCESS ONE CODE*/ /*Quantum Mechanical Computations*/ Console.WriteLine("Process " + comm.Rank + ": Quantum Mechanical Computations"); /*End of PROCESS ONE CODE*/ break; case 1: /*PROCESS TWO CODE*/ /*Semi-Classical/Molecular Computations*/ Console.WriteLine("Process " + comm.Rank + ": Semi-Classical/Molecular Computations"); /*End of PROCESS TWO CODE*/ break; case 2: /*PROCESS THREE CODE*/ /*Monte Carlo Computations*/ Console.WriteLine("Process " + comm.Rank + ": Monte Carlo Computations"); /*End of PROCESS THREE CODE*/ break; case 3: /*PROCESS FOUR CODE*/ /*Multi Scale Computations*/ Console.WriteLine("Process " + comm.Rank + ": Multi Scale Computations"); /*End of PROCESS FOUR CODE*/ break; default: /*Any other process*/ /*Code to be executed by rest of the processes*/ /*End of code*/ break;
2009 IEEE International Advance Computing Conference (IACC 2009)
3407
}/*End switch*/ /*All processes join here*/ comm.Barrier(); /*All processes completed*/ if (comm.Rank == 0) { Console.WriteLine("All processes finished"); } } } }} Result:
Fig. 4. Implementation of Multi Scale Modeling on WCCS setup
The implemented program.1 has been shown in fig.4. WCCS allows easy management of clusters which is an important feature as discussed in [18]. With WCCS we can divide and distribute the work load between different nodes in a desired manner [12, 13]. The calculations are broken into small jobs [19]. Many small jobs are created that perform a specific part of the whole calculation. Then these jobs are distributed among various cluster nodes. The jobs are scheduled to be executed on nodes. After the whole execution process the final result is compiled from the output of all the nodes. Extreme optimization Library can be used for more complex computations [16]. B. Parallel virtual machine (PVM) on Open Source MPI and PVM have been extensively used in HPC setups for communication from a long time [7, 8], and thus making PVM a matured framework to work [17]. In this section we have implemented PVM on a Linux based environment and the results are given as the output. Our program was coded in C and compiled using GNU gcc compiler, as shown in Fig. 5. When this program was executed on the virtual machine, the master process spawned four processes for the four types of computations, as shown in Fig. 6 and Fig. 7. The virtual machine configuration was done using pvm console configuration utility as shown in Fig. 6. We have shown the multi scale modeling as given in the code below. List of files that have been generated are masterSimulation.c (main file), monteCarlo.c, quantumMechanical.c, ,semiClassicalAndMolecular.c and multiScaleAndOther.c. Implementation of PVM on Open Source (Coding of masterSimulation.c) #include #include int main(){ int id, ps1, ps2, ps3, ps4; printf("Master Process ID: %x - Status: running\n", pvm_mytid());
3408
ps1 = pvm_spawn("monteCarlo", (char**)0, 0, "", 1, &id); ps2 = pvm_spawn("multiScaleAndOther", (char**)0, 0, "", 1, &id); ps3 = pvm_spawn("quantumMechanical", (char**)0, 0, "", 1, &id); ps4 = pvm_spawn("semiClassicalAndMolecular", (char**)0, 0, "", 1, &id); if(ps1==1) printf("Process one initiated. ID:t%x\n", id); else printf("cant start process one"); if(ps2==1) printf("Process one initiated. ID:t%x\n", id); else printf("cant start process two"); if(ps3==1) printf("Process one initiated. ID:t%x\n", id); else printf("cant start process three"); if(ps4==1) printf("Process one initiated. ID:t%x\n", id); else printf("cant start process four"); pvm_exit(); return 0; } Program 3 ( Coding of quantumMechanical.c) #include "pvm/pvm3.h" #include int main() { int ptid; ptid = pvm_parent(); printf("Quantum Mechanical Simulation computation process active. ID:%d, PID:%d\n", pvm_mytid(), ptid); pvm_exit(); return 0; } Result 2 and 3 are as follows:
Fig. 5. Compiling all the files of the program
2009 IEEE International Advance Computing Conference (IACC 2009)
Fig. 6. Spawning the master process through PVM
if(useHPC) disp('Using HPC Compution power.'); job = createJob(jm); set (job,'FileDependencies', {'rs_mems_main.m'}); createTask(job, @R_F, 1,{R}); createTask(job, @R_f, 1,{R}); createTask(job, @R_z, 1,{R}); submit(job); waitForState(job, 'finished', rs_HPCJobWaitTime); ans = getAllOutputArguments(job); The output of the code in program 4 has been illustrated in fig.9.
Fig. 7. Executing the masterSimulation Fig. 9. Result of program.4
III.
DISCUSSION AND COMPARISON
ABOUT THE VARIOUS SETUPS USED IN OUR WORK
Our prime motive is to characterize the current options available for HPC based setups for multi scale simulations in the field of Nanotechnology. The approach in this direction has to be carried out in view of the requirements of the users. Table 1 gives us the HPC libraries used and the respective OS platforms and math libraries being employed. TABLE 1 CONFIGURATION OF VARIOUS HPC SETUPS HPC Operating Math Comm. Setup System Library Library Fig. 8. Configuring the virtual machine
C. MATLAB based HPC setup Parallel MATLAB has been discussed before where the structure is commented in [10] which has been used in some areas for HPC. Application of parallel MATLAB has been in substantiated in many areas. Parallel MATLAB has been used in Digital Signal Processing (DSP) as shown by Nadya T. to solve many problems [15]. How to efficiently use Parallel MATLAB also shown by Ron Choy [16] helps us in optimizing MATLAB for HPC in reliability calculations. Thus we can see that MATLAB provided an excellent platform to work with Distributive computing and HPC. We have calculated the following using MATLAB HPC Distributed computing toolbox. Program 4 Implementation on MATLAB R = input ('Enter the function R(t) taking t as variable : ', 's');
MPI
PVM MATLAB
Microsoft Windows Server 2003 Compute Cluster Edition 64bit Open SUSE-11.0x86_64 64bit Windows Vista
Extreme Optimization Numerical Library for .NET C, parallel math library MATLAB Inbuilt Functions
MPI.NET
Pvm3.4.5.x86_64 Distributive computing toolbox
We conducted many tests and surveys for these different setups, for performance, reliability, efficiency, scalability, ease of use, hardware and resource requirement are as follows: A. Setup Complexities Our first test was to compare the complexities involved in configuring these setups. Setting up a MATLAB HPC
2009 IEEE International Advance Computing Conference (IACC 2009)
3409
environment was the easiest. MATLAB can use its own Mathworks job manager and third party manager such as TORQUE, MCCS, PBS, LSF etc. With Mathworks Job Manager an eight node cluster can be setup within no time. MCCS provided a handsome set of tools for configuring and tweaking the cluster, such as job scheduler and manager. PVM setup with the least set of tools and modules was found to be most non-entertaining, having an annoying console interface.
more work load compared to low end machines. PVM offered the highest level of control over execution and programming. As the language used for programming was C which is a low level language compared to C# and MATLAB, it offered the highest amount of control over programming. PVM offers high amount of control over execution. We can decide for a particular process to execute on a particular node, so that the right work can be given to right machine.
B. System Requirements We analyzed the system requirements of different setups. PVM setup had minimal system requirements, means you can create a cluster with low end machines. On the other hand MCCS had high system requirements, such as need of 64bit CPU, 4GB Ram, lots of Gig’s of hard disk, etc. It was found that PVM HPC cluster can be setup using any available machines capable of communicating with other nodes, although low end machines will bring a performance hit, but to get PVM running any machine will be sufficient. MATLAB has no specific OS or hardware requirements. It can be used on both 32-bit and 64-bit machines, minimum memory of 512MB, but for a decent performance, one needs a high end machine.
F. Reliability and Node Failure Mechanism MCCS comes with administrator tools which can be used to remove failed nodes easily and it has a strong built-in failover mechanism. PVM on other hand provides the least amount of facilities relating to such issues.
C. Efficiency and Resource Requirement Resource requirement and efficiency analysis included many phases. We performed various resource tests such as memory test, CPU usage test and network usage test. For doing a job with same speed, MATLAB was found requiring high memory and CPU resources compared to others, while PVM requiring the least. MCCS was found to deliver optimal performance to resource ratio. D. Performance Analysis The main topic of concern was performance delivered. PVM was found to provide the highest performance among the three, with MATLAB being a HLL because of its abstraction layers performing the slowest. MCCS was found to deliver a balanced performance and ease of use ratio. Our test of MCCS was done using C# programming language. Programs written in C# execute in a virtual machine of the .NET Framework due to which they lack some amount of performance over applications written in C or C++. Programs written in Visual C++ for MCCS environment can bring a significant gain in performance over those of C#. E. Access to low level Customizations This test was to see which one of the three provided more control over the programming and execution. MATLAB provided lots of tools and modules which help in doing the desired task but due to its abstraction we are left with little control over execution and programming. MATLAB’s library is a large collection of routines and functions for most common math and other computations. But to achieve highest performance one need its own customizations suitable to situation. MCCS provided tools which can be used to distribute and manage the work load among different nodes. This allows machines having different hardware configurations to be included in the cluster. The machines having high computation power can be allotted
3410
G. Ease of Use A survey taken by us revealed that MATLAB remains the most user friendly tools but one of the major drawbacks of this software package is time taken for analysis and simulation of a process requiring a decent computational power. MCCS was the choice of professionals with a lot of easy tools for managing the cluster. PVM was found to be to be the choice of those in need of total control of their work. PVM provides console interface for most of the tasks such as configuring and managing the cluster. H. Arithmetic Library and Tools Math library used in MCCS environment was Extreme Optimization Numerical Library for .NET. It is highly optimized library with a large collection of mathematical functions such as Fast Fourier Transform (FFT), Integral and Differential Calculus, etc., although its cost is high compared to other solutions. In our PVM setup we used Parallel Math Library, an open source library, and many functions were custom made. MATLAB is shipped with a large set of math library functions and routines which are all one can need, which balances its value for money. I.
Price Vs Performance MCCS setup was found to be the highest priced, as it included the Microsoft Windows Server 2008 Compute Cluster Edition, Microsoft Visual Studio for C#, and Extreme Optimization Numerical Library for .NET if we require professional math library. PVM in itself is open source with gcc compiler, parallel math library and Open SUSE, all of which accounts to the cheapest but the most trivial and unpopular solution. MATLAB on other hand comes with a huge set of library but with a high price. The above discussion shows that criterions of ease and performance need to be balanced out. There are some complexities involved with some of the available software packages but in other cases it gives more power throughput. Though there are some intricacies involved in PVM but it was found to have given the best implementation. We have made a C++ program on SUSE for a PVM based HPC setup. Though MATLAB has got a very rich and multifarious set of functions to perform complex mathematical computations but if performance was analyzed, it was the lowest out of the three packages. WCCS with extreme optimization library has been found to
2009 IEEE International Advance Computing Conference (IACC 2009)
be easy to setup and gave us a rich feature, instruction and function set. For professionals MCCS and Extreme Library combination suits the best with high scalability and reliability powered with high performance and ease of use. MATLAB on other hand is suitable for scientists and those who want to get started right off. PVM being the most trivial is for those who knows computer well and those want the highest performance out of their cluster setup.
[2] [3]
[4] [5]
CONCLUSION A comprehensive approach to cover all available HPC models for Multi Scale simulation was studied and implemented. We have shown the different models of HPC available and the challenges involved in connections and implementations. We have exposited and conferred the various problems that exist and have illustrated the connections and modeling. MPI, PVM on WCCS and Open SUSE Linux have been connected. The category wise performers have been demonstrated keeping the criteria as ease of handling, speed, and overall balance and support. The maximum performance in ease of handling was given by MATLAB with its inbuilt mathematical library and distributive toolbox but it had high memory consumption and greater processing power requirement. The best results in speed were given by PVM – Linux and MPI.NET. Ease of handling was found to be the least in PVM but it being open source it gave us freedom in changing the source code to meet out case specific requirements. Finally overall balance and support was found optimum in MPI.NET WCCS. PVM was found to be difficult to configure for a novice user and MATLAB was the easiest to work with in this regard. MATLAB is bundled with superior mathematical tools and a distributive computing toolbox that is quick to install and easy to work with. Open source PVM SUSE setup proved to be the most powerful and gave maximum performance. Also, availability of code makes it easy to be developed and tuned to case-specific requirements. Multi Scale Modeling needs to be optimized according to the tools we plan to work with. The efficient breakdown of the calculations depends on the Macro- to Nano- realms that are intended to be computed. The challenges are to combine the tools of system level to a powerful numerical library, in case of MPI.NET extreme optimization was used which found be very efficient. Other language such as F# can be used in a HPC setup using .NET framework. ACKNOWLEDGMENT
[6]
[7]
[8]
[9]
[10] [11]
[12] [13] [14] [15]
[16] [17]
[18] [19] [20] [21]
[22]
We would like to thank MicrosoftTM for funding a module of this project for implementation on WCCS. [23]
REFERENCES [1]
Nakano A., Bachlechner M.E., Kalia R.K., Lidorikis E., Vashishta P., Voyiadjis G.Z., Campbell T.J., Ogata S., Shimojo F., “Multiscale simulation of nanosystems,” Proc. IEEE Computing in Science & Engineering, Volume 3, Issue 4, pp. 56-66, Jul/Aug 2001.
Fox, B., Liu, P., Lu, C., Lee, H.P., “Parallel multi-scale computation using the message passing interface,” Proc. International Conference on Parallel Processing Workshops, pp. 199 – 204, 6-9 Oct. 2003. Stephane Velou Ble, Adam W. Skorek, “Parallel Approach to the Nanothermal Numerical Analysis,” Proc. IEEE Canadian Conference on Electrical and Computer Engineering, CCECE '06., pp. 21442146, May 2006. Allec, N., Knobel, R.G., Shang, L., “SEMSIM: Adaptive Multiscale Simulation for Single-Electron Devices,” IEEE Transactions on Nanotechnology, Vol. 7, Issue 3, pp. 351 – 354, May 2008. Ce Wen Nan, Junbo Wu, Jun Nan, Xisong Zhou, Jianzhong Zhang, “Multiscale approaches to thermoelectric materials and devices,” Proc. IEEE XX International Conference on Thermoelectrics, ICT 2001, pp. 18-23, June 2001. Demongeot, J., Bezy-Wendling, J., Mattes, J., Haigron, P., Glade, N., Coatrieux, J.L., “Multiscale modeling and imaging: the challenges of biocomplexity,” IEEE Engineering in Medicine and Biology Magazine, Vol. 91, Issue 10, pp. 1723 – 1737, Oct. 2003. Xiaotu Li, Jizhou Sun, Shi Chen, Yao Liu, Ce Yu, Zunce Wei, "An implementation scheme of PVM network parallel computing," IEEE Canadian Conference on Electrical and Computer Engineering, CCECE 2003, Vol. 2, Issue , pp. 1135 - 1138, 4-7 May 2003. Vaughan P.L., Skjellum A., Reese D.S., Fei-Chen Cheng, “Migrating from PVM to MPI.I. The Unify system,” Proc. Fifth Symposium on the Frontiers of Massively Parallel Computation, Frontiers '95., pp. 488-495, 6-9 Feb 1995. Haldar M., Nayak A., Kanhere A., Joisha P., Shenoy N., Choudhary A., Banerjee P., “Match virtual machine: an adaptive runtime system to execute MATLAB in parallel,” Proc. International Conference on Parallel Processing, pp. 145-152, 2000. Choy, R.; Edelman, A., “Parallel MATLAB: Doing it Right,” Proc. IEEE, Vol. 93, Issue 2, pp. 331 – 341, Feb. 2005. Lysiak K., Polendo J., “A generalized MATLAB-based distributedcomputing optimization tool,” Proc. IEEE/ACES International Conference on Wireless Communications and Applied Computational Electromagnetics, pp. 170 – 173, 3-7 April 2005. Windows Compute Cluster Server 2003 Administrators Guide. Windows Compute Cluster Server 2003 Users Guide. Using Windows HPC Server 2008 Job Scheduler, Microsoft Corporation, Published: June 2008, Revised September 2008. Bliss, N.T. Kepner, J. Kim, H. Reuther, A., “pMATLAB: Parallel MATLAB Library for Signal Processing Applications,” Proc. IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2007., Vol. 4, pp. IV-1189-IV-1192, 15-20 April 2007. Extreme optimization Library http://www.extremeoptimization.com/ Downloads.aspx Geist Adam, Beguelin Jack, Dongarra Weicheng, Jiang Robert, Manchek Vaidy Sunderam, “A Users Guide and Tutorial for Networked Parallel Computing,” Parallel Virtual Machine, The MIT Press Cambridge Massachusetts London England. Cameron Hughes, Tracey Hughes, “Parallel and Distributed Programming Using C++,” Addison Wesley, August 25, 2003. Windows HPC Server 2008: System Management Overview, Microsoft Corporation, Published: June 2008, Revised September 2008. Windows HPC Server 2008 Job Templates, Microsoft Corporation, Published: June 2008, Revised September 2008. Hamdi, M., Ferreira, A., “Multiscale design and modeling of nanorobots,” Proc. IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2007, pp. 3821-3827, Oct. 29-Nov. 2 2007. Bernholc, J.; Ranjan, V.; Ribeiro, F.; Lu, W.; Yu, L.; Nardelli, M.B., “Multiscale Simulations of High Performance Capacitors and Nanoelectronic Devices,” DoD High Performance Computing Modernization Program Users Group Conference, pp. 194 – 199, 1821 June 2007. Jae Hyun Kim, Jung Yup Kim, Byung Ik Choi, “Multi-scale analysis and design of nano imprint process,” Proc. Third IEEE Conference on Nanotechnology, IEEE-NANO 2003, Vol. 1, pp. 263- 266, 12-14 Aug. 2003.
2009 IEEE International Advance Computing Conference (IACC 2009)
3411
