Solving PDEs using PETSc in NetSolve - Springer

Solving PDEs using PETSc in NetSolve Xiaobin Zhang, Wu Zhang, Guoyong Mao, and Linfeng Yang School of Computer Engineering and Science, Shanghai University, Shanghai 200072, P.R. China [email protected]

Abstract. A new way of combine the widely used PETSc math library and NetSolve (gird computing middleware) is explored in this paper. Using PDF file of NetSolve, applications using PETSc libraries can be called by NetSolve. However, the disadvantage of NetSolve Server causes the MPI’s programs can not make good use of the Server’s resource, which will lead to performance degradation of Server. A middle layer is added between the daemon module of the Server and the application module, which can be used to estimate the demand of application module. Based on the estimation, realtime configuration file can be created to enhance the efficiency of the Server. Key words: PETSc, NetSolve, Grid Computing

1 Introduction In last two decades, the rapid growth in the supply of basic computing resources, like PETSc[1](Portable, Extensible Toolkit for Scientific Computing), and advance in hardware, networking infrastructure and algorithms, let us successfully attack a wide range of computationally intensive problems using network scientific computing. Grid computing middleware has supplied researchers with a better solution to implement modern computing technology. Netsolve[2] is specially designed to build a flexible, powerful, and easy computing environment. Based on NetSolve, Moore has succeeded in implementing remote numerical computing education platform, and researches surrounding NetSolve are on the rise[3]. Additionally, the high-performance computing community has explored a variety of approaches for implementing parallel scientific applications[4].For largescale PDE-based simulations, using existing math library - PETSc is a good choice. It is of great significance to integrate PETSc into NetSolve, because existing good math library can be employed to speed up the developing of high efficiency programs, and the middleware provides a universal calling interface to

606

Xiaobin Zhang, Wu Zhang, Guoyong Mao, and Linfeng Yang

directly use the computing resources. In this article, solving of PDEs problem is successfully implemented using PETSc, and this solution is integrated into an internal service of NetSolve. The following paper can be divided into 5 parts: Section 2, grid computing middleware; Section 3, numerical computing library; Section 4, integrate PETSc into NetSolve; Section 5, testing platform and performance analysis; Section 6, conclusion.

2 Computing Middleware of Web Service 2.1 NetSolve and scientific computing environments (SCEs) The NetSolve acts as glue layer that brings the application or user together with the hardware and/or software it needs to complete useful tasks. The NetSolve can be divided three layers: Client(API), agent and server. The NetSolve client library is linked in with the user’s application. Through NetSolve’s API, client-users gain access to aggregate resources without the users needing to know anything about computer networking or distributed computing.

Fig. 1. The Architecture of NetSolve

Figure 1 helps to show what the programming code would look like before and after the NetSolve API has been integrated[2]. The NetSolve agent represents the gateway to the NetSolve system. The NetSolve server is the computational backbone of the system. It is a deamon process that awaits client requests. The server can run on single workstations, clusters of workstations, symmetric multi-processors or machines with massively parallel processors. A key component of the NetSolve server problem is a source code generator which parses a NetSolve problem description file (PDF). This PDF contains information that allows the NetSolve system creates new modules and incorporates new functionalities[2]. In essence, the PDF defines a wrapper that NetSolve uses to call the function being incorporated. 2.2 Expanding the Server Capabilities NetSolve is highly scalable. PDF is the mechanism through which NetSolve enables services for the user. First, user must write PDF files if he want to

Solving PDEs using PETSc in NetSolve

607

add new functions to NetSolve. In addition, user should modify the server configuration file to enable or disable specific functionality to customize their server. At last, users recompile the server. This feature of NetSolve makes it easy to incorporate PETSc applications. 2.3 NetSolve Pitfall NetSolve supports programs of MPI type, which is popular in today’s parallel programs. Therefore, it is possible for NetSolve to provide services of large scale computation. Several schemes about MPI applications are provided by NetSolve installation software packages. In PDF, section @PARALLEL is used to point out that MPI is needed in scheme, and users can call MPI instances easily in section @CODE. After the NetSolve server is reconstructed, with the help of MPI, this scheme can use multiple processors to solve problems. However, only one MPI configuration file is allowed to use in one NetSolve server[2]. That is, many MPI schemes will use identical configuration, which can not be adjusted. For NetSolve server that can provide many services, this configuration will cause heavy performance degradation. For example, the job in Server is small, yet it is dispatched among multiple machines.

3 Problems, Algorithm and PETSc 3.1 Bratu’s problem and Newton-Krylov-Schwarz The Bratu problem is a classic PDE, the corresponding differential equation is[4]: ∇2 u − λeu = 0,

(1)

In this equation, λ is a constant, u is the variable to be computed whose value equals 0 at the boundary. The solution to this problem exists in PETSc sample examples. Our test is based on this example. We use 5-point stencil, a standardized approach[4], to discretize this problem, that is: f (u) = 4ui,j − ui−1,j − ui+1,j − ui,j−1 − ui,j+1 − h2 λeui,j ,

(2)

where f is the vector function of nonlinear residuals of the vector of discrete unknowns u, defined at interior and boundary grid point: ui,j ≈ u(xi , yj ); xi ≡ ih, i = 0, 1, · · · , n; yj ≡ jh, j = 0, 1, · · · , n; h ≡ 1/n. Newton iterative method can be used to solve Equation (2), but the precision may not be high due to an incomplete convergence employing the true Jacobian matrix or using an inexact or a ’lagged’ Jacobian[4][5]. In NewtonKrylov method, Krylov method is used to solve the Newton correction equations. While in the Krylov method, matrices are partitioned into submatrices,

608

Xiaobin Zhang, Wu Zhang, Guoyong Mao, and Linfeng Yang

the product of submatrices and vectors is computed and stored in each iteration. When the original matrix becomes Jocabian matrix after PDEs are discretized, the overhead of computation and communication for each matrixvector product are similar. 3.2 PETSc Implementation If Newton-Krylov method is used to solve PDEs using PETSc, solver and user’s code will deal with data in turn in one iteration. In general, five modules should be written by user: master process (or daemon process) module, initialization module, nonlinear function computing module, Jacobian matrix computing module and post processing module. Command line arguments can be used to select different way of solution and specifically set up parameters (i.e. times of iteration, convergence condition) when calling PETSc based applications.

4 Improve flexibility of PETSc applications However, from section 2 we can see that the support of NetSolve to MPI is not very good. In order to overcome this disadvantage, we changed the sharing of one configuration file into the realtime creation of different configuration file in accord with different tasks. A middle layer is put between NetSolve Server and solution modules of MPI based applications, the layer is to analyze the parameters send by client, estimates the amount of computation, and give out the demand of resources. Based on this estimation, realtime MPI configuration file is created. In our test, different numbers of processors are assigned only based on the size of matrices.

5 Testing environment and performance analysis All performance data reported in this study are measured on ZQ2000 cluster computer of Shanghai University. If there’s no middle layer between Server and test application, when the number of subintervals is fixed, the execution time will decrease slack with the increasing of number of processors for 256×256 matrix. The research of Hayder et.al[4] on PETSc also proved this. However, this may not always be the case, the execution time became longer if the the number of processors exceeds certain point. For equal number of processors, the execution time will increase almost linearly with the increasing of the number of subintervals. Compared with the former two tests, if the ratio of matrix size to the number of processors is a constant, as shown in Fig. 2, the execution time will increase

Solving PDEs using PETSc in NetSolve

609

Fig. 2. Scaling of execution time

very slowly, which indicates that a smaller number of processors can be used for matrices with smaller size so as to improve the efficiency of Server. In this paper, we proved that the realtime adjustment of MPI configuration file in accord with the amount of computation can significantly improve the overall performance of Server, and the resources can be saved. Acknowledgements. This paper was supported by “SEC E-Institute: Shanghai High Institutions Grid” project, the fourth key subject construction of Shanghai, and the Key Project of Shanghai Educational Committee (No.03AZ03).

References 1. Satish Balay, Kris Buschelman, Victor Eijkhout, William Gropp et. al. PETSc Users Manual, 2004, http://www.mcs.anl.gov/petsc 2. Dorian Arnold, Sudesh Agrawal, Susan Blackford, Jack Dongarra et al., Users’ Guide to NetSolve V2.0, 2003, http://icl.cs.utk.edu/netsolve/ 3. Shirley Moore, A.J. Baker, Jack Dongarra, Christian Halloy, Active Netlib: An Active Mathematical Software Collection for Inquiry-based Computational Science and Engineering Education, 2002, http://www.cs.utk.edu / shirley /jodi /active-netlib.html 4. M. Ethesham Hayder, David E. Keyes, Piyush Mehrotra, A comparision of PETSc library and HPF implementations of an archetypal PDS computation, Advances in Engineering Software, Vol. 29(3-6), 415-423, 1997 5. J. Dennis, R. Schnabel, Numerical Methods for Unconstrained Optimization and Nonlinear Equations, Prentice Hall, 1983