gGEM: A Gyrofluid Model to be Used on ... - Semantic Scholar

IEEE TRANSACTIONS ON PLASMA SCIENCE, VOL. 38, NO. 9, SEPTEMBER 2010

2111

gGEM: A Gyrofluid Model to be Used on Distributed Platforms Manuel Rodríguez-Pascual, Bruce D. Scott, Tiago T. Ribeiro, Francisco Castejón Magaña, and Rafael Mayo García

Abstract—The study of core turbulence represents a key line of research in fusion plasmas. By adding collisions and electromagnetic induction to the parallel dynamics of the standard six-moment toroidal model, it is possible to study the gyrofluid electromagnetic phenomena in the context of edge turbulence with the GEM code. Currently, the code describes the fluctuation free-energy conservation in a gyrofluid model by means of the polarization equation which relates the ExB flow and eddy energy to the combinations of the potential, the density, and the perpendicular temperature. To do so, supercomputers have been used only to date. In this paper, we demonstrate its feasibility as a cluster application on a production environment based on any kind of distributed memory, enhancing in this way its scope. The scalability (which grows linearly with a correlation factor of 0.99978) and the correctness of our solution with respect to the previous GEM version have been evaluated in a local cluster of 88 nodes. The fault tolerance and the Grid suitability have been demonstrated by executing our application in the EUrope Fusion for ITER Applications infrastructure by adapting the code to this paradigm and by improving its parallel Grid performance. It can be employed on its own or belonging to workflows in order to perform a wider more complex analysis of fusion reactors. Index Terms—Edge turbulence, GEM gyrofluid code, Grid applications.

I. I NTRODUCTION

T

HE BASIC IDEA OF the gyrofluid model is to replace the emphasis on space variables (density, velocity, etc.) by their guiding center counterparts. The complications of finite gyroradius effects on the ion inertia are replaced by treating the electron and ion guiding center densities separately and then joining them in a gyrofluid Poisson equation which takes care of the polarization. In this equation, the polarization density enters through the gyroradius rather than the Debye length; otherwise, the form of the equation is the same as the usual Poisson equation relating the electrostatic potential to a total charge density. If the Debye wavelength is much smaller than the drift wave

Manuscript received December 1, 2009; revised May 5, 2010; accepted June 18, 2010. Date of publication July 26, 2010; date of current version September 10, 2010. M. Rodríguez-Pascual and R. Mayo García are with the CIEMAT, 28040 Madrid, Spain (e-mail: [email protected]; [email protected]). B. D. Scott is with the Max-Planck-Institut für Plasmaphysik, 85748 Garching, Germany (e-mail: [email protected]). T. T. Ribeiro is with the Max-Planck-Institut für Plasmaphysik, 85748 Garching, Germany, and also with the Instituto de Plasmas e Fusão Nuclear, 1049-001 Lisbon, Portugal (e-mail: [email protected]). F. Castejón Magaña is with the CIEMAT, 28040 Madrid, Spain, and also with the Laboratorio Nacional de Fusíon, 28040 Madrid, Spain (e-mail: [email protected]). Color versions of one or more of the figures in this paper are available online at http://ieeexplore.ieee.org. Digital Object Identifier 10.1109/TPS.2010.2055163

dispersion scale, the actual charge density polarization term is always small and the gyropolarization term is the only one present to control the relationship between Φ and the density variables of the fluid species. The polarization equation actually arises in the gyrokinetic theory [1] which amounts to taking the drift approximation at the kinetic level before building the fluid equations through the moment hierarchy. A particularly clear exposition of the basic idea is given in [2]. According to this, the toroidal effects [3] and electromagnetic parallel electron dynamics so as to be able to treat the generalizations of such results as the nonlinear drift wave instability [4] can also be taken into account. The turbulence in a tokamak plasma consists observationally of the electron density fluctuations and the associated phenomena, as measured by diagnostics such as the Langmuir probes [5], which also measure the electrostatic potential, so the latest serves as the stream function for the ExB velocity. On a theoretical level, the basic mechanism active at the scales where these phenomena occur (low frequency and a moderately small scale perpendicular to the magnetic field) [6] is a competition between the largely incompressible fluidlike ExB eddy turbulence in the planes perpendicular to the magnetic field and the streaming or wavelike motion parallel to it [7]. The gyrofluid model is directly derived from an underlying gyrokinetic description and can be thought of as a simplified form of it. While gyrokinetic computations are making headway, the electromagnetic versions are just beginning, and no model has demonstrated capability in the strongly collisional regime or with spatial grids above 1000 points in the perpendicular dimensions. Several problems of interest, such as the wide-spectrum tokamak edge turbulence and global computations treating the interactions between the turbulence and magnetic islands, require arbitrary collisionality and wide scale separation in both space and time. These must be done without relaxing such parameters as the electron/ion mass ratio and the ion gyroradius/system size scale ratio. At present, this capability remains with the fluid models. On the other hand, the collisional fluid models are limited to the regimes of large collision frequency (short mean free path) and small gyroradius while the problems of interest involve the ion gyroradius at the unit order on the small-scale side and also the ion dissipation processes at both the weak and strong collisionality (sometimes within the same problem, e.g., edge turbulence, where the effective collisionality depends on the spatial scale). The gyrofluid model has been shown to cover all the aspects of the related low-frequency collisional fluid model in the large-scale/strongcollisionality regime while extending the validity to the unitorder gyroradii, weak-collisionality ion viscosity, and thermal conduction processes [8]. Its main shortcoming is that there

0093-3813/$26.00 © 2010 IEEE

2112

is no effective model to treat the magnetic mirror trapping of the charged particles in a toroidal magnetic field. While simple models exist for the electrostatic cases with adiabatic electrons, they have not managed in the electromagnetic settings. The problems involving both electromagnetic responses and trapped electrons should therefore await future gyrokinetic models. However, from the standpoint of the computational techniques, the gyrokinetic and gyrofluid codes commonly share a similar infrastructure, and therefore, any advance in one is applicable to the other and, therefore, relevant to future computational challenges. This also makes the gyrofluid model effective as a learning tool since it typically requires resources of about two orders of magnitude fewer than the corresponding gyrokinetic version. According to all of this and as a summary, the gyrofluid models have been constructed to incorporate finite ion gyroradius effects at arbitrary order into the simple computations of the turbulence occurring in largely 2-D fluid experiments. To treat the ion temperature gradient (ITG) turbulence, the temperatures were incorporated and the model acquired several new advection terms, producing nonlinearities as well as drift frequency corrections, resulting from the effect of temperature fluctuations on the gyroaveraging operator. However, although the nonlinearities were incorporated, much of the analysis involved linear frequencies and growth rates, and the nonlinearities were added largely as an afterthought. Specifically, there was no complete energetic analysis of the type familiar from the drift wave turbulence work, including the drift Alfvén Landaufluid model [9]. The GEM model [10] was introduced in the context of edge turbulence studies, which is a matter of adding collisions and electromagnetic induction to the parallel dynamics of the“standard” six-moment toroidal model previously used in the core turbulence studies [3]. The energetics was developed for the simplified two-moment version to highlight the correspondence between the drift-fluid and gyrofluid models under drift ordering [6]. The energetics was developed for the sixmoment version, including the temperature and parallel heat flux dynamics, placing the model on the same secure energetic footing as that of the older drift-fluid models. Some improvements on the GEM code [11] were made, including a different formalism for fully inhomogeneous equations where the diamagnetic cancellation was worked out for anisotropic temperature [12], so the standard gyrofluid model to preserve its energetic integrity was developed. The resulting system exhibits the same qualitative energy-transfer properties as that of the corresponding Braginskii or Landau-fluid model. One clear result was that the numerical model built on these equations behaves well for an arbitrarily large perpendicular wavenumber, allowing the exploration of the two-scale phenomena linking dynamics at the ion and electron gyroradii. When the numerical formulation is done in the globally consistent flux tube model, the results with the adiabatic electrons are consistent with the Cyclone base case results of the gyrokinetic models. By employing numerical simulations to recreate the natural processes, it is possible to simulate accurately the physical phenomena as mentioned previously. Thus, the stunning evolution of computing technology–in both software and hardware–has


allowed scientists to solve complex problems, the analytical resolution of which would be practically impossible as that of the turbulence in fusion plasmas. The vast amount of resources needed for building and maintaining an experimental device, together with the difficulties of observing the phenomena happening inside the plasma, which are sometimes in the microsecond scale, makes computing the simulation an essential tool. There are software applications that can simulate different sections of a nuclear reactor, whose results have been validated [13], [14]. In this context, the computational Grid arises as a solution for the fusion community, allowing its members to share their local infrastructures and create virtual organizations which can provide the final users a great amount of computational resources and storage. This can be the case for the study of the tokamak edge turbulences, and specifically, for the GEM code; a summary of its background theory and its related computation can be found in [15]. Although MPI [16] applications present some difficulties to be executed on the Grid networks (scalability, limited resources, and multiple versions of MPI libraries) as GEM has, several projects like the int.eu.grid (http://www.i2g.eu.), the EGEE (http://www.eu-egee.org/), or the EUrope Fusion fOR ITER Applications (EUFORIA) (http://www.euforia-project. eu/) have performed great efforts to improve their portability and simplify their usage. Grid computing has finally shown its capability to be used for executing this kind of applications, providing a massive amount of resources in the form of both computation time and storage. The aim of this paper is to explain how GEM has been adapted in order to improve its Grid execution by employing a DRMAA (Distributed Resource Management Application API) [17]. This Grid tool, called Grid-GEM (gGEM), is an MPI version of the application with no dependence on any other external libraries, suitable to be executed on Grid infrastructures. With it, several turbulence problems partitioning the plasma torus in slices can be resolved, as well as having an available serial version of the codes for testing purposes. In addition, more complex problems to be overcome by means of computational workflows can now be performed by coupling the gGEM code with some other such as ISDEP [18], Helena [19], and ETS [20]. II. C ODE AND I TS P HYSICS As previously mentioned, the gyrokinetic model is derived directly from an underlying gyrokinetic one, subject to the same approximations in the following cases: “delta-f” for the moderate-amplitude cases for a single spatial region and “total-f” for the global cases with strong stratification in the parameters as well as in the gradients [3], [10], [11]. The aim is to treat what can be called the transcollisional regime for problems with a large degree of scale separation. The kinetic processes, such as the Landau damping, which do not require magnetic mirror trapping are treated. Additionally, the collisional dissipation model naturally becomes inactive as the collision frequency becomes small, in contrast to a conventional fluid model in which this results in several pathologies, particularly in the ion dissipation mechanisms. The gyrofluid model treats the

RODRÍGUEZ-PASCUAL et al.: GGEM: GYROFLUID MODEL USED ON DISTRIBUTED PLATFORMS

nonlinear ion inertia details in a different way than the fluid model, but it has been shown to recover the fluid model naturally in the collision-dominant regime [8]. Gyrofluid models have been usually applied to tokamak core turbulence for incorporating finite ion gyroradius effects at arbitrary order into simple computations of the turbulence occurring in the largely 2-D fluid experiments [2]. Then, the temperatures were incorporated in order to treat the ITG turbulence, and the model acquired several new advection terms, producing nonlinearities as well as drift frequency corrections, resulting from the effect of the temperature fluctuations on the gyroaveraging operator [3]. Thus, the GEM model was introduced in the context of edge turbulence studies [10], which is a matter of adding collisions and electromagnetic induction to the parallel dynamics of the standard six-moment toroidal model previously used in the core turbulence studies [3]. Several improvements have been done in the code since then, and now, it is able to describe the fluctuation free-energy conservation in a gyrofluid model. The polarization equation relates the ExB flow and eddy energy to the combinations of the potential, the density, and the perpendicular temperature. These combinations, which appear under the derivatives in the moment equations, imply that not only the thermal free energy but also its combination with the ExB energy is properly conserved by the parallel and perpendicular compressional effects. This is achieved by placing the standard local gyrofluid model on energetically consistent grounds, with the moment variables and the electrostatic potential given a full finite Larmor radius (FLR) treatment at the same level of sophistication. The main difference with the similar codes using six-moment variables per species [3], [10], [21] is that GEM restores the consistency in the polarization and moment equations by setting the FLR term series associated to both equations in the same order for the energetics. This is of outmost importance because it will be possible to recover the results emerging from the gyrokinetic computations with a computationally more tractable model. The model can also increase or decrease its level of sophistication while retaining energetic and geometric consistency, so its flexibility has been improved. This system shows the same qualitative energy-transfer properties as that of the corresponding Braginskii or Landau-fluid model and behaves rightly for an arbitrarily large perpendicular wavenumber, allowing the exploration of two-scale phenomena linking dynamics at the ion and electron gyroradii. Among the problems that GEM deals with, we can mention that the proper edge turbulence in the electromagnetic regime can be simulated, as well as the density and parallel velocity for both electrons and ions, showing the role of the 3-D drift wave nonlinear instability in the context of the tokamak edge turbulence. It also addresses the turbulence in the ITG and ETG regimes. GEM provides a wide set of diagnostics relevant for the proper assessment of the system and employs the graphic library PGPLOT [22] to draw readily the results and make them easily understandable (although the output of the data files for later visual processing is also possible). Fig. 1 represents an example of a GEM output (as any of the ones present in this

Fig. 1.

2113

Two-dimensional morphology of the turbulence, calculated by gGEM.

paper), showing the 2-D spatial morphology of the turbulence, after a short run (only 100 time steps) when the system is starting from the initial conditions. A 3-D grid, mapped into a 2-D grid of processors, is employed to simulate the plasma. Then, the communication is established among the logical neighbors (those that simulate neighbor cells), and the state of each cell is calculated along the time. To perform these operations in an efficient way, the cells are assigned to the processors in such a way that the communication is kept to a minimum. As cells have to communicate with each other in every step of the simulation, the application is highly coupled, so the MPI is employed in order that the different processors can communicate. III. W HAT I S THE G RID The problem that the Grid aims to solve is how to coordinate resource sharing and problem solving in dynamic multiinstitutional virtual organizations [23]. Of course, this sharing must be highly controlled, so both the users and the service providers know what, when, to whom, and under which conditions a certain resource is shared. These conditions define the groups of users called the Virtual Organizations (VOs). An intuitive definition of the Grid could be as follows: While the World Wide Web is a service to share information on the Internet, the Grid is a service to share computational and storage resources on the Internet. A more classical definition can be found in [24]. According to this, a Grid infrastructure is a system that does the following: 1) Coordinates resources that are not subject to centralized control; 2) Uses standard open general-purpose protocols and interfaces; 3) Deliver nontrivial qualities of service. The first point indicates that the resources do not belong to a single person or organization, so there is no centralized absolute

2114


control. On the contrary, the employment of these resources is subordinated to the authorization of their administrator, as well as the installed software, security police, or any other criteria. The second point explains that the Grid aims to be a generalpurpose tool, not oriented to a certain kind of application or usage. To achieve this, it is mandatory to be an open environment, so everybody can adapt it to his/her specific needs and can be implemented it in as many architecture and systems as possible. The employment of standard protocols avoid fragmentation–the existence of Grid networks employing different protocols that cannot collaborate. However, the aforementioned facts are useless if the resulting infrastructure is not stable and reliable and its utility is not greater than the sum of its parts. After all, the Grid is a tool to provide its users the resources they need to carry on their tasks.

way, GEM was ported to a 32-b architecture to maximize the places it can be executed at the moment this work was carried out, particularly given that the 64-b processors can also execute 32-b applications. Of course, when the 64-b processors spread in the Grid sites, the compiling options will be adapted in order to fit this requirement. Given that the original code is designed for the 64-b processors, this operation should be seamlessly carried out. 3) Serial Application: With GEM being a CPU-intensive application that needs multiple processors for obtaining relevant results in a reasonable amount of time, a serial version of the code is not suitable to be employed on a production environment. Anyway, a serial version of the code not needing MPI support is extremely useful for testing purposes. It can be employed for checking the feasibility of a given set of input parameters, for example, or during the development of a workflow.

IV. I MPLEMENTATION In this section, the Grid porting process of GEM is detailed, including the practical details of the creation of an MPI version of GEM suitable to run on Grid infrastructures, a serial version of GEM, and the DRMAA application, gGEM. The idea behind this multiple-version approach is to have the most suitable application depending on the user needs: MPI version for wide simulations and serial version for short runs and testing purposes. gGEM works as an interface that simplifies the task of performing any number of concurrent executions of the aforementioned versions on the Grid. A. Grid-Enabled MPI Application 1) Removal of the Dependences: The first task consisted of the removal of the dependence on external libraries. When running an application on a remote Grid site, no assumption about the installed software should be done, so the binary to be executed must be fully independent. GEM employs the mathematical library Fastest Fourier Transform in the West (FFTW3) [25] to compute the discrete Fourier transform, so this library must be correctly installed and configured in the system. To remove this dependence, a static compilation of the library was performed. Then, the makefile of GEM was modified in order to employ this static FFTW3 installation, and the compiler was forced to include this library in the resulting binary file. In this way, the application can be executed on any remote site independent of whether FFTW is installed or not, and if it is installed, whether it is the correct version. 2) Architecture and MPI Requirements: As GEM is going to be executed on the Grid networks, it has to fit the requirements of the standard sites. In this case, it is designed to be executed on the EUFORIA infrastructure [26], so it must employ OpenMPI 1.2.5. This infrastructure employs clusters of 32-b x86 processors, so it was necessary to compile GEM to fit this requirement. A small adjustment was made in order to allow the application to be compiled with the 8-B option for reals and integers. Users must be aware that 32-b processors have a limited lifetime, as 64-b processors are quickly replacing them. Any-

B. DRMAA Application As part of the Grid porting process, a DRMAA-enabled application to simplify the Grid execution of GEM was created. gGEM employs the Grid to execute multiple concurrent instances of the MPI or serial versions of GEM. With a very intuitive interface, the final user can submit to the Grid any number of GEM instances, each one with a different set of parameters and computing requirements. gGEM reads the input files, extracts the number of required CPUs, and executes GEM on a suitable resource. The main advantage of this DRMAA application is that it manages all the Grid-related operations, so the final user is released from these tasks. This does not only simplify the work of the final user but also reduces the risk of failures or errors by providing the user a very intuitive interface (see Section IV-B2 for details). In this way, the final users have all the computation capabilities of the Grid computing without the need of acquiring new knowledge or managing new tools. 1) Metascheduler: DRMAA constitutes a standard interface to communicate the application with many schedulers and metaschedulers. In this case, GridWay metascheduler has been employed, as it provides some key functionalities. 1) It automatically performs the steps involved in job submission: system selection, system preparation, submission, monitoring, migration, and termination [27]. 2) It has a user-friendly DRMAA connection. By linking the DRMAA library of GridWay with our own C++ application, the submission and control of the job execution is delegated to GridWay. 3) It has scheduling capabilities. The scheduler of GridWay is able to perform a dynamic scheduling, adapting to the changing hardware conditions. Moreover, it employs the information published by each node to submit each application to the best nodes, depending on its requirements. 4) It has fault detection and recovery capabilities. Transparently, to the end user, GridWay is able to detect and recover from any of the Grid elements the failure, outage, or saturation conditions [28].


2115

2) Interface and Functionality: When developing gGEM, the interface was designed in order to be as simple as possible. The proposed solution is explained in ggem number [mpi|serial f ile1 f ile2] . . .

(1)

where number indicates the number of simulations to be executed, mpi|serial allows the user to choose between the serial or the MPI version of GEM, f ile1 represents the path to the file with the input parameters, and f ile2 is the file containing the initial state. gGEM reads the input data and, in the case of the simulations employing the MPI library, extracts the number of desired nodes. Then, it builds a GridWay template for every simulation and submits it to the Grid. In the remote node, a script renames the input files (to fit the GEM requirements), executes the application, and packs the results into a .tar.gz file. Then, this file is copied back, and the simulation is complete. After all the simulations are finished, gGEM notifies the user.

Fig. 2. gGEM execution time when running on 1–32 processors.

V. E XPERIMENTAL R ESULTS A. Test-bed Description The behavior of gGEM has been analyzed on a local cluster and on the EUFORIA Grid infrastructure. The local cluster, Lince, sited at the CIEMAT facilities consists of an 88-node (290 slots) cluster composed by Dual Xeon 3.2 GHz processors. All the nodes are connected to two networks, a 1-Gb/s Ethernet, and an Infiniband, and the user decides which one he wants to employ for each application. As most Grid sites employ Ethernet at this moment, that has been the chosen connection, therefore, the usual Grid environment is reproduced as accurately as possible. The scalability of the MPI version of gGEM has been analyzed in a local cluster in order to have a well-known controlled environment. In this way, the alterations due to the usage of different hardware resources are eliminated. Then, gGEM has been executed on the EUFORIA infrastructure to check the feasibility of the proposed solution in a production environment. The gLite 3.1.0 middleware has been employed to submit gGEM to a EUFORIA Grid site, namely i2gce01.ifca.es. B. MPI Version Performance To analyze the performance of the MPI version, it was executed several times with an increasing degree of parallelism. It is a key factor that the Lince cluster was in a production status when the tests were performed. It was not devoted only to this performance analysis. Several users were executing different applications; therefore, the scalability can be considered as very close to a real Grid environment. Fig. 2 shows the execution time when running gGEM from 1 to 32 processors with a common use case devoted to study the evolution of the turbulence in a fusion plasma. Fig. 3 depicts the total CPU time, that is, the product of the execution time multiplied by the number of nodes. The reason for studying the scalability up to 32 processors lays on the fact that the design

Fig. 3. Total CPU time after an execution of gGEM.

of the application and the lattice to be simulated are associated to a power of two. In addition, 32 processors represent a common Grid threshold for running distributed tasks on a unique Grid production site, so it would keep the Grid overhead as reasonable. As it can be seen, the total CPU time increases with the number of nodes. This is due to the employment of the Ethernet: As this application is highly coupled, it would profit from a more powerful network (Infiniband, for example) between the CPUs in order to better approximate to a linear scale. This fact could be considered as a future drawback, but it will probably be overcome according to the current tendency for building up new computational data centers, the concept of which has been taken into account in this GEM porting process. These data centers are structured in several computing islands of some hundreds of cores (even a little thousands) connected by Ethernet among them but by Infiniband inside their own network domain. With such a kind of platform, gGEM can profit from both aspects since it can make the most of the Infiniband link for the final execution of coupled tasks and of the Ethernet island connection for the submission and management of these jobs.

2116


Application availability is defined in [29] to be the fraction of time that the application is free to perform non-MPI-related work. The following equation is proposed Application availability = 1 − (overhead/transf er time). Regarding Fig. 3, it is noticeable that the total CPU time grows linearly with the number of processors, showing that gGEM is executing non-MPI-related work for a constant time, and the MPI-related work for a time that depends on the number of MPI tasks. Defining the work performed as the product of the application availability by total time and employing the previous equation, the following statements can be made W ork perf ormed =total time(1 CPU) W ork perf ormed =total time(32 CPU)−32 ∗ Overhead. With the information provided in Fig. 3, the values obtained are a work-performed length of 4780 s and an overhead of about 153 s/processor. This can serve as an upper bound to the degree of parallelization, depending on the needs of the final user: He/she can choose a low-level degree of parallelization in order to employ the computing resources to do as much useful work as possible or choose a high degree of parallelization when he needs the results in a shorter period of time. 1) Grid Overhead: When evaluating the Grid performance of a certain application, it is important to quantify the overhead induced by the porting process as well as the Grid execution model. The execution of gGEM requires copying the input files to the site where the application will be executed and bringing the output files back. The size of the input files plus the executable files and our script is about 10 MB, so any action over them is performed by means of a storage element. The overhead scales linearly with the number of nodes and is parallelizable: It is possible to transmit several files without performance loss (depending on the network connection of both the local and remote sites). Thus, it is difficult to quantify its influence on the execution time. Anyway, it is usually on the seconds scale, so we consider that it does not affect the scalability of gGEM. The most important overhead of gGEM, which in fact could limit its viability as a Grid application, is the size of the output files. This characteristic depends on the size of the simulated grid and an input parameter, nplot, that defines how many plot steps are made on each execution. Each plot step includes a snapshot, storing the state of each cell in that very moment. Obviously, the size of the output file linearly depends on this parameter and usually varies from a few hundreds of megabytes to some gigabytes. Depending on the length of the execution and the number of desired snapshots, this could become a bottleneck, so a good ratio between the number of required plot steps and the computational time should be achieved. For the sake of evidence, Fig. 4 shows some plasma parameters calculated by gGEM after a short run. The electrostatic potential and density fluctuation energies are among the parameters that are provided by the code. As can be seen, the system is still in the initial phase, so the time traces are mostly all still rising, but they show for longer times if the turbulence

Fig. 4.

Some volume-averaged quantities after an execution of gGEM.

Fig. 5. s/sink spectra during an execution of gGEM.

has reached a saturated state. With respect to the turbulence, other important quantities, such as the A and E spectra or the s/sink profile and spectra as a function of distance, have been calculated (see, for example, Fig. 5 for the latest). VI. F UTURE W ORK As it was explained in Section I, gGEM is only not useful by itself: It can show its full potential when integrated in a complex workflow. In such a software design, a more realistic modeling of a fusion reactor can be achieved by employing the Grid networks. This is a task to be done within the frame of the EU project, EUFORIA. So far, the proposed workflows are gGEM and Helena [19], coupled through ETS [20] (to simulate self-consistent turbulent transport, including equilibrium evolution), and ISDEP [18] and gGEM (to calculate the influence of turbulence on ion trajectories). To achieve a good performance on the workflow executions, our intention is to create a software infrastructure based on Kepler [30] and GridWay. With this design, a user with neither Grid nor workflow knowledge can employ Kepler as a graphic interface to design the workflow and then have the Grid related


capabilities of the GridWay, even if it is not installed on his/her local PC. The future work also includes working on the gGEM overhead (detailed in Section V-B1). By analyzing the data flow inside the workflow as well as the physical location of each task, the information flux will be optimized. VII. C ONCLUSION In this paper, GEM has been ported to the Grid, and its proper functioning has been checked in the two infrastructures it will be executed on: the EGEE and the EUFORIA. Its scalability in terms of the number of nodes has also been demonstrated. The size of its output data has been analyzed, as well as how it can affect the scalability depending on the size of the problem to simulate and on the desired number of snapshots. A DRMAA application to simplify the usage of GEM in the Grid, called gGEM, has been presented. At last, gGEM has been connected with the GridWay metascheduler to maximize its Grid performance and robustness. R EFERENCES [1] W. Lee, “Gyrokinetic approach in particle simulation,” Phys. Fluids, vol. 26, pp. 556–562, 1983. [2] G. Knorr, F. R. Hansen, J. P. Lynov, H. L. Pécseli, and J. J. Rasmussen, “Finite Larmor radius effects to arbitrary order,” Phys. Scr., vol. 38, no. 6, pp. 829–834, 1998. [3] M. A. Beer and G. W. Hammett, “Toroidal gyrofluid equations for simulations of tokamak turbulence,” Phys. Plasmas, vol. 3, no. 11, pp. 4046– 4064, Nov. 1996. [4] B. D. Scott, “Self-sustained collisional drift-wave turbulence in a sheared magnetic field,” Phys. Rev. Lett., vol. 65, no. 26, pp. 3289–3292, Dec. 1990. [5] C. Hidalgo, “Edge turbulence and anomalous transport in fusion plasmas,” Plasma Phys. Control. Fusion, vol. 37, no. 11A, pp. A53–A67, Nov. 1995. [6] B. D. Scott, “Computation of electromagnetic turbulence and anomalous transport mechanisms in tokamak plasmas,” Plasma Phys. Control. Fusion, vol. 45, no. 12A, pp. A385–A398, Dec. 2003. [7] M. Wakatani and A. Hasegawa, “A collisional drift wave description of plasma edge turbulence,” Phys. Fluids, vol. 27, no. 3, pp. 611–618, Mar. 1984. [8] B. D. Scott, “Nonlinear polarisation and dissipative correspondence between low frequency fluid and gyrofluid equations,” Phys. Plasmas, vol. 14, no. 10, p. 102 318, Oct. 2007. [9] B. D. Scott, “Three-dimensional computation of drift Alfven turbulence,” Plasma Phys. Control. Fusion, vol. 39, no. 10, pp. 1635–1668, Oct. 1997. [10] B. D. Scott, “ExB shear flows and electromagnetic gyrofluid turbulence,” Phys. Plasmas, vol. 7, pp. 1845–1856, 2000. [11] B. D. Scott, “Free-energy conservation in local gyrofluid models,” Phys. Plasmas, vol. 12, no. 10, p. 102 307, Oct. 2005. [12] D. Strintzi, B. Scott, and A. Brizard, “Nonlocal nonlinear electrostatic gyrofluid equations: A four-moment model,” Phys. Plasmas, vol. 12, no. 5, pp. 1–10, May 2005. [13] V. Tribaldos, “Monte Carlo estimation of neoclassical transport for the TJ-II stellarator,” Phys. Plasmas, vol. 8, no. 4, pp. 1229–1239, Apr. 2001. [14] A. Gómez-Iglesias, M. Vega-Rodríguez, F. Castejón, M. R. del Solar, and M. C. Montes, “Grid computing in order to implement a threedimensional magnetohydrodynamic equilibrium solver for plasma confinement,” in Proc. 16th Euromicro Conf. Parallel, Distrib. Netw.-Based Process., 2008, pp. 435–439. [15] B. D. Scott, “Tokamak edge turbulence: Background theory and computation,” Plasma Phys. Control. Fusion, vol. 49, no. 7, pp. S25–S41, Jul. 2007. [16] “Document for a standard message-passing interface,” Univ. Tennessee, Knoxville, TN, Tech. Rep., 1994. [17] P. Tröger, H. Rajic, A. Haas, and P. Domagalski, “Standardization of an API for distributed resource management systems,” in Proc. 7th IEEE Int. Symp. CCGrid, 2007, pp. 619–626.

2117

[18] F. Castejón, L. A. Fernández, J. Guasp, V. Martin-Mayor, A. Tarancón, and J. L. Velasco, “Ion kinetic transport in the presence of collisions and electric field in the TJ-II ECRH plasmas,” Plasma Phys. Control. Fusion, vol. 49, no. 6, pp. 753–776, Jun. 2007. [19] A. B. Mikhailovskii, G. Huysmans, W. Kerner, and S. Sharapov, “Optimization of computational MHD normal-mode analysis for tokamaks,” Plasma Phys. Rep., vol. 23, no. 10, pp. 844–857, Oct. 1997. [20] D. Kalupin, J. F. Artaud, D. Coster, S. Glowacz, S. Moradi, G. Perverzev, R. Stankiewicz, M. Tokar, V. Basiuk, G. Huysmans, F. Imbeaux, Y. Peysson, L.-G. Erikkson, M. Romanelli, and P. Strand, “Construction of the European transport solver under the European integrated tokamak modelling task force,” in Proc. 35th EPS Conf. Plasma Phys., 2008, vol. 32D, p. P-5.027. [21] W. Dorland and G. W. Hammett, “Gyrofluid turbulence models with kinetic effects,” Phys. Fluids B, vol. 5, no. 3, pp. 812–835, Mar. 1993. [22] The PGPLOT Tool. [Online]. Available: http://www.astro.caltech.edu/ ~tjp/pgplot/ [23] I. Foster, C. Kesselman, and S. Tuecke, “The anatomy of the grid: Enabling scalable virtual organizations,” Int. J. High Perform. Comput. Appl., vol. 15, no. 3, pp. 220–222, 2001. [24] I. Foster (2002, Jul.). What is the grid? A three point checklist. Grid Today [Online]. 1(6). Available: http://www-p.mcs.anl.gov/~foster/Articles/ WhatIsTheGrid.pdf [25] M. Friggo and S. G. Johnson, “The design and implementation of FFTW3,” Proc. IEEE—Special Issue on ‘Program Generation, Optimization, and Platform Adaptation’, vol. 93, no. 2, pp. 216–231, Feb. 2005. [26] P. Strand, B. Guillerminet, I. Campos Plasencia, J. M. Cela, R. Coelho, D. Coster, L.-G. Eriksson, M. Haefele, F. Iannone, F. Imbeaux, A. Jackson, G. Manduchi, M. Owsiak, M. Plociennik, A. Soba, E. Sonnendrucker, and J. Westerholm, “A European infrastructure for fusion simulations,” in Proc. 18th Euromicro Conf. Parallel, Distrib. Netw. Based Process., 2010, pp. 460–467. [27] E. Huedo, R. S. Montero, and I. M. Llorente, “Grid resource selection for opportunistic job migration,” in Proc. Euro-Par, vol. 2790, Lecture Notes in Computer Science, 2003, pp. 366–373. [28] E. Huedo, R. S. Montero, and I. M. Llorente, “Evaluating the reliability of computational grids from the end user’s point of view,” J. Syst. Archit., vol. 52, no. 12, pp. 727–736, Dec. 2006. [29] D. Doerfler and R. Brightwell, “Measuring MPI send and receive overhead and application availability in high performance network interfaces,” in Proc. EuroPVM/MPI, vol. 4192, Lecture Notes in Computer Science, 2006, pp. 331–338. [30] The Kepler Workflow Tool. [Online]. Available: http://kepler-project.org/

Manuel Rodríguez-Pascual received the M.Sc. degree in computing sciences from the Universidad Complutense de Madrid (UCM), Madrid, Spain. He is currently working toward the Ph.D. degree in CIEMAT, Madrid, where he is involved in several international R&D projects. His research interests include the application optimization for Grid computing, scheduling, and middleware development.

Bruce D. Scott received the Ph.D. degree from the University of Maryland, College Park, in 1985. Since 1988, he has been with the Max-Planck-Institut für Plasmaphysik, Garching, Germany. He is active in the field of magnetized plasma physics for over 25 years, with over 60 published papers.

Tiago T. Ribeiro received the Ph.D. degree from the Instituto Superior Técnico, Universidade Técnica de Lisboa, Lisbon, Portugal, in 2005. He entered the field of laboratory plasma research almost ten years ago, doing experimental work on two major tokamaks: JET in U.K. and ASDEX Upgrade in Germany. He then continued into the field of theoretical plasma research. Since then, he has been with the Max-Planck-Institut für Plasmaphysik, Garching, Germany, where he has conducted, in strong collaboration with the Instituto de Plasmas e Fusão Nuclear, Lisbon, his research work on the topic of turbulence in magnetized plasmas.

2118

Francisco Castejón Magaña received the Ph.D. degree in physics from the Universidad Complutense de Madrid (UCM), Madrid, Spain. He was with Cadarache, France, in 1988, and with KhPTI, Kharkov, Ukraine, in 1992. Presently, he is with the National Fusion Laboratory, CIEMAT, Madrid, Spain, and is the Head of the Plasma Theory Unit and the Leader of the fusion area in EGEE-III. He has published 90 papers in international scientific journals and more than 200 presentations to the International Congresses in Plasma Physics and Controlled Fusion. He has acted as the Supervisor of five Ph.D. students. Dr. Castejón Magaña has participated in three expert ad hoc groups and is currently a member of the Steering Committee of the European Fusion Development Agreement.


Rafael Mayo García received the Ph.D. degree in physics from the Universidad Complutense de Madrid (UCM), Madrid, Spain, in 2004. He was with the UCM as a Researcher in experimental and computational plasma physics. Since 2005, he has been with the ICT Division on Supercomputing and Grid Developments, CIEMAT, Madrid. He is the author of more than 60 publications and conference presentations and has been involved in 20 international and national R&D projects where he has also organized managerial and coordinating activities. Dr. Mayo García belongs to the Spanish Royal Society of Physics and the Latin American Bioinformatics Society. He has obtained several postdoctoral fellowships such as the Marie Curie and Juan de la Cierva Grants.