Towards automated code parallelization through ... - Semantic Scholar

Procs. 3rd Workshop on Program Comprehension , November 14-15, 1994, Washington D.C., USA

Towards automated code parallelization through program comprehension B. Di Martino and G. Iannello Dipartimento di Informatica e Sistemistica Universita di Napoli v. Claudio, 21 { 80125 Napoli fdimartin,[email protected]

TR-CPS-020-94 June 1994

Abstract

Currently available parallelizing tools are biased in favor of a particular parallel execution model for generating the output parallel program. This obviously limits the generality of these tools, since in general programs may be parallelized according to dierent programming paradigms. In this paper we propose a novel approach to automated code parallelization that tries to overcome these limitations. This approach consists in recognizing rst the paradigm that is best suited to a given program to be parallelized, and then applying paradigm-speci c transformation to generate the nal parallel code. We argue that the recognition phase can be fully automatized using techniques developed in the framework of automated program understanding. With the help of a case study, we discuss how this new approach could be implemented and propose the basic structure of a paradigm-oriented parallelizer.

1 Introduction Parallelization of sequential code by (semi) automatic tools turns out to be of interest for many reasons: 



there is a lot of sequential programs implementing potentially parallelizable algorithms and it would not be cost eective to recode these applications in a new parallel language; most users do not want to (and perhaps neither can) code new applications using an explicit parallel model of computation; hence even when an application has to be developed from scratch, an approach that allows the derivation of a parallel program from a previous sequential version would be better.

So far, the focus of research on computer-aided parallelization has been on computational science and virtually all available tools are oriented to FORTRAN as source language. The main goal of these tools is to make explicit the parallelism inherent to iterative control 1

structures, through a sophisticated data- ow and dependence analysis [8, 12, 14]. During the parallelization process the user can be requested to interact with the parallelizer to con rm and possibly correct the results of the dependence analysis during the analysis phase, and to decide how data structures and code must be distributed or replicated on the target parallel machine during the parallelization phase. Currently available tools are all biased to a particular parallel execution model referred to in the literature as the SPMD (single program multiple data) model [6], allowing parallel code to be generated relatively simply by (semi) automatic tools. On the other hand, several authors have recently argued that parallel programs can be classi ed according to their algorithmic skeleton [4] or programming paradigm [3], i.e. to the way processes making up the parallel program are created, synchronize and communicate, abstracting from the details of their sequential code. According to this view, the SPMD model can be viewed as a family of paradigms, all characterized by common synchronization and communication patterns. Unfortunately, the use of a xed target paradigm, even one very exible and quite general like the SPMD model, prevents from choosing the parallelization strategy that is best suited to the characteristic of the algorithm to be parallelized. Consequently, algorithms that cannot be transformed in an ecient SPMD program must be parallelized by hand or with very poor automatic support. The main reason currently available parallelizers cannot choose among dierent parallelization strategies is that they perform a purely structural analysis of the sequential code. Conversely, knowledge about the most convenient parallelization strategy for a speci c class of algorithms could be applied only if parallelizers were able to \recognize", at a proper abstraction level, what the sequential code does. In other words the program should be analyzed for discovering abstract concepts and assinging them to their realizations within the program itself. This problem has been referred to in the literature as the concept assigning problem [2]. While the concept assigning problem does not seem to be completely automatable in its general form, it could be fully automated if the recognition focused chie y on programming-oriented concepts, such as searches, sorts, structure transformations, numerical integration, etc. For instance, in [9, 11], program understanding is viewed as a parsing process that looks for speci c signatures representing concepts in the target program. The recognizer program uses a nite set of pattern templates (called cliches in [11]) that identify the concept signatures in a parsing process in which the less abstract concepts are recognized rst, and then they become features of larger-grained composite concepts. These results appear to be especially interesting with respect to parallelization since the abstract concepts that should be recognized in order to apply eective parallelization strategies can be classi ed as programming-oriented and are of the same kind as those recognized by the tool described in [11]. Moreover, a feature of parsing-based recognizers that could turn out to be particularly useful in the parallelization process is the hierarchy of concepts generated during recognition. If the links between increasingly abstract concepts of the hierarchy and sequential code implementing them are properly represented, sequential code restructuring needed for parallel code generation will be greatly simpli ed. In this paper, we develop these ideas and propose a parallelization procedure based on automated program understanding that can overcome limitations of existing parallelizers. After a brief introduction to currently available parallelizers, we outline the main limi2

tations of the structural approach taken by them and contrast it with a parallelization process based on a multitude of parallel programming paradigms. We then propose, with the help of a case study, a parallelization procedure that rst tries to recognize those algorithmic properties of the sequential program concerning parallelization, then chooses the best suited parallelization strategy (the paradigm ), and eventually generates the output parallel program starting from the input (sequential) code, from a skeleton code associated with the chosen paradigm, and from other output information produced during the recognition phase. Finally, the basic architecture of a tool implementing this parallelization procedure is presented and available techniques for its realization are discussed.

2 Tools for automated parallelization Parallelizers are usually made up by a front end , that is substantially independent of the target parallel architecture, and a back end that is architecture-dependent and perform actual code generation [14]. The front end translates the program text in an internal representation suitable for program manipulation and performs code normalization . The front end includes syntactic and semantic analysis of the program text, followed by a control- ow, a data- ow and a data-dependence analysis to construct the program dependence graph and the call graph. Some standard code transformations such as the constant propagation and the dead code elimination can be applied during this phase. Code normalization includes other transformations on loops, on other control structures and on subscripts of arrays, in order to represent the program in a standard form that simpli es subsequent phases. The main goal of the back end is the generation of a parallel version of the program. This process relies on a catalog of transformations and on a strategy for their application which takes into account the results produced by the front end. These transformations can be classi ed according to the nature of the code they can be applied to. There are therefore transformations for acyclic code, DO loops, WHILE loops, recursive constructs and code manipulating pointers. Finally, there are transformations that postpone at run-time the decision of which constructs are to be parallelized. The strategy controlling the application of these transformations obviously depends also on the architectural characteristics of the target machine and on the cost of its basic operations. Consequently, most parallelizers also rely on a quantitative characterization of the underlying parallel machine and use this characterization to select transformations when dierent alternatives are possible [8]. Since parallelization is so dependent on the target architecture, in our discussion we focus on distributed memory computers. These machines, also referred to as multicomputers in the literature [1], consist essentially of a number of autonomous processing units (nodes ), each with a private local memory, connected by a message-passing network. The intrinsic scalability of this organization and its very favorable cost/performance ratio with respect to other alternatives make multicomputers the best candidate for future high performance machines and motivate the increasing attention which is being paid to improve their usability. Among all the techniques mentioned above, transformations for DO loops are by far the most suited for parallelization in a distributed memory framework. Indeed, DO loops are the main source of coarse-grained parallelism, and this kind of parallelism is generally the most convenient to be exploited on multicomputers today, since remote interactions 3

still have a much higher cost than access to local data. When only parallelization of DO loops is performed, code generation is greatly simpli ed if a Single Program Multiple Data (SPMD) execution model is chosen for the output (parallel) program. According to the SPMD model [6], a parallel program consists of a number of processes started together and running in parallel. All processes execute the same sequential code that is divided into sections . All processes synchronize between subsequent sections. Sections can be serial , parallel or replicated , corresponding to code executed by only one process, executed by all processes on dierent data and executed by all processes on the same data, respectively. In the rst case all processes but one remain idle till the next synchronization point is reached. In the second case processes are not constrained to execute the same code, but can switch to dierent code segments depending on local state. In the latter case the same work is done by all processes in order to save costly communication and synchronization. Provided that loop iterations are not constrained to be executed serially, in the SPMD framework parallelization of DO loops is easily performed by executing the loop in a parallel section and by distributing loop iterations among the processes. Since in parallel sections loop iterations assigned to dierent processors operate on different data (typically elements of arrays), in a multicomputer environment a further data distribution problem arises from loop iteration distribution. Actually, data distribution and loop iteration distribution are tightly related problems and the solution for one of them implicitly induces a solution for the other [14]. Unfortunately, available parallelizers cannot solve the problem in the general case and user intervention is generally requested to perform either data distribution or loop iteration distribution. In particular, since data distribution aects both where computation takes place and which communications are performed, most parallelization tools leave to the user the responsibility of specifying it. Then they parallelize the code taking into account information provided by the user as well as the results of the data-dependence analysis automatically performed by the front end. From this brief overview on parallelization tools, it is apparent that they basically perform a structural analysis of the program to identify which loops can be distributed among the processors and which cannot. Since the nal program is conforming to the SPMD model, the code produced consists of a single sequential program obtained by applying to the original program text local transformations such as: insertion of synchronization points between sections, conditional execution of code corresponding to serial sections, loop tiling to insert intra-loop communications, etc. Consequently, the parallelization process can be applied only to those computations for which an ecient data parallel version can be written. It is worth noting that not only automatic parallelization of algorithms that do not t the data parallel pattern, but also the application to irregular data parallel computations of sophisticated parallelization strategies that would exhibit better performance than SPMD is prevented [7]. These considerations lead to the conclusion that we should do away with the constraint represented by the SPMD, as the only model output program have to conform to, if we want to improve the performance of parallelization tools. The rst step in this direction would be to determine alternative execution models to SPMD that could be integrated in an automatic parallel code generator. In our opinion, such alternatives do exist and they have been pointed out in the recent literature on parallel programming techniques, though in contexts other than automatic parallelization. In fact, more than a decade in parallel program development has shown 4

that the \parallel structure" of most programs can be classi ed according to a limited number of patterns, that can be viewed as representative of the basic ways to organize a parallel computation. This idea has been explicitly expressed in the context of parallel programming methodologies [3, 4] and parallel programming languages [5]. Even though the notion of \parallel structure" of a program has not been well formalized yet, it can be informally de ned as the way processes forming the parallel program are created, synchronize and communicate, abstracting from the details of their sequential part . Following [3], we will refer to this notion of parallel structure of a program with the term Parallel Programming Paradigm , or more brie y Programming Paradigm (PP). From this informal de nition it is apparent that a PP is just a template, parametrically de ned with respect to the actual number of processes instantiated, the number of similar interactions between them during the computation, the length of messages exchanged, etc. In this respect, the SPMD model can be considered as a PP, or, more speci cally, as a family of PPs, since synchronization and communication patterns of SPMD programs may slightly dier within the constraint that all processes must participate to these events. The availability of dierent PPs can be used to extend the capabilities of automatic parallelization tools. The idea is that parallelizers should be able to recognize which paradigm (or paradigms) are best suited for parallelization of a given sequential algorithm and to apply paradigm-oriented transformations to derive the nal parallel program. Assuming that a catalog of PPs is available, to make eective the paradigm-oriented approach to parallelization just outlined, we have to solve essentially two problems: (a) how the best PP is selected and (b) how transformations are selected and applied to generate parallel code. We believe that the solution to these problems involves some kind of program comprehension, possibly integrated with traditional parallelization techniques based on structural analysis. In particular, automated program understanding techniques developed for maintaining sequential code appear to be especially suited for this task. In the following sections, through the presentation of case studies, we will further develop these ideas and we will informally propose a paradigm-oriented parallelization procedure amenable to being fully integrated in an automatic tool.

3 Parallelization using programming paradigms In this section we discuss the parallelization process for two case studies that does not t the SPMD model of parallel execution, in order to better understand how this process can be automated. We have chosen two programs that have the same basic structure, but dier in minor aspects that are otherwise relevant with respect to parallelization. We made this choice both to stress the recognition capabilities that a paradigm-oriented parallelizer should possess and to analyze how dierent parallel programs can be derived starting from structurally similar sequential code. Both programs have been coded by students with moderate programming experience, without having in mind any concerns about parallelization. The rst program, coded in Pascal, implements the iterative version of the quicksort algorithm and includes the inputting of the list to be sorted and the outputting of the sorted list. Figure 1a shows the complete program, with subprograms entirely macro expanded by hand. The second program, coded in C, implements binary branch-andbound search, and includes inputing of initial data and outputting of the nal result. Figure 1b shows the complete program, that did not contain subprograms. 5

Program Quicksort; CONST limit =50; TYPE parola = string[15]; data = RECORD Key : integer; name : parola; surname : parola; vote : integer; END; stackpointer = ^stacknode; stacknode = RECORD left : integer; stack right : integer; next : stackpointer; END; VAR i,j : integer; ii,jj,iii : integer; left,right : integer; M,n : integer; problem notempty : boolean; R : array[0..limit+1] of data; top : stackpointer; head of stack flag : boolean; K : integer; RR : data; node,temp : stackpointer; BEGIN {of main program} write(’inserisci numero dati e passo:’); readln(n,M); R[0].key:=0; R[n+1].key:=10000; writeln(’inserisci matricola nome cognome voto’); FOR iii:=1 to n DO WITH R[iii] DO BEGIN readln(key); readln(name); readln(surname); readln(vote); END; top :=nil; initialize "stack" to empty left :=1; initialize "problem" to "initial problem" right:=n; notempty :=true; WHILE notempty DO repeat until "stack" is empty BEGIN "problem" is not "trivial problem" WHILE ((right-left)>=M) DO BEGIN i:=left; j:=right; K:=R[left].key; RR:=R[left]; flag:=true; WHILE flag DO BEGIN WHILE K < R[j].key DO j:=j-1; IF j