Dynamic Distributed Data in a Parallel Programming ... - CiteSeerX

Dynamic Distributed Data in a Parallel Programming Environment DDD { Reference Manual Rev. 1.0 Klaus Birken

Rechenzentrum der Universitat Stuttgart Allmandring 30, 70550 Stuttgart E-mail: [email protected]

Forschungs- und Entwicklungsberichte

September 94

RUS-23

Abstract The parallel implementation of algorithms based on dynamic data structures on architectures with distributed memory access leads to various technical problems as for data redundancy, consistency and load balancing. The DDD library allows easier development of those dynamic parallel algorithms. This document contains the complete speci cation of the DDD library interface, including a detailed reference manual, the memory manager speci cation and a list of DDD errors.

Chapter 1 Introduction DDD (short for dynamic distributed data) is a library for implementing parallel programs based on algorithms which use dynamic data structures on MIMD computers. It is based on a corresponding formal model of distributed data structures, which assists the application programmer in designing the distributed structure as well as in developing a correct data parallelization method (cf. [3]). Most of the technical problems of parallel implementations on distributed memory machines (cf. [5]) can be tackled with DDD: managing object redundancy, de ning consistency protocols and changing the topology of the distributed data structure. In many parallel applications (e.g. in the computational uid dynamics eld) the overall eciency of the data-parallel algorithm results from a trade-o between data consistency and convergence rate; the DDD library supports consistency protocols governed by numerical demands (lean consistency, [2]). The main goals of DDD are eciency in runtime and memory, ease of use and portability across many platforms. To reach the last goal the current implementation is based on a general parallel processing interface (PPIF [1]). Concerning the rst two goals see [2]. Load balancing of data structures has been integrated into DDD only as far as managerial aspects are concerned. Well-known load balancing methods (e.g. recursive spectral bisection [6]) may be integrated in parallel application programs based on DDD easily; recent developmentof general partitioning tools (e.g. Chaco, [4]) made integrating DDD and the necessary load balancing functionality on a modular basis even more simple. 1

2

CHAPTER 1. INTRODUCTION

The current DDD implementation has been programmed in C and oers interfaces for C and C++ programs. All allocation actions of dynamic memory are mapped onto calls of some memory manager (MEMMGR) to enhance eciency. This manual contains the complete DDD reference manual with detailed speci cation of all DDD functions, a speci cation of the MEMMGR and a list of DDD errors with some explanations and hints.

Chapter 2 Reference Manual On the following pages all DDD functions are listed in a case-insensitive, alphabetical order with a detailed description of their usage and actions. Most of them are included in the DDD library, some of them are implemented via the standard C macro mechanism and may be replaced by inline-functions in a C++ context. The C header le ddd.h contains all necessary constants, macros and function prototypes for using the DDD library. The basic model of parallel execution for DDD applications is the SPMD (single program multiple data) paradigm, therefore two classes of routines exist:

Global routines: These routines have to be called in all current processes of

the parallel program. Usually they imply a communication procedure on the current processor topology. Local routines: These routines may be called after DDD initialization by each process independently. Usually they don't imply any communication procedure.

All global routines are marked as global in the upper right of their description page.

3

4

CHAPTER 2. REFERENCE MANUAL

DDD ConsCheck

global

Synopsis void

DDD ConsCheck

()

Return Value No value is returned.

Description of Parameters No parameters.

Action This routine performs a combined local/global consistency check on the object data structures managed by DDD. This may be used for debugging purposes; if errors are detected, then some understanding of internal DDD structures will be useful. The following single aspects will be checked:

double existence of global id numbers in all local object sets consistency of coupling lists and object copies

5

DDD DisplayIF

Synopsis void

DDD DisplayIF

()



Action This routine displays an overview table containing all DDD-interfaces, their de nition parameters and the current number of constituing objects on the calling processor. For each interface and each neighbour processor corresponding to that interface a relation line is displayed, which contains the overall number of objects inside the interface, the number of oneway relations outwards, the number of oneway relations inwards, the number of exchange relations and the neighbour processor number.

6


DDD Exit

global

Synopsis void

DDD Exit

()



Action This routine frees memory previously allocated by DDD and nally nishes up the PPIF library. After the call to DDD Exit() further usage of the DDD library is no longer possible during this program run. The clean-up of the memory manager should happen afterwards and is left to the DDD application programmer.

7

DDD HandlerRegister

Synopsis void

DDD HandlerRegister (type id, h type, h adr, ..., HANDLER END)


Description of Parameters DDD TYPE type id

Id of DDD object type for which a list of handlers should be registered.

HandlerType h type

Type id of handler to be registered.

HandlerPtr h adr

Memory address of user-supplied handler routine.

The last two parameters may be repeated for all handlers to be registered; the last parameter has to be the constant HANDLER END to indicate the end of the variable argument list.

Action This routine registers a list of handlers for a certain DDD object type. The list may contain each handler type HANDLER as listed in this manual. Usually DDD HandlerRegister() will be called once after initialization of the corresponding DDD TYPE.

8


DDD IdentifyBegin

global

Synopsis void

DDD IdentifyBegin

()



Action A call to this routine establishes a global identi cation operation. It should be issued on all processors. After this call an arbitrary series of Identifycommands may be issued. The global identi cation operation is carried out via a DDD IdentifyEnd() call on each processor.

9

DDD IdentifyEnd

global

Synopsis void

DDD IdentifyEnd

()



Action This routine starts the object identi cation process. After a call to this routine (on all processors) all Identify-commands since the last call to DDD IdentifyBegin() are executed. This involves a set of local communications between the processors.

10


DDD IdentifyNumber

Synopsis void

DDD IdentifyNumber (obj, proc, ident)


Description of Parameters OBJECT obj

DDD local object which has to be identi ed with another object.

PROC ID proc

Owner (processor number) of corresponding local object. This processor has to issue a corresponding call to this DDD Identify-routine.

int ident

Identi cation value. This is an arbitrary number to identify two corresponding operations on dierent processors.

Action After an initial call to DDD IdentifyBegin(), this routine identi es two object copies on separate processors. It has to be called on both processors with the same identi cation value. The necessary actions (e.g. message transfer) are executed via the nal call to DDD IdentifyEnd(); therefore a whole set of DDD Identify-operations is accumulated. After the identi cation both objects have the same DDD global object id, which is build using the minimum of both local object ids.

11 The identi cation speci ed here may be detailed even further by additional calls to DDD Identify-operations with the same local object.

12


DDD IdentifyObject

Synopsis void

DDD IdentifyObject (obj, proc, ident)




PROC ID proc


OBJECT ident

Identi cation object. This is an arbitrary global object which is known to both processors involved to identify the two corresponding operations on these processors.

Action After an initial call to DDD IdentifyBegin(), this routine identi es two object copies on separate processors. It has to be called on both processors with the same identi cation object. The necessary actions (e.g. message transfer) are executed via the nal call to DDD IdentifyEnd(); therefore a whole set of DDD Identify-operations is accumulated.

13 After the identi cation both objects have the same DDD global object id, which is build using the minimum of both local object ids. The identi cation speci ed here may be detailed even further by additional calls to DDD Identify-operations with the same local object.

14


DDD IdentifyString

Synopsis void

DDD IdentifyString (obj, proc, ident)




PROC ID proc


char * ident

Identi cation string. This is an arbitrary string to identify two corresponding operations on dierent processors.

Action After an initial call to DDD IdentifyBegin(), this routine identi es two object copies on separate processors. It has to be called on both processors with the same identi cation string. The necessary actions (e.g. message transfer) are executed via the nal call to DDD IdentifyEnd(); therefore a whole set of DDD Identify-operations is accumulated. After the identi cation both objects have the same DDD global object id, which is build using the minimum of both local object ids.

15 The identi cation speci ed here may be detailed even further by additional calls to DDD Identify-operations with the same local object.

16


DDD IFDe ne

global

Synopsis INTERFACE

DDD IFDe ne (lenO, O[ ], lenA, A[ ], lenB, B[ ])

Return Value Id of DDD interface

Description of Parameters int lenO

Number of entries in array O[ ] (as speci ed with second argument).

DDD TYPE O[ ]

Array of DDD object type ids; the new interface will only contain objects with one of these types.

int lenA

Number of entries in array A[ ] (as speci ed with forth argument).

int A[ ]

First array of DDD priorities; the object set A from the new interface will only contain objects with one of these priorities.

int lenB

Number of entries in array B[ ] (as speci ed with sixth argument).

17

int B[ ]

Second array of DDD priorities; the object set B from the new interface will only contain objects with one of these priorities.

Action This routine de nes a new DDD interface. Its argument list contains three arrays: the rst one speci es a subset of the global DDD object set, the second and third ones specify a subset of all DDD priorities. After the initial creation of the new interface its id is returned. During all following DDD operations (Identify- as well as Transferoperations) the interface will be kept consistent, and can be used for communication via DDD IFExchange() and DDD IFOneway().

18


DDD IFExchange

global

Synopsis void

DDD IFExchange (if id, item size, gather proc, scatter proc)


Description of Parameters INTERFACE if id

DDD interface id, specifying the interface which has to be used during the communication.

int item size

Size (number of bytes) of data portion for each object contained in interface.

ComProcPtr gather proc

User-de ned routine for gathering data which will be transfered across the speci ed interface.

ComProcPtr scatter proc

User-de ned routine for scattering data which had been transfered across the speci ed interface.

Action This routine initiates a communication across a previously de ned DDD interface. It has to be issued on all processors.

19 The communication is carried out in both directions, i.e. from object set A to object set B and vice versa (see DDD IFDe ne()). The user-de ned routine gather proc is called for each object inside the interface. It has to pack all necessary information into a special memory location. Afterwards the actual communication based on the current processor topology happens. After all data had been scheduled onto the destination processors, it has to be incorporated into the destination part of the DDD interface via the user-de ned routine scatter proc. Both communication procedures should have the following declaration: int (*ComProcPtr) (OBJECT obj, void *data)

20


DDD IFOneway

global

Synopsis void

DDD IFOneway (if id, dir, item size, gather proc, scatter proc)


Description of Parameters INTERFACE if id

DDD interface id, specifying the interface which has to be used during the communication.

IFDirection dir

The communication direction, one of f IF FORWARD, IF BACKWARD g.

int item size

Size (number of bytes) of data portion for each object contained in interface.

ComProcPtr gather proc

User-de ned routine for gathering data which will be transfered across the speci ed interface.

ComProcPtr scatter proc

User-de ned routine for scattering data which had been transfered across the speci ed interface.

21

Action This routine initiates a communication across a previously de ned DDD interface. It has to be issued on all processors. The communication is carried out one-directional, dependent on parameter dir. In case of IF FORWARD information will ow from object set A to object set B, in case of IF BACKWARD it will be vice versa (see DDD IFDe ne()). The user-de ned routine gather proc is called for each object inside the interface. It has to pack all necessary information into a special memory location. Afterwards the actual communication based on the current processor topology happens. After all data had been scheduled onto the destination processors, it has to be incorporated into the destination part of the DDD interface via the user-de ned routine scatter proc. Both communication procedures should have the following declaration: int (*ComProcPtr) (OBJECT obj, void *data)

22


DDD InfoGlobalId

Synopsis int

DDD InfoGlobalId (obj)

Return Value Global id of a local DDD object.


Pointer to a DDD object as returned by a DDD ObjGet() call.

Action This routine returns the global id of the local DDD object.

23

DDD InfoPriority

Synopsis int

DDD InfoPriority (obj)

Return Value Priority of a local DDD object.



Action This routine returns the priority of the local DDD object.

24


DDD InfoProcList

Synopsis int *

DDD InfoProcList (obj)

Return Value Array with information about object copies on other processors.



Action This routine returns a list of existing copies of one local object. The list is implemented as an integer array of length n and is organized like this: id of calling processor priority of local copy id of processor which holds a copy priority of that copy last two elds repeated int[n-1] -1 (ending mark) int[0] int[1] int[2] int[3]

:::

The storage required for the array is made available by DDD and will be valid until the next call to a DDD function.

25

DDD InfoRange

Synopsis int

DDD InfoRange (obj)

Return Value Range of a local DDD object.



Action This routine returns the range of the local DDD object.

26


DDD Init

global

Synopsis void

DDD Init (argc, argv)


Description of Parameters int argc

Number of command line arguments, same as in main(argc, argv).

char * argv[ ]

Vector of command line argument strings, same as in main(argc, argv).

Action This routine initializes the DDD library. It has to be called before any other function of the DDD library is called. This will initialize the underlying PPIF library, too. As some of the memory handler calls will be initiated during the execution of this routine, the memory manager has to be initialized before calling DDD Init().

27

DDD ListLocalObjects

Synopsis void

DDD ListLocalObjects

()



Action This routine displays a list of all local DDD objects with all data inside the DDD HEADER managed by DDD. The objects are sorted according to their global object id. The lists produced on each processor may serve as a debugging aid for small sized test examples.

28


DDD ObjDelete

Synopsis void

DDD ObjDelete (obj)




Action This routine destroys a DDD object. First internal DDD structures are released, then the optional handler HANDLER OBJDELETE is called. At last the object memory is freed via calling the user-supplied memory manager routine memmgr FreeOMEM(). If foreign copies of the object were registered before, then all necessary local coupling information is also being deleted by calls to memmgr FreeAMem().

29

DDD ObjGet

Synopsis void *

DDD ObjGet (type id)

Return Value This routine dynamically creates a new DDD object and returns a pointer to it. This pointer has to be casted to its particular object type from the application program; therefore a dummy type is used inside DDD.


Type id of newly created DDD object.

Action This routine dynamically creates a new DDD object. The user-supplied memory manager routine memmgr AllocOMEM() is called to allocate the necessary memory; afterwards the new object is properly initialized to t into the DDD management. An unique global object id is reserved and assigned to the new object. Finally the optional handler HANDLER OBJINIT is called.

30


DDD ObjMoveTo

Synopsis void

DDD ObjMoveTo (obj, new loc)




char * new loc

New object location in local memory.

Action This routine moves a DDD object to a new memory location. All links onto that object from the DDD management are changed and the object's memory is copied to the new location as a whole.

31

DDD ObjUnlink

Synopsis void

DDD ObjUnlink (obj)




Action This routine destroys a DDD object without removing the object's memory. It releases all internal DDD structures and returns control to the application program. Neither the handler HANDLER OBJDELETE nor the memory manager are called (cf. DDD ObjDelete()). If foreign copies of the object were registered before, then all necessary local coupling information is also being deleted by calls to memmgr FreeAMem().

32


DDD PrioritySet

Synopsis void

DDD PrioritySet (obj, prio)




int prio

New priority for local copy of DDD object.

Action This routine changes the priority of a local DDD object.

33

DDD RangeSet

Synopsis void

DDD RangeSet (obj, range)




int range

New range for this DDD object.

Action This routine changes the range of a global DDD object.

34


DDD StructDisplay

Synopsis void

DDD StructDisplay (type id)



Id of DDD object type for which structure information should be displayed.

Action Some information about the memory layout and internal structure of a DDD object type will be displayed.

35

DDD StructRegister

global

Synopsis DDD TYPE

DDD StructRegister (name, hdr, obj,

:::

, EL END, next)

Return Value id of DDD object type

Description of Parameters char * name

Name of new DDD object type. Used for debugging and listing.

OBJ HEADER * hdr

Memory address of DDD header in dummy object. This parameter may be NULL, which registers a non-DDD object (additional data object). Refer to DDD XferAddData() for more information on additional data objects.

OBJECT obj

Memory address of dummy object.

ElemType element type

DDD element type, has to be one of: f EL GDATA, EL LDATA, EL DATAPTR, EL OBJPTR g.

char * element adr

Memory address of one element contained in the dummy object structure.

36


int pointer count (optional!)

When element type is one of f EL DATAPTR, EL OBJPTR g, this parameter counts the number of pointers contained in that element. Otherwise this parameter has to be left out.

The parameters element type, element adr and (optionally) pointer count may be repeated arbitrary often, but in xed order. The last-but-one parameter has to be the constant value EL END to indicate the ending of the variable sized list.

OBJECT next

Memory address of next dummy object. This is equivalent to the rst free memory location behind the dummy object, while taking into account the current compiler alignment strategy.

Action All objects used in a distributed manner have to be registered via this DDD routine. Its purpose is to get a rough description of the memory structure of a certain object type. Lateron all objects of that type will be referred to with their DDD object type id, which will be returned by a call to this function. The recommended way to use this routine is to declare a dummy object of the particular type at rst, and then reference this dummy object during the call to DDD StructRegister. This method allows some fault-tolerance concerning some implementation aspects due to the registration of all structure elements at runtime. After registration the dummy object may be deleted. Aside from a textual description of that object type and the dummy object's start- and end-address the parameterlist contains a variable part, which is simply a list of all element types contained in that object type.

37

DDD XferAddData

Synopsis void

DDD XferAddData (cnt, type id)


Description of Parameters int cnt

Number of data objects in array (i.e. array length).

DDD TYPE type id

Type of data objects. All data objects inside one array should have the same object type. This object type is de ned by registering the object structure via DDD StructRegister() as usual, but without including the DDD object header.

Action This routine transfers an array of additional data objects corresponding to one DDD object to the same destination processor. Therefore the latest call to DDD XferCopyObj() de nes the primary object and the destination. An arbitrary number of DDD XferAddData()-calls may be issued to transfer various data object arrays at once. This serves as a mechanism to send object references which cannot be handled by the native DDD pointer conversion technique. Just before the actual Xfer-operation happens, the (optional) handler HANDLER XFERGATHER will be called to ll the data object array into some reserved storage space. After the Xfer-operation the handler HANDLER XFERSCATTER is called on the receiving processor to rebuild the data objects.

38


As the data objects had to be registered and equipped with a usual type id (as with the DDD objects), the standard DDD pointer conversion will also take place for the data objects.

39

DDD XferBegin

global

Synopsis void

DDD XferBegin

()



Action A call to this routine establishes a global transfer operation. It should be issued on all processors. After this call an arbitrary series of Xfer-commands may be issued. The global transfer operation is carried out via a DDD XferEnd() call on each processor.

40


DDD XferCopyObj

Synopsis void

DDD XferCopyObj (obj, proc, prio)



DDD object which has to be copied into another processors memory.

PROC ID proc

Destination processor which will receive the object copy

int prio

DDD priority of new object copy

Action After an initial call to DDD XferBegin(), this routine creates a copy of one local DDD object on another processor with a certain priority. The necessary actions (packing/unpacking of object data, message transfer) are executed via the nal call to DDD XferEnd(); therefore a whole set of DDD Xfer-operations is accumulated. Caution: As the original object data is not copied throughout this call due to eciency reasons (transfering a large number of objects would result in a huge amount of memory copy operations), the object may not be changed or deleted until the actual transfer has happened.

41 Two dierent mechanisms allow the transfer of data depending on the stated object:

Right after the routine DDD XferCopyObj() has been called, the optional handler HANDLER XFERCOPY is executed by DDD. Basically this mechanism allows the transfer of dependent DDD objects (or: hierarchies of objects), although arbitrary actions may occur inside the handler. No relationship between the primary object and the additional objects can be expressed, as many dierent destination processors might be involved. Via an arbitrary number of additional calls to DDD XferAddData() arrays of data objects (i.e. without the usual DDD object header) may be sent. Using this mechanism the data objects are strongly linked to the primary object; all data objects are transferred to the same destination processor.

After the object copy has been established on the destination processor, rst the optional handler HANDLER OBJUPDATE is called to update locally invalid parts of that object (e.g. locally linked lists). Afterwards the optional handler HANDLER OBJMKCONS is called to establish consistency for that object (e.g. backward references on the new object copy). For handlers related to routine DDD XferAddData() refer to its description.

42


DDD XferDeleteObj

Synopsis void

DDD XferDeleteObj (obj)



Local DDD object which should be deleted.

Action This routine deletes a local DDD object. This routine is regarded as a

Xfer-operation due to its in uence on DDD management information on neigh-

bouring processors. Therefore the routine has to be issued between a starting DDD XferBegin() and a nal DDD XferEnd() call. During the actual Xfer-operation all data corresponding to that object and the object memory itself will be deleted.

43

DDD XferEnd

global

Synopsis void

DDD XferEnd

()



Action This routine starts the object transfer process. After a call to this routine (on all processors) all Xfer-commands since the last call to DDD XferBegin() are executed. This involves a set of local communications between the processors.

44


HANDLER COPYMANIP

HANDLER COPYMANIP is a DDD constant describing one user-supplied routine (a handler), which happens to be called by DDD during runtime. In the following this handler routine is called Handler CopyManip.

Synopsis void

Handler CopyManip (copy)


Description of Parameters OBJECT copy

DDD object copy inside Xfer-message for which handler had been called.

Action This handler is called just after one object copy has arrived on the destination processor. The new object had not been established up to this point, therefore the parameter for this handler is the copy of the object's memory inside the message. This handler allows to manipulate data very deep inside the system, so usage shouldn't be necessary in most applications and should be avoided as far as possible.

45

HANDLER OBJDELETE

HANDLER OBJDELETE is a DDD constant describing one user-supplied routine (a handler), which happens to be called by DDD during runtime. In the following this handler routine is called Handler ObjDelete.

Synopsis void

Handler ObjDelete (obj)



DDD object for which handler had been called.

Action This handler is called just before an object is removed from local memory (its the rst action in DDD ObjDelete). It is the counterpart of HANDLER OBJINIT. This handler may be used e.g. to remove the object from local object lists or to remove references to this object right before its own deletion.

46


HANDLER OBJINIT

HANDLER OBJINIT is a DDD constant describing one user-supplied routine (a handler), which happens to be called by DDD during runtime. In the following this handler routine is called Handler ObjInit.

Synopsis void

Handler ObjInit (obj)




Action This handler is called right after an object has been created in local memory for the rst time (during explicit or implicit calls to DDD ObjGet()). This handler may be used e.g. to init local variables and link the new object into local object lists.

47

HANDLER OBJMKCONS

HANDLER OBJMKCONS is a DDD constant describing one user-supplied routine (a handler), which happens to be called by DDD during runtime. In the following this handler routine is called Handler ObjMkCons.

Synopsis void

Handler ObjMkCons (obj)




Action During Xfer-operations new local objects may be created. The references from the new object onto other previously existing objects can be handled by DDD, but backward references pointing at this new object cannot be converted/inserted automatically. This handler can be used to keep the data structure consistent, it has to ensure that all references pointing at the given DDD object are actualized.

48


HANDLER OBJUPDATE

HANDLER OBJUPDATE is a DDD constant describing one user-supplied routine (a handler), which happens to be called by DDD during runtime. In the following this handler routine is called Handler ObjUpdate.

Synopsis void

Handler ObjUpdate (obj)




Action During Xfer-operations new local objects may be created. This handler is called after the objects had been established in local memory, but right before handler HANDLER OBJMKCONS is called. This handler may be used to update data, which is not valid on the destination processor after a Xfer-operation. For consistency updates one should use HANDLER OBJMKCONS, which will be called right after this handler for every transfered object.

49

HANDLER XFERCOPY

HANDLER XFERCOPY is a DDD constant describing one user-supplied routine (a handler), which happens to be called by DDD during runtime. In the following this handler routine is called Handler XferCopy.

Synopsis void

Handler XferCopy (obj, proc, prio)




PROC ID proc

Destination processor object will be copied to.

int prio

Priority of new local object.

Action This handler will be called right after a DDD XferCopyObj() call has been issued during Xfer. It will get the same parameters as this corresponding DDD XferCopyObj()-call.

50


The handler can be used to express dependencies in object transfer; the usual way to use this facility is to place new, dependent Xfer-calls to the same destination processor there.

51

HANDLER XFERDELETE

HANDLER XFERDELETE is a DDD constant describing one user-supplied routine (a handler), which happens to be called by DDD during runtime. In the following this handler routine is called Handler XferDelete.

Synopsis void

Handler XferDelete (obj)




Action This handler will be called right after a DDD XferDeleteObj() call has been issued during Xfer. It will get the same parameters as this corresponding DDD XferDeleteObj()-call. The handler can be used to express dependencies in object deletion; the usual way to use this facility is to place new, dependent XferDelete-calls there.

52


HANDLER XFERGATHER

HANDLER XFERGATHER is a DDD constant describing one user-supplied routine (a handler), which happens to be called by DDD during runtime. In the following this handler routine is called Handler XferGather.

Synopsis void

Handler XferGather (obj, cnt, type id, data)




int cnt


DDD TYPE type id

DDD type id of data objects.

void * data

Address of the memory location the data objects should be lled in.

53

Action This handler is called during the actual Xfer-phase and has to ll an array of data objects into a (given) empty memory buer. The handler is called once for each DDD XferAddData() operation that was issued for a given DDD XferCopyObj()-call. The several DDD XferAddData()-calls can be distinguished via the type id parameter. The data objects have to be aligned to memory locations according to the compiler speci cation.

54


HANDLER XFERSCATTER

HANDLER XFERSCATTER is a DDD constant describing one user-supplied routine (a handler), which happens to be called by DDD during runtime. In the following this handler routine is called Handler XferScatter.

Synopsis void

Handler XferScatter (obj, cnt, type id, data)




int cnt


DDD TYPE type id

DDD type id of data objects.

void * data

Address of the memory location the data objects can be restored from.

55

Action This handler is called during the actual Xfer-phase and has to restore an array of data objects out of a given memory buer. There is one call to Handler XferScatter() for each preceding call to Handler XferGather(). The data objects are aligned to memory locations according to the compiler speci cation.

56


Chapter 3 Memory Manager As DDD aims at managing dynamic data, dynamic memory allocation is a basic functionality being used by the DDD library. To avoid garbage collection (e.g. by the operating system) as far as possible and to enhance eciency all dynamic memory is conceptually divided up into dierent classes, depending on the duration of the memory's validity. A memory manager interface (MEMMGR) has been designed to specify the functionality which DDD will use to allocate and free memory of these dierent memory classes. DDD then uses MEMMGR as a memory management tool. The application programmer has to provide a MEMMGR implementation tting the MEMMGR interface de nition described below.

3.1 Memory Classes All DDD calls which allocate or free dynamic memory have been mapped onto four dierent classes:

PMEM - Permanent Memory

This memory class is conceptually equivalent to memory for global variables. PMEM is allocated during DDD Init() and will be used for global tables or permanent buer space. The size of PMEM depends on DDD global parameters and linearly on the number of processors.

OMEM - Object Memory

This memory class represents storage for all objects managed by DDD. 57

58

CHAPTER 3. MEMORY MANAGER Thus, its size changes whenever objects are created or deleted, e.g. during Xfer-operations.

AMEM - Administration Memory

This is storage for management of parallel data. AMEM is the major part of the parallel storage overhead, as it contains couplings and DDD interface data.

TMEM - Temporary Memory

TMEM is storage with the shortest lifetime. It is used for temporary object tables, sorting, message buering and many other temporary uses.

3.2 MEMMGR Interface For each of the four memory classes two functions have to be implemented: one memmgr Alloc()-function for getting some storage space and its counterpart memmgr Free() for returning control over that storage space back to the memory manager. The C header le memmgr.h speci es the MEMMGR interface, here is an excerpt containing the declaration of all necessary functions: void * memmgr_AllocPMEM (unsigned long size); void memmgr_FreePMEM (void *storage); void * memmgr_AllocOMEM (unsigned long size, DDD_TYPE id); void memmgr_FreeOMEM (void *storage, DDD_TYPE id); void * memmgr_AllocAMEM (unsigned long size); void memmgr_FreeAMEM (void *storage); void * memmgr_AllocTMEM (unsigned long size); void memmgr_FreeTMEM (void *storage);

The OMEM-handlers have to get the DDD type id of the new object as a parameter in order to classify equally sized objects across this handler.

3.3. THE STANDARD MEMMGR

59

3.3 The Standard MEMMGR The application programmer has to provide a MEMMGR implementation tting the MEMMGR interface de nition described above. For rapid program development one standard memory manager is supplied with DDD, which serves as a simple, but working default MEMMGR as well as an example for implementing a new, more sophisticated memory manager on one's own. In the current memmgr std-implementation all memmgr Alloc()-calls are mapped onto the standard malloc() call of the native C compiler; its counterparts memmgr Free() are mapped onto the standard free() call.

60

CHAPTER 3. MEMORY MANAGER

Chapter 4 Error List The following section gives an overview of errors which can occur during usage of the DDD library. The list has been sorted by error numbers and parted into errors from dierent modules of the library. The rst letter of the error number indicates the error level: letter represents W warning E "normal" error F fatal error

61

62

CHAPTER 4. ERROR LIST

4.1 General Module E1000:

not enough memory in DDD Init

E1150:

too many elements in DDD StructRegister

E1500:

not enough memory in TopoInit

Temporary buers couldn't be allocated during DDD initialization (using PMEM). This should never happen under normal conditions.

The hard-wired maximum of elements per DDD type id has been reached. Try restructuring the object de nitions by sorting elements according to the EL-types. A temporary buer couldn't be allocated during DDD initialization (using PMEM). This should never happen under normal conditions.

4.2. OBJECT MANAGER

63

4.2 Object Manager E2000:

no more objects in DDD ObjInit

E2100:

not enough memory in DDD ObjGet

E2120:

no more objects in DDD ObjGet

E2200:

not enough memory in ObjCreateCopy

E2220:

no more objects in ObjCreateCopy

E2500:

not enough memory in AddCoupling

E2520:

no more couplings in AddCoupling

W2600:

coupling already known in AddCoupling

The hard-wired maximum of objects has been reached. Try to use more processors (i.e. less objects per processor) or use a bigger DDD library.

Memory for DDD object couldn't be allocated (using OMEM). Check your memory manager. The hard-wired maximum of objects has been reached. Try to use more processors (i.e. less objects per processor) or use a bigger DDD library. Memory for DDD object couldn't be allocated (using OMEM). Check your memory manager.

The hard-wired maximum of objects has been reached during Xfer. Try to use more processors (i.e. less objects per processor) or use a bigger DDD library.

Memory for object coupling couldn't be allocated (using AMEM). Check your memory manager.

The hard-wired maximum of objects which are capable of holding couplings has been reached. Try to use more processors (i.e. less objects per processor), reduce object copies or use a bigger DDD library. Duplicate coupling record has been refused during Xfer. This warning has been deactivated in DDD 1.0.

64


4.3 Identi cation Module E3000:

not enough memory in IdentifySort

E3100:

not enough memory in DDD IdentifyEnd

E3200:

not enough memory in DDD IdentifyNumber

E3201:

not enough memory in DDD IdentifyString

E3202:

not enough memory in DDD IdentifyObject

E3210:

not enough memory in IdentifyIdEntry

Memory for identi cation array couldn't be allocated during Identoperation (using TMEM).

Memory for message buers couldn't be allocated during Ident-operation (using TMEM). Memory for special Ident-command couldn't be allocated during Identoperation (using TMEM). Memory for special Ident-command couldn't be allocated during Identoperation (using TMEM). Memory for special Ident-command couldn't be allocated during Identoperation (using TMEM).

Memory for general Ident-command couldn't be allocated during Identoperation (using TMEM).

4.4. INTERFACE MODULE

65

4.4 Interface Module E4000:

not enough memory in IFCreateFromScratch

F4020:

std if too large in IFCreateFromScratch

F4021:

interface too large in IFCreateFromScratch

E4100:

no more interfaces in DDD IFDefine

E4200:

timeout in DDD IFExchange

E4201:

timeout in DDD IFOneway

Memory for coupling array couldn't be allocated during interface creation (using AMEM). Hard-wired coupling limit has been reached. The xed coupling array size is too small for standard interface. Use bigger DDD library. Hard-wired coupling limit has been reached. The xed coupling array size is too small for one of the user DDD interfaces. Downsize the user DDD interfaces or use bigger DDD library.

The hard-wire interface number has been exceeded during interface de nition. Try to use less interfaces or use bigger DDD library. The corresponding interface communication on a partner processor hadn't been initiated in a reasonable time. In case of a slow communication network DDD may be too sensible in reporting this error; try to use another DDD library compiled with dierent timeout parameters then. The corresponding interface communication on a partner processor hadn't been initiated in a reasonable time. In case of a slow communication network DDD may be too sensible in reporting this error; try to use another DDD library compiled with dierent timeout parameters then.

66


4.5 Transfer Module E6000:

not enough memory in DDD XferEnd

E6100:

not enough memory in XferBuildMsgInfos

E6300:

not enough memory in NotifyPrepare

E6301:

not enough memory in NotifyPrepare

E6320:

not enough memory in NotifyTwoWave

E6321:

msg-info array overflow in NotifyTwoWave

E6400:

not enough memory in XferPackMsgs

E6500:

not enough memory in XferInitRecv

E6501:

not enough memory in XferInitRecv

E6520:

not enough memory in LocalObjectsList

E6540:

unused coupling (old) in XferUnpack

Memory for message description couldn't be allocated during a Xferoperation (using TMEM).


Memory for noti cation info couldn't be allocated during noti cation of transfer destination processors (using TMEM). Memory for routing table couldn't be allocated during noti cation of transfer destination processors (using TMEM).

Memory for noti cation info couldn't be allocated during noti cation of transfer destination processors (using TMEM). Over ow during noti cation process. Serious DDD internal error dependent on processor number. Memory for message buers on sender side couldn't be allocated during a Xfer-operation (using TMEM). Memory for message description on receiver side couldn't be allocated during a Xfer-operation (using TMEM). Memory for message buers on receiver side couldn't be allocated during a Xfer-operation (using TMEM). Memory for local object list couldn't be allocated (using TMEM).

Coupling will be discarded during Xfer-operation.

4.5. TRANSFER MODULE E6541:

unused coupling (new) in XferUnpack

E6542:

unused coupling (no object) in XferUnpack

E6560:

not enough memory in XferUnpack

E6561:

not enough memory in XferUnpack

67





68


4.6 Other Errors E9900:

not enough memory in ConsBuildMsgInfos

E9920:

not enough memory in ConsCheckGlobalCpl

Memory for message buering couldn't be allocated during DDD consistency check (using TMEM). Memory for message buering couldn't be allocated during DDD consistency check (using TMEM).

Bibliography [1] P. Bastian. Parallel adaptive multigrid methods. Preprint 93-60, Interdisziplinares Zentrum fur Wissenschaftliches Rechnen, Heidelberg, Oktober 1993. [2] K. Birken. An ecient programming model for parallel and adaptive CFDalgorithms. In Proceedings of Parallel CFD Conference 1994, Kyoto, Japan, 1994. [3] K. Birken and P. Bastian. Dynamic Distributed Data (DDD) in a parallel programming environment { speci cation and functionality. Forschungsund Entwicklungsberichte RUS{22, Rechenzentrum der Universitat Stuttgart, Germany, September 1994. [4] B. Hendrickson and R. Leland. The Chaco user's guide version 1.0. Report SAND93-2339, Sandia National Laboratories, Albuquerque, NM, 1993. [5] D. Roose and R. Van Driessche. Distributed memory parallel computers and computational uid dynamics. In H. Deconinck, editor, Lecture Notes on Computational Fluid Dynamics, LS 1993-04, Von Karman Institute for Fluid Dynamics, Belgium, 1993. [6] H. D. Simon. Partitioning of unstructured problems for parallel processing. Computing Systems in Engineering, 2(2/3):135{148, February 1991.

69

70

BIBLIOGRAPHY

Contents 1 Introduction

1

2 Reference Manual

3

DDD ConsCheck DDD DisplayIF DDD Exit DDD HandlerRegister DDD IdentifyBegin DDD IdentifyEnd DDD IdentifyNumber DDD IdentifyObject DDD IdentifyString DDD IFDe ne DDD IFExchange DDD IFOneway DDD InfoGlobalId DDD InfoPriority DDD InfoProcList DDD InfoRange DDD Init

: : : : : : : : : : : : : : : : : : : : : : : : : : : : : :

: : : : : : : : : : : : : : : : : : : : : : : : : : : : : : :

: : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : :

: : : : : : : : : : : : : : : : : : : : : : : : : : : : :

: : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : :

: : : : : : : : : : : : : : : : : : : : : : : : : : : : :

: : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : :

: : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : :

: : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : :

: : : : : : : : : : : : : : : : : : : : : : : : : : : : : : :

: : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : :

71

4 5 6 7 8 9 10 12 14 16 18 20 22 23 24 25 26

72

CONTENTS DDD ListLocalObjects DDD ObjDelete DDD ObjGet DDD ObjMoveTo DDD ObjUnlink DDD PrioritySet DDD RangeSet DDD StructDisplay DDD StructRegister DDD XferAddData DDD XferBegin DDD XferCopyObj DDD XferDeleteObj DDD XferEnd HANDLER COPYMANIP HANDLER OBJDELETE HANDLER OBJINIT HANDLER OBJMKCONS HANDLER OBJUPDATE HANDLER XFERCOPY HANDLER XFERDELETE HANDLER XFERGATHER HANDLER XFERSCATTER

: : : : : : : : : : : : : : : : : : : : : : : : : : :

: : : : : : : : : : : : : : : : : : : : : : : : : : : : : : :

: : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : :

: : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : :

: : : : : : : : : : : : : : : : : : : : : : : : : : : : :

: : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : :

: : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : :

: : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : :

3 Memory Manager

3.1 Memory Classes 3.2 MEMMGR Interface 3.3 The Standard MEMMGR

27 28 29 30 31 32 33 34 35 37 39 40 42 43 44 45 46 47 48 49 51 52 54

57

: : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : :

57 58 59

CONTENTS

73

4 Error List

61

4.1 4.2 4.3 4.4 4.5 4.6

General Module Object Manager Identi cation Module Interface Module Transfer Module Other Errors

: : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : :

: : : : : : : : : : : : : : : : : : : : : : : : : : :

: : : : : : : : : : : : : : : : : : : : : : : : : : : :

: : : : : : : : : : : : : : : : : : : : : : : : : : : : : :

62 63 64 65 66 68