A Component Object Model Binding for MPI ... - Semantic Scholar

A Component Object Model Binding for MPI Yoginder Dandass and Anthony Skjellum High Performance Computing Laboratory Department of Computer Science and Engineering Research Center for Computational Field Simulation Mississippi State University Mississippi State, MS 39762 {yogi,tony}@aurora.cs.msstate.edu

Abstract COM is a specification for creating languageindependent interfaces between client applications and server components. MPI is a standard that defines the API provided by communication libraries. While MPI’s communication abstraction provides a portable means for communicating in a number of disparate parallel environments, the implementations are usually limited to providing C and FORTRAN bindings. In order to provide a language-independent API for MPI, we have implemented an object-oriented COM wrapper around MPI/Pro, an implementation of MPI with the traditional C language bindings. The motivation behind this effort is to enable the creation of sophisticated applications from components consisting of off-the-shelf applications, all communicating via MPI. In this paper we describe how we built the COM interface wrappers around MPI’s opaque objects using Visual C++ and the Active Template Library. We describe the various design decisions resulting from the interface management and client-server data movement rules imposed by COM. Finally, we present an analysis of performance data that suggests that the additional overhead incurred in order to achieve language independence is relatively small. Therefore, the overhead does not prevent our implementation from being a viable binding for MPI.

1

Introduction

MPI has become a widely accepted tool for writing parallel programs in the scientific community. This paper focuses on describing an object oriented (OO) binding for MPI using the Component Object Model (COM). COM is a specification for the construction of language-independent programming interfaces between components (service providers) and clients. COM also consists of a standardized set of services necessary for the management of

components, and provides a means for binding clients with components at runtime. The COM based interface for MPI enables implementers to provide an API for use with languages like C++, Delphi, Smalltalk, PowerBuilder and Visual Basic, without having to write custom interfaces for each supported language. In addition, the COM Automation interface also allows applications written in non-compiled scripting (or macro) languages, such as those available in off-the-shelf applications, e.g. Microsoft Word and Microsoft Excel, to access MPI. A COM binding for MPI enables the creation of novel applications such as spreadsheets running in parallel or word processors driving high-performance servers and communicating via MPI. In order to achieve efficient parallel execution using the native Win32 API, a programmer must use different communication mechanisms depending on whether the application is targeted for an SMP or a clustered environment. By abstracting away the communication details, MPI allows a single code to run efficiently over both SMP and clustered environments. Only the underlying communication device needs to be tuned for a particular environment. Therefore, a COM binding for MPI eases the softwareengineering burden for parallel Win32 programming by presenting a single API and eliminating the need for maintaining multiple versions of source code for various possible parallel hardware configuration. 1.1 Implementation Environment Our MPI COM component is compiled into a Win32 DLL that exposes the MPI API to client applications via a COM compliant interface. A COM component itself can be written in any

language (even non-OO languages), but provides services in an OO fashion to its clients. We chose to implement our component using Visual C++ 6.0. We also leveraged the COM interface management support functionality offered by the Active Template Library (ATL). The C++ classes in ATL serve as ancestors for our COM classes, and handle the routine functionality that is a part of all COM interfaces and components [1]. This allowed us to focus our development effort on implementing the MPI bindings. Rather than implementing the entire MPI functionality ourselves, we chose to create a COM wrapper around MPI/Pro for NT, a commercial C language implementation of MPI [2]. However, these concepts should apply to any Win32 implementation of MPI with C bindings. In order to exercise our component, we wrote several client MPI applications in Visual Basic (VB). We chose VB because it is very different from the languages traditionally used with MPI and can produce a compiled executable file. Additionally, it hides most of the details behind invoking COM components from the programmer, a service provided by many scripting languages. The remainder of the paper is organized as follows: • Section 2 provides a brief background of COM as it applies to our effort, • Section 3 outlines the various MPI interface objects we have implemented for our component and discusses the issues raised during its design and implementation, • Section 4 describes the performance overhead of our component, and • Section 5 discusses our conclusions and possible avenues of future work.

2

COM Background

A client acquires a component interface by calling the COM support function CoCreateInstance. This function takes a component ID and an interface ID, and returns a pointer to the requested interface in the component. Figure 1 illustrates the underlying process. First, the COM support library, provided by the Win32 operating system in our case, looks for the executable file that implements the

component. This association between components and executable files (DLL or EXE) is maintained in the system registry. Next, COM loads the executable file, and invokes specialized interfaces to instantiate a component object. Finally, COM queries the component’s IUnknown interface to obtain and return the requested interface to the client.

Client 1. Client calls CoCreateInstance(…)

COM Support

4. Client invokes component methods directly via the interface pointer.

3. Component returns the interface pointer.

COM Component

Win32 Registry 2. COM locates and starts the component

Figure 1: Invoking a COM component [3] A COM component exposes an interface it implements via a pointer to an array of pointers to the methods in the interface. This array is very similar to the C++ class dispatch table, and is known as the vtable. A component may publish several distinct interfaces, each represented by a separate vtable. Figure 2 illustrates this concept for a COM component. In order to call a method in the interface, a client obtains the corresponding function pointer from the interface’s vtable. 2.1 IUnknown The first three methods in an interface belong to the IUnknown interface. The remaining methods are native to the component. IUnknown is the basis for all COM interfaces. It must be supported by all COM components, and all interfaces exported by a component are inherited from it. IUnknown contains the following three methods: 1. QueryInterface – this method is called by clients to retrieve the specified interface implemented by the component. That is, once

a client has an interface to a component, it can use this method to ask for any other interface implemented by the component. 2. AddRef, and 3. Release – These methods are used to perform reference counting by the component, in order to determine when it is safe to destroy unused interfaces.

Client

Component QueryInterface()… AddRef()… Release()… Method1()… Method2()… Method3()… Method4()… vtable

Method functions

Figure 2: A vtable based COM interface. IUnknown methods are shown in gray [3]. The vtable is similar to the conventional C/C++ technique of invoking library code, and is not very flexible. The client programmer must be given a language dependent specification file (C or C++ header files for example) that describes the layout of the vtable, the function prototypes containing the function name and parameter types for each method in the interface. Therefore, custom interfaces designed using the vtable offer no relief from language dependence that we are seeking to eliminate. 2.2 IDispatch and Dispinterfaces The developers of VB defined an alternative interfacing technique called the dispinterface, that eliminates the need for language dependent binding information at compile time. This interface is implemented via another standard COM interface, IDispatch, which consists of the following methods: 1. GetTypeInfoCount – clients call this method to determine if the GetTypeInfo method will return type information at runtime. 2. GetTypeInfoCount – clients use this method to retrieve another standard COM

interface ITypeInfo. This interface is used to query type information about the methods and parameters in the dispinterface at runtime. 3. GetIDsOfnames – This method is used to return the index of a function pointer in the dispatch table corresponding to a method name. 4. Invoke – This method is called by clients to invoke a function at a specified index in the dispatch table.

Client

Component QueryInterface() AddRef() Release() GetTypeInfoCount() GetTypeInfo() GetIDsOfNames() Invoke() vtable DispIDs

1 MethodA() 2 MethodB() Method functions

Figure3: An IDispatch based interface. The dispinterface methods are shown in gray [3]. The dispatch table itself is not exposed to the clients, and its implementation is left up to the component writer. It can be implemented in a switch construct, or a vtable-like array of function pointers. Figure 3 depicts a dispinterface. Note that the methods in IDispatch are in the vtable, whereas the methods of the dispinterface are not. COM interfaces are typically defined using the interface description language (IDL). The interface definition is then compiled using MIDL, Microsoft’s IDL compiler. The compilation process results in a number of useful source-code files, and a type library file. The type library is used by the ITypeInfo interface to return dispinterface method runtime binding information to languages such as VB. Since most of these details are handled automatically, additional consideration of ITypeInfo and type libraries is not required for designing a COM interface for MPI.

In order to achieve inter-language portability, dispinterfaces only allow those parameter types compatible with the VARIANT datatype to be passed to its methods. The VARIANT datatype is implemented as a structure holding a standard form of the parameter, along with a type identifier (such as integer, float, array, and pointers).

Client

Component QueryInterface() AddRef() Release() GetTypeInfoCount() GetTypeInfo() GetIDsOfNames() Invoke() 1 MethodA() 2 MethodB() vtable Method functions

Figure 4: A dual interface. The vtable has pointers to the dispinterface methods [3]. 2.3 Dual Interface A dispinterface is convenient for VB users because the language handles the details behind obtaining type information, dispatch table indices and stuffing parameters into variants. However, this is a cumbersome process for C and C++ programmers. A dual interface caters to both macro and compiled languages by combining the IDispatch and vtbl interfaces into a single interface. Therefore, a dual interface consists of the methods from the IUnknown and IDispatch interfaces, as well as the dispinterface. IDispatch simply invokes the appropriate function in the vtable on behalf of the dispinterface. We use the dual interface scheme in our COM binding for MPI.

3

COM Considerations for MPI

In our implementation, the entire MPI API is contained within a single COM component. However, the API is split into several nonintersecting interfaces. The logical objects in MPI, such as communicators, datatypes and groups, serve as the basis for the OO

decomposition we have adopted for our interfaces. Functions are grouped according to the primary objects they act upon, and each interface corresponds to a single object. Figure 5 shows a sample VB program that obtains an MPI interface, initializes MPI, broadcasts 100 integers stored in a 10x10 two dimensional array, and then terminates. Dim RC As Integer ’return code Dim Rk As Integer ’rank ’declare an uninitialized interface Dim MPI As Object ’allocate an array of 10x10 integers Dim Buffer(9, 9) As Integer ’obtain the IMsgPass interface Set MPI = CreateObject(“MPI.IMsgPass”) Call MPI.Init Rk = MPI.COMM_WORLD.Rank

• RC = MPI.COMM_WORLD.BCast(Buffer,100, MPI.INT,0)

• Call MPI.Finalize Set MPI = Nothing

Figure 5: Client code in VB performing a broadcast The C and FORTRAN bindings specified in the standard provide clear guidelines for implementers targeting these languages. However, implementations targeting other languages must interpret the language independent specification (LIS) of the API to fit the native language environment. For example, since the MPI LIS does not natively support object-oriented languages, a direct implementation of the LIS severely limits the utilization of language features that eases programming. A common example of this situation is that some C++ implementations have extended the API in order to take full advantage of C++ features [5, 6, 7]. We have also deviated from the MPI LIS in minor ways to make COM-based MPI programming convenient. In particular, we have eliminated the MPI_ prefix from all the MPI entities because the interface pointer provides the context for the entity name. Also, we take advantage of a special kind of method in dispinterfaces known as properties. Property methods provide to VB programmers a shorthand for getting and setting values within components. For example, in the statement:

Rk = MPI.COMM_WORLD.Rank COMM_WORLD is a read-only property of interface IMsgPass, implemented via the get_COMM_WORLD function, that returns an IComm interface (which represents MPI’s communicator objects) initialized to contain the MPI_COMM_WORLD communicator. Similarly, Rank is a read-only property of the IComm interface, implemented via the get_Rank function that returns the rank of the current process in the communicator. Interface properties are specified in the IDL file for the component. This automatically generates the prototype code for the appropriate get_ and set_ methods for the corresponding interface when the IDL file is compiled. The set_ methods are not generated for read-only properties. Another important aspect of our implementation is that all API parameters that represent MPI objects (such as datatypes, communicators, etc.), take pointers to the interfaces that represent them. The COM interfaces themselves are implemented using C++ classes that wrap the underlying implementation’s entities. For example, in the following VB statement: MPI.COMM_WORLD.BCast(Buffer,10 0 MPI.INT,0) the third parameter, MPI.INT, evaluates to an IDatatype interface pointer that represents an MPI_INT datatype. The C++ class implementing IDatatype, has a MPI_Datatype type member that holds the underlying implementation’s representation of the MPI datatype being considered. In general, an instance of a logical MPI object is represented within a client as an interface pointer. The interface is implemented within our component by an instance of a C++ class that, in turn, wraps the underlying MPI implementation’s opaque object. 3.1 The IMsgPass Interface In a COM component, functions must be accessed through an interface. Therefore, we needed to define a “MPI” interface, IMsgPass, in order to support functions such as MPI_INIT,

MPI_FINALIZE, and others that are not associated with particular MPI objects. This interface also contains read-only property methods that return predefined instances of MPI objects such as the communicator MPI_COMM_WORLD and datatype objects MPI_DOUBLE, MPI_INT, etc. Since these are read-only properties only the corresponding get_ functions (such as get_COMM_WORLD, get_DOUBLE, and get_INT) need to be implemented for IMsgPass. The IMsgPass interface is implemented using a C++ class IMsgPass. This is a singleton class; the component creates only one instance of this class. Therefore, multiple requests to create the IMsgPass interface simply return the pointer created to accommodate the first request. 3.2 The IComm Interface The IComm interface encapsulates the MPI communicator functionality. This includes the various forms of Send and Recv operations, topology functions and communicator grouping functions. HRESULT IComm::get_Rank(int *pValue) { MPI_Comm_rank(m_Comm, pValue); Return S_OK; }

Figure 6: Implementation of the read-only Rank property of IComm. The underlying MPI implementation’s communicator is stored in the member variable, m_Comm, of the C++ class used to implement IComm. This communicator is used when calls to the underlying MPI implementation are made, as shown in Figure 6. MPI_COMM_WORLD is a constant that represents an implementation defined unique handle to the communicator object representing all processes in an application. In our implementation, shown in Figure 7, the Init method of IMsgPass creates an instance of the IComm interface, initializes its m_Comm member variable to MPI_COMM_WORLD, and saves a pointer to this IComm instance. Subsequent calls to the get_COMM_WORLD function (i.e. the COMM_WORLD read-only property) simply returns this saved pointer, rather than creating and initializing new instances of IComm.

In contrast, non-constant MPI objects are handled differently. For example, consider the communicator splitting operation that takes an input communicator and returns a new one. In our implementation, a new IComm interface is created to wrap the newly created communicator. The pointer to the new IComm is not stored within the component, but instead, is returned to the client which must call the new IComm’s Release method when it needs to destroy the new communicator. IMsgPass::Inti() { … CoCreateInstance(CLSID_COMM, …, (void**)&m_pCommWorld); m_pCommWorld->m_Comm=MPI_COMM_WORLD; … } HRESULT IMsgPass::get_CommWorld(IComm **ppV) { *ppV = m_pCommWorld; Return S_OK; } IMsgPass::Finalize() { … m_pCommWorld->Release(); … }

Figure 7: Initialization and implementation of the CommWorld property of IMsgPass. In MPI, the data transfer operations specify that the buffers may contain data of any type. We achieve this functionality in our implementation by specifying in the IDL file that the buffer parameters are passed as variants. Therefore, a client application can pass a buffer consisting of a single variable by value, or an array by reference. The interface method extracts the buffer pointer as shown in Figure 8, and calls the underlying implementation’s function. 3.3 VARIANTs and SAFEARRAYs Clients pass parameters to dispinterfaces in a VARIANT type. A variant is a structure type consisting of a union of different datatypes [4]. A separate field in the structure indicates the datatype of the variant, and whether the variant

contains the data value itself, a pointer to it, or a pointer to an array of values. The Invoke method in the dispinterface extracts the parameters from the input variants and translates them to the format expected by the method function. Therefore, the number of datatypes that can be passed to dispinterfaces is limited to what is allowed in a variant. Since most of the MPI API functions use types compatible with variants, this limit does not pose a problem for our implementation. IComm::Send(const VARIANT *Buffer, int iCount, IDatatype *pIDatatype, int iDest, int iTag) { if (V_ISARRAY(Buffer)) { // Buffer is an array SAFEARRAY FAR *psa; psa = V_ARRAY(Buffer); SafeArrayAccessData(psa,&pBuffer); MPI_Send(pBuffer,iCount, pIDatatype->m_mpiType, iDest iTag); SafeArrayUnaccessData(psa); } else if (V_ISARRAYREF(Buffer)){ // Buffer is an arry pointer SAFEARRAY FAR **ppsa; ppsa = V_ARRAYREF(Buffer); SafeArrayAccessData(*ppsa,&pBuffer); MPI_Send(pBuffer,iCount, pIDatatype->m_mpiType, iDest iTag); SafeArrayUnaccessData(*ppsa); } else { // buffer passed by value pBuffer = &(V_I2(X)); MPI_Send(pBuffer,iCount, pIDatatype->m_mpiType, iDest iTag) }; return S_OK; }

Figure 8: Use of VARIANTS and SAFEARRAYS. MPI expects pointers to the data buffers participating in message transfers. However, a client can send parameters by value in a variant. Therefore, for send operations, the interface

automatically passes the address of the data being sent if it is passed by reference. For the receive operations, the buffer variant must contain a pointer to the buffer. Therefore, the interface returns an error if the buffer is passed by value. Arrays are passed indirectly in variants. The variant points to a SAFEARRAY structure that contains information such as the number of dimensions, and a pointer to the array’s contents [4]. Therefore, any interface method that accepts an array must extract the associated data pointer to pass onto the underlying MPI library. Figure 8 shows code for the Send method. Since it is very likely that the buffer parameter contains an array, the method contains code to handle safearrays. 3.4 Interface Reference Counting In order to guarantee that all the interfaces destroy themselves, the client must ensure that the interface reference counting occurs properly. COM automatically calls an interface’s AddRef method when it is created via a call to CoCreateInstance [4]. In VB, additionally, whenever a client aliases (copies an to another variable) an interface pointer, VB automatically calls the interface’s AddRef method. A client implemented in C or C++ will need to call an aliased interface’s AddRef method explicitly. Similarly, it is the client’s responsibility to call the interface’s Release method to indicate that the interface is no longer needed [4]. VB automatically calls the corresponding Release method if a variable containing an interface pointer goes out of scope. However, global interface pointer variables should be assigned a value of Nothing in order to have VB call the interface’s Release method. Therefore, in order to cause proper shutdown of the interfaces, and ultimately the component, a VB client must ensure that none of the interface variables are valid when the program ends and a C/C++ client must explicitly call the Release method for all interface pointers it has. As described earlier, the IMsgPass interface creates several interfaces that wrap MPI’s predefined constant objects (e.g. MPI_COMM_WORLD) when the Init method is called. Essentially, the component assumes the role of a client for its own interfaces. The

Release method for each of these interfaces is called in IMsgPass’s Finalize method. The reference counting mechanism of the interfaces in our component is implemented by the ATL base classes from which we derive our interface classes [1]. ATL destroys the C++ object implementing an interface when the interface’s reference counter becomes zero. If all of the component’s interfaces have destroyed themselves, the DLL implementing the component unloads itself. 3.5 Issues and Challenges Nearly all of the logical objects in MPI are constructed from existing objects. For example, MPI_COMM_WORLD is manipulated to create other communicators. Therefore, the constructor methods for these objects are located in the interfaces corresponding to the base objects. There are a few exceptions however. The Struct user defined datatype is constructed by replicating several different datatypes. Therefore, the Type_struct method is member of the IMsgPass interface, and not of the IDatatype interface. Similarly, the user defined operator object used in reduce and scan operations is another example of an object not constructed from an existing MPI object, and its constructor, the Op_create method, is also a member of the IMsgPass interface. The client defined callback functions encapsulated by user defined operation objects require special handling by our COM component. The underlying implementation of MPI expects callback functions to accept pointers to two arrays of operands. However, a VB callback function only accepts arrays that are wrapped inside safearrays. Therefore, our component’s IOp interface object has the underlying MPI implementation call a proxy function within the component. This function creates safearrays from the operand array pointers and then calls the client’s callback function. This procedure however, is further complicated because safearrays are typed. Therefore, we had to add an additional integer parameter to the Op_create method which specifies the type of the elements of the safearray. This type specifier uses the same values as those used to specify the datatype of variant parameters.

4

Performance

Our implementation, consisting of a COM wrapper around a traditional MPI library, imposes two levels of processing overhead. The first level of overhead is caused by the additional function call involved in invoking a COM interface method, which, in turn, calls the underlying MPI library’s corresponding routine. The second and more significant overhead is caused by COM compliance. A contributing factor to this overhead is the reference counting performed by the component and client whenever an interface is created or aliased. The reference counting of interfaces is essential for the automatic garbage collection performed by the component. Another contributing factor is the language independence afforded by the dispinterface. Whenever the client invokes a method in the dispinterface, it must determine the datatype of the parameters from the dispinterface’s GetTypeInfo method, stuff parameters into varaints, lookup the index of the method name using the GetIDsOfNames method, and then call the Invoke method to indirectly execute the selected method. Furthermore, within the executed method, the parameters are extracted from the variant and converted to the format native to the component’s code. The situation is a little better for those clients that can call the interface methods directly from the vtable of the dual interface. However, these clients must also stuff parameters into variants before calling the interface methods.

20 15 C

10

VB

5 0 10K

20K

30K

40K

Number of 0-Byte Messges

Figure 9: Time to Transmit 0-Byte Messages using C and VB. Figure 9 shows timing results from our experiments in which zero-length messages were

transferred between a pair of C processes and a pair of VB processes in a “ping-pong” fashion. The VB application passed parameters to our component in the expected format, thereby minimizing parameter translation overhead. Each column in the chart represents the total time taken to complete the indicated number of 0-byte transfers. From the data it is clear that our COM wrapper imposes an additional latency overhead of 50% over that imposed by the underlying MPI/Pro implementation.

5

Summary and Conclusions

We were successful in our attempt to wrap a traditional MPI library with C bindings within a COM component. We used the dual COM interface style to make our bindings languageindependent, and are now able to implement parallel programs in Visual Basic using our COM component for MPI services. The introduction of the IMsgPass interface, the elimination of the MPI_ prefix and the use of properties are the three most significant differences from the traditional MPI API. Of these, only the first one is essential for COM; the other two provide syntactic convenience. The additional latency introduced by our component, as compared to the latency of the underlying MPI/Pro implementation, is small enough to result in a useful COM binding. We can eliminate a small portion of the overhead by natively implementing the MPI services within our component. Unfortunately, the COM induced overhead cannot be eliminated, and must be accepted as the price for language independence and the convenience COM offers to client programmers. A COM based implementation of MPI is well suited for teaching parallel programming using message passing in a “visual” environment or for rapidly prototyping a parallel algorithm in a high level language. Our implementation will also meet the communication needs of software developers that are writing business applications consisting of several concurrently running programs written in Visual Basic or PowerBuilder. Such applications do not need the highest-speed or lowest-latency communication channels, but do need portable inter-process communication with an easy to use interface. Our implementation offers the high-level message passing abstraction

of MPI coupled with the syntactic convenience of COM. In the future, we plan to examine various methods for reducing overall overhead and improving the implementation of user-defined reduction operations. We will also compare the performance of our wrapper-based implementation to that of one using DCOM for performing data transfers.

6

References

1. Anonymous, Active Template Library (ATL) Reference. Microsoft Developer Network, 1999, http://msdn.microsoft.com/library/devprods/vs 6/vc++/vcmfc/atl.htm, last accessed on February 1999. 2. Anonymous, MPI/Pro™ for Windows NT®. MPI Software Technology, http://www.mpisofttech.com/products/mpi/mpipro/index.html, accessed on February 1999. 3. D. Chappell, Understanding ActiveX and OLE, pages 39-104. Microsoft Press, 1996 4. D. Rogerson, Inside COM, pages 279-309. Microsoft Press, WA, 1997. 5. A. Skjellum, et al., Object-Oriented Analysis and Design of the Message Passing Interface. Concurrency: Proactice and Experience. Accepted December 1998, in process. 6. A. Skjellum, et al., Explicit parallel programming in C++. In G. Wilson, editor, Parallel Programming Using C++. MIT Press, 1996. 7. J. Squyres, B. McCandless, and A. Lumsdaine, Object Oriented MPI, A C++ Class Library for MPI Version 1.0.2f. http://www.lsc.nd.edu/research/oompi/documentation.htm, accessed on February 1998.