Reification of Foreign Type Systems - CiteSeerX

Reification of Foreign Type Systems Mark Grechanik, Don Batory, and Dewayne E. Perry UT Center for Advanced Research In Software Engineering (UT ARISE) University of Texas at Austin Austin, Texas 78712 {gmark, batory}@cs.utexas.edu, [email protected]

Abstract. Building systems from existing applications and data sources is common practice. Semi-structured data sources, such as XML, HTML, and databases, and programming languages, such as C# and Java, conform to welldefined, albeit different, type systems, each with their own unique underlying representations. As a consequence, writing programs that access and update data in foreign type systems (FTSs), i.e., type systems that are different from the host programming language, is a notoriously difficult task. In this paper, we present a simple, practical, and effective way to develop and maintain FTS-based systems. We accomplish this by abstracting foreign data as graphs and using path expressions for traversing and accessing data. Path expressions are implemented by type reification — turning foreign types into first-class objects and enabling access to and manipulation of their instances. Doing this results in multiple benefits, including coding simplicity and uniformity (neither of which was present before), that have been demonstrated in a complex commercial project. The contribution of this paper is an approach that allows programmers to operate on foreign types and their instances without writing or generating additional code. We know of no other approach with comparable benefits.

1 Introduction Building software systems from existing applications is a well-accepted practice. Applications are often written in different languages and provide data in different formats. An example is a C++ application that parses an HTML-based web page, extracts data, and passes the data into a relational database. A fundamental problem of engineering these systems is how to operate on different formats and types without introducing unnecessary complexity. Different data formats and languages have different type systems. A markup language type (for example, an HTML tag) does not have a direct counterpart in C++, and a C++ class has no explicit counterpart as an HTML tag. Sometimes keywords describing types in grammars may be the same, for instance, the keyword int

describes the integer types in C++ and Java, but their internal representations can differ. In this respect every language type system is unique. As a consequence, systems that manipulate data in foreign type systems (FTSs) are very difficult to develop. Currently, programmers must map foreign types and their instances to types and their instances in a host programming language. Complicating this development is the sheer multiplicity of built-in and user-defined types required to do this mapping. Recall the C++ program that parses HTML-based data and updates a database with the information retrieved. Even though this task sounds trivial, in reality it is complex and is composed roughly of the following steps: •

Locate and analyze the HTML-based data;

•

Map these types onto C++-specific types;

•

Analyze the database schema and determine the mapping between the C++ types and database entities;

•

Write functions that parse HTML documents, retrieve HTML data, and convert this data to its C++ counterparts;

•

Write functions that convert C++ objects into SQL queries that are executed against the database.

This approach suffers from multiple drawbacks. First, it leads to software that is extremely difficult to extend and scale. Since C++ types are different from HTML types, programmers must go through a complex process of parsing and mapping HTML data to its counterparts in C++. Moreover, a programmer must create a set of functions whose semantics reflects the operations on these types. With a growing number of types and functions, program complexity becomes increasingly unmanageable and thus limits scalability and extensibility. Second, this approach often leads to non-uniformity in FTSbased code. Since programmers are constrained neither in their mappings between foreign-types-to-host-types nor in their operations on mapped types, the resulting code for different type systems looks different, even when it is written

1

by the same programmer. That is, given two different representations, for example, HTML and XML, the C++ code that implements the same operation on these representations will be different. Even when only one representation is used, nonuniformity can result from using either different low-level APIs (e.g. the Xerces and MS XML APIs are not the same) or different type mappings and operations. The loss of uniformity has a negative impact on a programmer’s ability to reason about FTS-based programs1. This is an example of architectural mismatch [1]. Existing technologies do not address this problem. Distributed object infrastructures, such as CORBA and J2EE, or virtual machines, such as .Net CLR and JVM, introduce their own type systems with limited capabilities to alleviate some of the programming difficulties of FTS-based systems. That is, they either define interfaces (which must be implemented in different programming languages) or they define a common underlying type representation shared by programming languages. However, this is not sufficient for the problems that we address. For example, there is no virtual machine that translates HTML types to Java and C# types. There are different low-level tools, such as generators, that can do this mapping for individual languages, but they are both vendorspecific and dissimilar. These tools and their APIs introduce significant accidental complexity and nonuniformity into programming FTSs, and do not eliminate the steep learning curve required to master the APIs for each platform. We discuss these and other approaches in Section 8. Our approach to solve this problem is to abstract instances of a foreign type system as a graph of objects and to provide language-neutral specifications based on path expressions, coupled with a set of basic operations, for traversing this graph to access and manipulate its objects. We implement traversals by dynamically converting foreign types into first-class host language objects so that we enable access to and manipulation of their instances. This is the concept of type reification. Reifying dynamically eliminates the need for generating potentially huge numbers of conversion classes, and allows us to access and manipulate semi-structured data that have no schemas2. In this respect, this is superior to existing approaches, such as CORBA and DOM, because it provides programming uniformity without requiring programmers to explicitly define common interfaces or use different low-level APIs. Ours is a simple and effective solution for dealing with FTS applications. We call our approach the Reification Object-Oriented Framework (ROOF). 1. Non-uniformity leads to the loss of generality and the presence of special case reasoning. 2. A schema is a set of artifact definitions in a type system that defines the hierarchy of elements, operations, and allowable content.

2 A Motivating Example Consider a schema that describes the organizational structure of a company shown in Figure 1. It is a directed graph where each node is named after an organizational entity within a company and edges describe the subordination of one entity to the other. Each node has attributes shown as line connectors with filled circles followed by the names of the attributes. The CEO’s subordinate is the CTO who in turn supervises two departments shown as Test and Geeks. An instance of this schema may be given in existing markup languages such as HTML, XML, or SGML. We need to write a function that parses an instance of this schema and returns a number of employees in the Geeks department. Suppose the desired function is GetNumberOfGeeks. The steps to write this program are illustrated in Figure 1. First, we create C++ types that represent the types in the schema. Second, we define operations over these types. An example is the function GetCEO in Figure 2. Its implementation assumes the schema is in XML and the parser is the MS XML parser using COM and the Active Template Library. These tools and libraries are among the best available and are widely used today. The complexity of the GetCEO code is clearly evident. Note that if we use a different XML parser (e.g., for improved performance), we must rewrite this code because their lowlevel APIs would be different. Second, writing the code for GetNumberOfGeeks is based on assumptions made when defining the mapped types and the operations over them. For example, the pointer to the type CEO returned by GetCEO should be checked for validity before used, and the pointer to the type _employee may be a linked list with the member variable pNext pointing to the next member of the list. These details enormously complicate programming. This example shows how a simple problem of counting employees turns into a daunting task. But more than requiring large amounts of code to do trivial things from a user’s point of view, it can require enormous resources for maintenance and program evolution.

3 Type Graphs Schemas can be represented as graphs whose nodes are composite types, leaves are primitive or simple types, and parentchild relationships between nodes or leaves defines a type containment hierarchy [2]. In order to operate on such graphs, a programmer must be able to reach nodes at arbitrary depth. This is accomplished via path expressions that are queries whose results are sets of nodes. A path expression is a sequence of variable identifiers or names of subordinate (or

2

CEO

Name Bonus

Name Salary

CTO

Geeks

Test

Employees

Employees

Struct String String Struct Struct }

_ceo { m_ceoName; m_TotalBonus; _cfo *pCfo; _cto *pCto;

Struct String String Struct Struct }

_cto { m_ctoName; m_AnnualSalary; _test *pTest; _geeks *pGeeks;

Struct _cto *GetCTO(Struct _ceo *ceo )

Struct String String Struct }

_employee { m_empName; m_AnnualSalary; _employee *pNext;

Struct _employee *GetCollectionOfGeeks( Struct _cto *cto)

Struct _ceo *GetCEO()

Types

Schema

Int GetNumberOfGeeks() { Int n; Struct _ceo *TheCEO; Struct _cto *TheCTO; Struct _employee *emp;

Operations

TheCEO=GetCEO(); TheCTO=GetCTO( TheCEO ); emp=GetCollectionOfGeeks( TheCTO ); n = 0; do { if ( emp != NULL ) ++n; else break; } while( emp=emp->pNext ); Return( n );

}

Figure 1: Steps to write a C++ program that takes instances of a schema describing the organizational structure of a company and compute the number of employees in the Geeks department. containment) types that define a unique traversal through a schema.

class ReificationOperator { public: ReificationOperator &GetType( string t ); int Count( void ); }; ... ReificationOperator R;

Suppose we have a handle to an object that is an instance of a foreign type. We declare this handle as an instance R of a ReificationOperator class shown in Figure 3. R enables navigation to a type in the referenced type graph by calling its method GetType with a path expression as a sequence of type names t1, t2, ..., tk as parameters to this method.

Figure 3: Declaration and instantiation of ReificationOperator class in C++

R.GetType(t1).GetType(t2)...GetType(tk) struct _ceo *GetCEO( void ) { HRESULT hr = CoInitialize( NULL ); if( FAILED(hr) ) return( NULL );

CComVariant Val; CComQIPtr spAtt; spAtt = nodeTmp; hr = spAtt->get_nodeValue( &Val ); nodeTmp.Release(); if( FAILED(hr) ) return( NULL );

CComPtr spDomDoc; hr = spDomDoc.CoCreateInstance( __uuidof(DOMDocument40) ); if( FAILED(hr) ) return( NULL );

if( vValue.vt == VT_BSTR ) { CEO->Name = BSTRToAscii( Val.bstrVal ); SysFreeString( Val.bstrVal ); }

CComPtr node; node = spDomDoc; BSTR b = AsciiToBSTR( "CEO" ); hr = node->selectNodes( b, &childList ); SysFreeString( b ); if( FAILED(hr) ) return( NULL );

b = AsciiToBSTR( "Bonus" ); hr = attrMap->getNamedItem( b, &nodeTmp ); SysFreeString( b ); if( FAILED(hr) ) return( NULL );

CComPtr nodeCEO; hr = childList->get_item((long)0,&nodeCEO ); if( FAILED(hr) ) return( NULL );

spAtt.Release(); spAtt = nodeTmp; hr = spAtt->get_nodeValue( &Val ); nodeTmp.Release(); if( FAILED(hr) ) return( NULL );

struct _ceo *CEO = new struct _ceo; if( retValue == NULL ) return( NULL ); hr = nodeCEO->get_attributes( &attrMap ); if( FAILED(hr) ) return( NULL );

if( vValue.vt == VT_BSTR ) { CEO->Bonus = atof( BSTRToAscii(Val.bstrVal) ); SysFreeString( Val.bstrVal ); }

b = AsciiToBSTR( "Name" ); CComPtr nodeTmp; hr = attrMap->getNamedItem( b, &nodeTmp ); SysFreeString( b ); if( FAILED(hr) ) return( NULL ); (continued next column)

CoUnintialize(); return( CEO ); }

Figure 2: Implementation of GetCEO using C++, COM, and ATL

3

We simplify this notation by introducing array access operators [] that replaces the GetType method. For example, if R denotes an instance of the schema shown in Figure 1, we can rewrite the C++ program GetNumberOfGeeks that counts the number of employees in the Geeks department as: int n = R[“CEO”][“CTO”][“Geeks”].Count();

The method Count returns the number of child nodes under a given path. Such notation is useful since fifteen lines of obscure code in Figure 2 are reduced to a single line of lucid code. No additionally defined types and operations are required. Path expressions symbolize simplicity and uniformity. These are the properties that we inherit from path expressions and they enable programmers to uniformly navigate through instances of foreign types. Further, since type names are used as they are defined in foreign schemas, there is no need to redefine them again in the host language. That is, we use the existing names of foreign types; we do not create corresponding types in the host language! We will show how this is accomplished shortly. Very often FTS-based applications change each other’s structures. A common example is a C++ application that changes the structure of an XML document. These modifications are complex and require carefully crafted software called transformation engines. However, since all type systems can be represented as graphs, these modifications can be reduced to transformations on graph structures. Thus, we reduce the task of manipulating FTS structures to that of manipulating graphs. A comprehensive set of basic operations used to manipulate FTS graph structures is shown in Table 1. By implementing these operations using a standardized notation we achieve uniformity of FTS-based code. Indeed, FTS applications written in different languages that perform the Operation Copy

Move

Add Remove Relational Logic set

Conversion

Description Creates a copy of a node and adds it to its new location. All properties of the node are cloned. Identical to the copy operation except for the automatic removal of the original node upon completion of copying. Appends a node under a given path. Removes nodes from the given path. Compares graphs with constants, variables, or other graphs. Computes various logic set operations such as intersection, union, cartesian product, complement, and difference. Transforms one instance of a graph into another

TABLE 1. Basic operations that can be performed over abstract type graphs

same operation on the same schema will look the same. This very important property of uniformity enables effective program evolution of and automated reasoning about FTS applications.

4 Type Reification In this section we show how to reify types. We first explain reification concepts, then their abstraction as reification operators, and then present detailed notes on implementation.

4.1 Reification Concepts Reflection. Reflection is a powerful and common mechanism in contemporary programming languages and programming infrastructures (e.g., reflection in virtual machines). Reflection exposes the type of a given object; it reveals the public data members, method names, type signatures of method parameters and results, and superclasses (if any) of an object’s class. Further, reflection enables a program to invoke methods of objects whose classes were not statically known at the time the program was compiled. It also allows a program to navigate a graph of interconnected instances without statically knowing the types of these objects. Mutable reflection allows the structure of programs to be modified and new programs to be created [3]. All of this information and power is available to a program at run-time. Connectors. A reification connector (RC) is an example of an architectural connector [4][5]; it is a communication channel between a host language application and an application with a foreign type system. At the host language end, there are one or more classes — corresponding to reification classes as in Figure 3 — which accepts navigation instructions starting from a given foreign object. These instructions are transmitted via the RC to one or more classes in the foreign application, which executes these instructions and returns a reference to the resulting object. This is similar to the way methods of remote objects are executed in CORBA and the result is returned to the calling language. The difference is that there is no need to define explicit CORBA interfaces between the host (or client) application and the foreign (or server) application. Internally, we use a low-level API to transmit names of object attributes, names of methods, and primitive values to execute in the foreign application, and use reflection (on the foreign application side) to generate the appropriate method call. Combining. Using reflection and reification connectors, it is possible for a host program in one type system to navigate a graph of objects in a foreign type system. Suppose we are given a foreign object x and a path expression x.a.b. That is, starting with foreign object x, we access its “a” attribute to

4

obtain some object y, and then we access the “b” attribute of y, as the result of the path expression. In more detail given x, we transmit “a” to the foreign executable. Using reflection, we can validate that “a” is indeed a public member of x, and by invoking the appropriate get method (or simply variable access), we can access the “a” value of x.3 We return the handle of the resulting object y back to the host language, and repeat the process for attribute “b”. This is the essence of our implementation.

4.2 The Reification Operators We mentioned in the previous section that a host application has a set of classes that hide the details of our implementation. In fact, the handle to a foreign object is the object R of Figure 3. R implements a reification operator (RO) that provides access to objects in a graph of foreign objects. We give all ROs the same interface (i.e., the same set of methods) so that its design is language independent; reification operators possess general functionality that can operate on type graphs of any FTS. By implementing R as an object-oriented framework that is extended to support major computing platforms, we allow programmers to write FTS programs using a uniform language notation without having to bother about peculiarities of each platform. That is, for Java we have separate extensions of the framework that allows Java programs to manipulate C# objects, another extension to manipulate XML documents, etc. Similarly, for C# we have separate extensions of (an equivalent) framework that allows C# programs to manipulate Java objects, another extension to manipulate XML documents, etc. Reification operators are thus nonsymmetrical, i.e. reifying types from FTSI to FTSJ is not the same as the converse. RO has the transitive property, i.e. by applying RO RIJ to an instance of FTSI we reify it to the FTSJ. Then by applying RO RJK to an instance of FTSJ we reify it to the FTSK. The same result is achieved by applying RO RIK to an instance of FTSI. This property is useful since it enables the composition of reification operators to obtain a new RO. Finally, an identity RO reifies FTS types to the same type system, and interestingly, this is useful. Consider a Java program that needs to analyze its own structure. The identity RO enables such a reflective capability and extends it to all FTS applications.

4.3 Reification of Operations So far, we have described how host programs can navigate a graph of foreign objects. But in addition to navigation, we would like to invoke methods on foreign objects as well.

One way to reify operations is to implement an RO so that it reads binary instructions of some function in an FTS application and moves them to another FTS application where it converts these instructions to the new platform before executing them on the reified type instances. This approach is laborious and difficult to implement. Instead we instruct the RO to set all parameters for a desired function and execute it in its native FTS, and then reify the returned result. We propose a reification model where operations >, used to set and get values of reified type instances, and constitute the basis for operations on reified types. In the notation shown below, we use parentheses to specify attribute name ar of type tk. RIJ[t1]...[tk](ar)> EJ

The operation > instructs the RO to obtain the value of the particular attribute of a type and assign it to some variable EJ. Consider setting the Salary attribute of the CTO to the value of integer salary and retrieving the value of the Bonus attribute of the type CEO into the variable bonus for an instance of the organizational schema shown in Figure 1. int salary = 10000, bonus; R[“CEO”][“CTO”](“Salary”) > bonus;

Our notation can be used to reify operations in FTSs. By assigning a typed operation name to tk and treating its parameters as a set of attributes {ar}, we set values for each parameter using the operation