An XML based framework for enterprise application

An XML Based Framework for Enterprise Application Integration Vishnu S. Pendyala, Simon S.Y. Shim, Jerry Z. Gao Internet Technology Laboratory1 Department of Computer Engineering San Jose State University, CA, USA [email protected], [email protected], [email protected] Abstract Enterprise Applications are increasingly moving towards providing web access to their users through "thin clients". There is a strong case for their integration using weboriented technologies. This paper brings out the need for a framework for web-based Enterprise Application Integration and evolves it. The framework provides a very flexible architecture for seamless integration of enterprise applications using available technologies. XML is used as a complete solution for EAI. This paper describes the design of the framework and explains the implementation of a prototype based on the framework.

1. Introduction Enterprise Applications have grown in number and complexity, as the enterprises and their needs grew. Companies have been sinking enormous investments into deploying enterprise applications in a big way. The requirement now is to integrate the existing and upcoming enterprise applications in a seamless fashion, for effective e-business architecture. It would have been ideal, if all the enterprise applications had a single user interface in the front-end and a single database at the back-end, with each application implementing its respective business logic. The applications would then be acting like individual components of a single, huge application, which implements a wide gamut of business operations. There would not be any duplication of data, no transformations to the data would be required, and more importantly, the users would be trained for a single interface. But as it turns out, the world is not perfect. There are no major standards in the enterprise application world and each vendor has come up with configuration that best suits its architecture and requirements. For all these applications to exchange information and work together, there is a need for another application. 1

This new application handles the differences in the schema and storage structures among the enterprise applications in a business environment. This paper presents an XML-based framework for EAI. Since the framework is based on XML, it derives all the inherent strengths of XML. The “EAI Suite”, as this new application is called, moves data between one application to the other, while ensuring integrity, consistency, performance and other transaction processing constraints. A framework is a template that can be repeatedly applied to different situations. It is quite common in business modeling and design, that some of the patterns of relationship and constraints occur repeatedly. The scenario in case of this project is application integration. An enterprise could be using several applications, like eCRM, ERP, and SCM at any given point of time. Each of these applications could be provided by one of the many vendors like Siebel or Amdocs for eCRM, PeopleSoft, Oracle or SAP for ERP, Keenan, Portal for Billing, etc. If we picture the permutations and combinations of the different implementations of the enterprise applications, the repetitive process of integrating them comes out clear. A framework for Enterprise Application Integration (EAI) is therefore, highly desirable. A framework contains several modeling constructs described using text and figures. In addition to being reusable, frameworks provide an additional advantage of envisioning the situation in its entirety and figuring out missing links, much ahead of the actual implementation.

2. An XML based framework for EAI Adaptability and flexibility have been the key considerations in evolving the framework. The framework is therefore general enough to suit a wide variety of scenarios. It suggests an open architecture and spans the four crucial areas of EAI: communication, data, process and monitoring these three through human intervention.

The Internet Technology Laboratory is supported in part by grants from Intel, Microsoft, and Informix

Proceedings of the IEEE International Conference on E-Commerce (CEC’03) 0-7695-1969-5/03 $17.00 © 2003 IEEE

An efficient, secure and robust communication mechanism for enterprise applications to exchange information is the backbone of the framework. Data is the second pillar of EAI. The third important aspect of EAI is the co-ordination and sequence of the flow of information. The fourth requirement of EAI is the provision of human intervention for controlling and configuring the way information is exchanged between applications.

that finally happens, the enterprise data has to be converted back and forth from DBMS’s to XML. The XML Interface provides this functionality. It converts data from conventional databases to XML on one side and vice versa on the other. Th e E A I F ram ew ork

CR M U ser B u siness Inter- L ogic face D ata

SFA

XML interface

E vent Editor

2.1. Architecture Overview The application integration architecture revolves around the envisaged framework. Figure 1 gives an overview of the components of the framework. As can be seen, the framework is XML-based. EAI is achieved through XML and related technologies. The components of the framework are: a) XML Interface, which is part of the enterprise applications b) Enterprise Integration Engine, which handles the core functionality and c) EAI portal, used to configure and define events and processes. Simply put, the XML interfaces of the applications handle DBMS to XML conversion and vice versa, while the Enterprise Integration Engine transforms XML data in one format to the other, based on the inputs statically generated from the event and process editors. The applications shown in the figure could be anything like CRM, ERP or SCM and they could be web-based, constructed around the Client / Server paradigm, or legacy mainframe applications. The framework encompasses the XML interface, built as part of these applications, together with the Enterprise Integration Engine and the GUI-based editors for content and process. These components handle the aforementioned four crucial areas as described in the next few sections.

2.2. XML Interface Enterprise applications like CRM, SCM, ERP or a Billing application are the existing players in the framework. Each of these applications consists of a User-Interface (UI), a database, and the business logic for the respective enterprise function. The new framework envisages an XML Interface, as part of the applications, to facilitate integration. This interface manages the “data” aspect of EAI. In legacy applications and the software integrating them, this component has typically been implemented using a different middleware such as CORBA; RMI or COM. Enterprise data has conventionally been stored in data stores provided by major DBMS vendors such as Oracle and Informix. The trend will soon change the storage systems to XML and Schema repositories. Until


Integration En gine (translations and m essagin g)

Inv entory U ser B u siness X M L Inter- L ogic interface face D ata

XML interface

B usiness User L ogic

In terface

Da ta

P rocess Editor B illin g

Ev ents rep ository

P rocess M od el R epo sitory

XML interface

B usiness User L ogic

In terface

Da ta

EA I Portal

Figure 1. Components of the EAI Framework

2.3. Enterprise Integration Engine At the heart of the framework is the Enterprise Integration Engine. This handles the communication aspect of EAI. It handles the transformation of the schema and data between applications for the XML interfaces to move the data to and from the conventional databases. The Enterprise Integration Engine facilitates the exchange of transformed information either through the conventional messaging and queue management, or through an XMLbased transaction-processing (TP) monitor. XML-based TP monitor is a project in itself and is beyond the scope of this report. The former is often referred to as “fire and forget” message mode and the latter is called online mode. The business logic in the data / schema conversions is provided to the Enterprise Integration Engine as XML data, produced by the process editors. The transformations themselves are specified in the stylesheets produced by the data editors, which are used by the Enterprise Integration Engine.

2.4. The EAI Portal The portal comprises of tools to graphically define, configure, map and capture business data idiosyncrasies and data flow sequence, along with the repositories to store the resulting schemas. This is where end-user of the EAI suite has a provision to monitor and control the business processes and data flow. The tools collectively meet the third and fourth requirements of EAI, namely, workflow implementation and human intervention. All three editors are envisioned as point-and-click, configuration only tools with absolutely no programming

effort. The portal can be managed by an “EAI admin” belonging to the IT department. 2.4.1. Process Editors for Data Integration. The Schema mapping editor provides for the user to create sufficiently large, complex and static maps of the XML schemas for each permutation of the enterprise applications. It is capable of generating XSLT stylesheets from the mouse clicks and minimal data inputs provided by the user. The XSLT stylesheets act as rules-based maps for the data transformations. The Value Transformation Editor is similar to the Schema mapping editor and captures the value transformations such as (firstName, lastName) to (lastName, firstName) etc through a GUI and stores them in stylesheets. 2.4.2. Event Editor for Process Integration. This tool specifies the business semantics for message exchanges. It is a tool for the workflow technology. It handles the definition and configuration of message exchange sequence and graphically models the business processes. Processes can be configured and reconfigured “on the fly”, dynamically using the tool. The tool maps trading contracts and agreements to process configuration and coordinates the flow of logic among the integrated applications. Just like in case of data editors, IT staff can use point-and-click techniques instead of programming to specify processes and events. The resulting business process definition schemas can be stored in a repository and reused. New applications and interactions can be introduced as variants of the process definition. Business process definition schemas can be stored in a repository and reused

2.5. Repositories Conventionally, repositories of data have either been DBMS tables or files in a file system. XML, so far, has been residing in file systems. Pretty soon, file systems will reach a point of breakdown because of file access load, that XML operations demand. The framework in particular, depends excessively on efficient implementation of repositories, because of the repeated use of the information stored in the repositories shown in figure 1. The repositories should ideally use storage structures, which are more amenable to the querying mechanism of XML technologies. XSLT processors, to be used in the Enterprise Integration Engine would immensely benefit from such a storage structure.

3. Design of a Prototype A Prototype is the first and primary type of anything created. It gives an improved understanding of the actual system before it is built. Prototyping also helps in


straightening the ambiguities present in the requirements. To prove the effectiveness of the framework evolved, this chapter describes the design of a prototype system for EAI.

3.1. A Scenario The design of the prototype can be better described, based on a scenario. The scenario can then be put in proper perspective, using appropriate technologies in conformance with the framework evolved earlier. Let us consider the scenario of a retail seller of consumer electronics, such as PCs. When a new order comes in, the customer information is entered through the CRM GUI. Purchase Order details and customer details are updated in the CRM database, by the CRM application. The XML interface extracts the product information from the CRM database and generates corresponding XML file. The Enterprise Integration Engine, driven by the XML process data generated by the process editors, then takes this XML file, transforms the required data fields and passes them over to the XML interfaces of other applications. The Enterprise Integration Engine passes the product information to the XML Interface of the Inventory control application, which makes the necessary JDBC calls to Event as XML JDBC Data

JSP

JDBC XML

XML

Data

DOM

DOM XSLT SFA Vocabulary

eCRM Vocabulary Stylesheets

Figure 2. Prototype Architecture

update its database to reflect that the specified product is marked for shipping and then kicks off the business logic to ship the product. The Enterprise Integration Engine supplies the sales person information, in form of a transformed XML data, to the XML Interface of the SFA application. The XML Interface of this application updates the database, so that the sales person is credited with the commission due to him or her and the relevant sales history record is updated. Next application, which receives the transformed XML data from IE, is the billing application. The customer information received by the XML interface of the billing application is used to create a new record. The product and service information, and the payment method provided to the billing application, is used by the internal logic to generate an invoice. All the XML Interface is supposed to do is to update the database and make relevant calls to the business logic functions.

3.2. Prototype Architecture The goals of the prototype system are to remain simple in design and still be able to demonstrate the significant aspects of the framework to bring out the advantages of the framework and also demonstrate that the framework is applicable. For the purposes of the prototype, only the database and the user interface of the Enterprise Applications are considered. The components of the prototype are:

individual modules of the EAI suite. Components (iv) and (v) are part of the EAI portal (vii). Event in this context is a sequence of processes and process is a sequence of data transformations. While the event editor works for process integration, the process editor works for data integration. The components are illustrated in figures 2, 3, and 4. The design considerations for each of the components are described below.

3.3. Data Fetch Converter i) Data Fetch Converter for fetching data in RDBMS and posting it as XML ii) Enterprise Integration Engine for schema and data transformations and messaging iii) Data Store Converter to convert XML into RDBMS data and store it iv) Editor for Schema / Value transformation (Process Editor) v) Editor for Process Configuration (Event Editor) vi) Schema templates for processes and data. vii) EAI Portal for easy access to the Event / Process Editors and invocation of the above modules. Components (i) and (iii) form the XML interface of an enterprise application. The rest of the components are Data Value Data Value

This part of the XML interface should read the data from the database using the JDBC API and ideally construct a DOM tree, based on the schema of the database. The node names correspond to the field names of the schema, and the values correspond to the data in the data in the database fields. The advantage of using a DOM tree is extensibility and ease of conversion to XML. Representing data in form of a DOM tree make it easy to incorporate changes to the database schema. A new field coming into the schema, for example, can easily be added to the DOM tree, by creating a new node. A schemaspecific application, on the other hand will have to be modified all over, to accommodate any schema changes.

Data Value Data Value

Table 1. Templates for the XML and the DTD generated by the XML interface

Once the DOM tree is constructed, generating XML follows automatically. The element names are mapped to the DOM tree node names, which are same as the field names of the database. Similarly, the values of the XML elements correspond to the values stored in the database. Table 1 shows a template of the XML and the corresponding DTD, which needs to be generated. The XML file has 3 levels: is the outermost level, followed by and . The slots in red


italics are filled in by the data fetch converter, based on the database schema and values. As can be seen, the algorithm is quite straightforward. For the purposes of the prototype, the application specific vocabulary, shown in figure 2 contains only the names of the schema. In case of the prototype system, the generated XML file is kept in the directory of the generating Enterprise Application, so that the Enterprise Integration Engine would know from where to pick the in-file.

3.4. Event Editor The Event Editor is used to capture the sequence of processes that need to be evoked on the occurrence of an event. This is a java application with a friendly GUI to elicit the required information. A block diagram of the Event Editor is shown in figure 3. [corresponding stylesheet].xsl [corresponding stylesheet].xsl
(Step+ )> id CDATA #IMPLIED name CDATA #IMPLIED > Table 2. Event Editor: XML template and the corresponding DTD

The GUI should contain the following: • Login label and corresponding input field • Password prompt and input field • Login button to be pressed when the user information is entered • Event Id label and input field, for the event that is being defined. • Event Name label and input field • List of processes, multiple of which can be selected by the user. • A “New Process” button to invoke the process editor screen, if the listed process does not contain what the user wants to be done. • Reset, Exit and Done buttons which have the usual meaning. The Algorithm of the Event Editor is as follows: • Construct the process list to be displayed on the editor screen, from the table stored by the process editor on the disk.


Process Configurator

XML (for use by the integration engine)

User Input

Figure 3. Block diagram of the Event Editor

• Authenticate user by checking the login/password information. • Ensure that the Event Id field, filled in by the user is unique. Else, prompt the user to choose a different Id and suggest some. • If the New Process button is clicked, show the GUI of the Process Editor. Once the process editor is done, the control comes back to the Event Editor GUI. • Fill in the XML template shown in Table 2, with the fields entered by the user. The fields in red are the ones which need to be filled-in. To get the .xsl filename of the corresponding user selected process, look up the mapping table generated and maintained by the process editor. • Display the resultant XML file on the screen and store it in the Enterprise Integration Engine’s repository, which for now is a directory, with the filename as [Event Name].xml. The DTD file for the generated xml is events.dtd, shown in the figure, is the same for any event, so is statically determined The template can be easily understood. An event has a name and an id, given to the event element as attributes. It comprises of several steps, each of which contain several processes. Each step, like event, has an id and a name, given as attributes. The process element does not need to have any attributes. It just contains the name of the stylesheet as its value.

3.5. Process Editor The Process Editor is actually, part of the java application written for Event Editor. A process, in this context is the set of data manipulations that need to be carried out as part of the data integration. The Editor is basically a user interface to capture the data / schema transformations and the SQL query to be performed on the destination database. Therefore, it has four screens for the four DML operations of SQL: Select, Update, Delete and Insert. For sake of brevity, we consider the Insert and Update screens only, here. Other screens can be implemented similarly. The following are the input fields that are common to all the screens:

• Process description, which goes into the process list displayed by the event editor • Source Application, displayed as a menu from which the user has to select one • Destination Application, also displayed as a dropdown menu • Destination table name, on which the final SQL operation needs to take place • Reset, Exit and To Event Editor buttons, having their usual meanings For the Insert Screen, the following input fields are necessary: • Source Database Field, whose value needs to go into the second application • Destination Database Field, which needs to be updated with the value from source database field • The Value Transformation that needs to be performed on the value of the source database field. This is a statically determined drop-down menu. S c h e m a / V a lu e T r a n s f o r m a t io n E d it o r

XSL S ty le s h e e ts (f o r u s e b y th e in te g r a ti o n e n g in e )

CRM SFA M ap (U s e r In pu t)

Figure 4. Block Diagram of the Process Editor

For the Update screen, each destination field could contain the value of an expression derived from source and destination fields. The screen should accordingly contain all necessary input fields to facilitate this: • Destination Field • Source / Destination Fields aligned as an expression around • Operators, like +, -, *, / etc • Label for “where” • Destination Field which has to be equal to • Expression containing source / destination fields, like described above

The template is quite simple to understand. It specifies what to do when the required elements and attributes occur. For example, when the root element occurs, a


Process: Process Description, as entered by the user Update DestTable
operator