System integration with autonomous components

System integration with autonomous components1 Jaroslav Král, Michal Žemlicka, Department of Software Engineering, Faculty of Mathematics and Physics, Charles University, 11800 Prague, Czech Republic {kral,zemlicka}@ksi.ms.mff.cuni.cz Summary. The system integration was in the last decade based on customizable software packages like R/3 by SAP. This situation is about to change substantially due to new requirements and/or conditions to be met by wide area information systems (e.g. information systems of worldwide enterprises or information systems of state administration) and due to the possibilities offered by new middleware and by powerful network technologies. It is now technically feasible to interconnect large autonomous software units (autonomous components) being (almost) complete applications. It brings new quality to users and new technological turns and/or tool s but new problems discussed in the paper. Key words: Autonomous business components, message oriented middleware, middleware orientation, message formats, XML, middleware services and problems, wide area information systems.

Introduction The system integration is at present usually based on the customization of large packages produced by large vendors like SAP or Oracle. Customization is a quite cumbersome process. The customization determines the function to be included into the system. The properties of the functions are further modified by further parameters. It is essential that the properties of the resulting system must be inside some bounds and have characteristic properties if the package. It is often an advantage as the system use know-how of the vendor of the system and the corresponding business process reengineering can help a lot. But there are different cases. Good companies have often their specific processes and culture which is optimal for them. Customizable packages could induce no restructuring of processes that are not necessary and need not be optimal. Such restructuring can be more difficult, have worse properties, and be more expensive then in the case of the system developed from scratch. Customization is a lengthy and cumbersome process. Adaptation of such universal packages like R/3 by SAP is a difficult task. Nevertheless the customized information systems (CIS) are quite common now. Customized information systems generated from one package have very similar properties as they are based on the same philosophy. It is to a high degree valid for CIS generated from packages from various vendors. It is clear that such systems avoid competition disadvantage, i.e. missing modern information system, rather than to bring competition advantage. Prefabricated solution provided by CIS cannot bring any competition advantage, which must be based on local knowledge specific for the customer. It is possible only in the case when the substantial parts of the developed systems are developed from scratch. Several years ago it meant to write almost all the system from scratch as the packages were not suited to integrate components newly written by customers or by the third parties or by the customers. The development of the complete system was and is very expensive and requires skills often not available.

1

1 Supported by the Grant Agency of the Czech Republic, grant 201/99/0244.

This seems to be an unsolvable dilemma. But there is a way out. It is possible to use a solution, which must be used in the development of the information system of a word-wide company. Such system are worldwide, distributed and build from almost independent components. We call such system wide area information system (WAIS). Let us discuss the properties of WAIS of a worldwide company.

Information system of a worldwide company Globalization enforced rise of international companies made by network of relatively independent divisions. Such network will be called worldwide enterprise network (WWEN). The following features are typical for WWEN: 1. Dynamic changes of size. Quick changes of activity topics. Heterogeneous activities. 2. Acquisition of previously independent companies (AOL has bought Netscape). 3. Outsourcing or dividing previously compact company (example: the outsourcing of health technologies from the Hewlett Packard). Te reasons may be different: business process restructuring or a consequence of an antimonopoly trial. 4. WWEN are active in countries with different legislatives. 5. Individual divisions can differ both in the kind of activities and/or in their business culture. 6. The management of WWEN needs a wide support of analytical tools that is best to get from specialized vendors (OLAP, workflow systems). 7. The system must be open and able to communicate with unpredictable set of collaborating systems outside the company (banks, municipal or state systems etc.). These systems must be used as such provided that they have appropriate gates. 8. The system is very large. Such a system is difficult to develop and maintain in a centralized way as a monolith (as one whole). If built as a monolith then according to the known empirical laws the development time must be long (it is proportional to the third root of the number of lines of the programs implementing the system, [5,15]). The maintenance as well as modifications of such monolithic systems are very expensive 9. Continuous and seamless modification of the system is crucial. The modification process should not be too centralized, as it would cause unacceptable delays. 10. System cannot be built from scratch because there are in use a lot of information systems of various organizational subunits. The points 10, 2, and 3 imply the need of easy integration of existing applications and/or information systems (legacy systems) into the information system of WWEN. System extension should be easy. There therefore have further conditions regarding subsystems • The existing information (sub)systems must be integrated into the resulting system as such with the exception of gates and/or services allowing to make certain services and/or data generally accessible for authorized people. • The system must be designed in such a way that almost independent development of information subsystems is possible. • As the requirements of outsiders on the information system could vary substantially the subsystem interface should be easily modifiable. • Assume that an effective middleware is available. Then the information systems of particular offices can be integrated with small modifications as AC's. The modifications should provide access to the data to be exported from the systems, i.e. interface must be broaden. • Information subsystems therefore ought to be autonomous and able to integrate various subsystems. The constituent units must have various structure and functionality. A rather big autonomy of the development of individual information subsystems is needed. The reason is that some activities are needed only inside given unit. Components then must therefore be autonomous, used as black boxes, easily integratable and developed almost independently.

There are yet another reasons for such architecture. • The development of huge IS for WWEN as a monolith (almost unstructured whole) must be expensive, time consuming, and the resulting system is usually inflexible [15]. • If the subsystems are autonomous or loosely related, a possible failure of one of the subsystems need not necessary lead to more important consequences for other subsystems and for IS as a whole. (Compare WWW). • The splitting of a company or sale of one of its part is substantially easier and economically more advantageous if it is possible to sell the part together with its IS. The easy separation of such IS from the whole and the possibility to sell as little knowledge about IS of all the WWEN as little as possible is advantageous. The communication between units needs a powerful middleware that can be used also for the communication with business partners and state authorities. Middleware should also allow an easy access of end users (employees and business partners' agents) to the information system functions. The enterprises in countries with different legislative systems and enterprise cultures have consequences for the message formats. Varying rules (e.g. shape and somehow also the contents of invoice) forces that data formats have to allow transformation of the document data to different forms and partially to different contents. It is in the obvious that WWEN IS must be distributed. These facts imply following requirements: 1. IS has to be made as network of loosely coupled subsystems 2. Subsystems must be autonomous - a failure of a subsystem should not spoil the run of the system as a whole 3. The parallel run of the subsystems is necessary, this is the reason for the asynchronous communication protocol 4. There must be a powerful middleware 5. The communication should allow exchange of data together with metadata to ensure flexibility needed for the cooperation of the components in heterogeneous environments (countries) 6. Management needs various analytical tools. The easiest solution is the integration of third party products (e.g. OLAP or workflow systems). The problem of wide area information system is to a high degree the problem of the design of communication infrastructure (middleware) between components offering: • Changeable access of all users to the system. • Security services (access rights, cryptographic services, rules of data replications, etc). • Collaboration between autonomous subsystems under the condition that the individual components can be added and/or modified quite freely. • General and powerful message formats and standards for (meta)data communicated between information subsystems. • Regulations and methods how to implement communication and/or collaboration between components. The management need not take part in all negotiations. It is not difficult to see that the communication subsystem must fulfill further general requirements concerning the data replication, general firewall tools, data warehouses etc. The task of system integrator is to find an optimal balance between centralized and decentralized activities. The project management should solve the following issues. a) Middleware technology should be used. The question to what extend Internet should be used is crucial. An analysis must be performed whether this solution provides a good access to SIS for majority of citizens and enterprises and what the solution is technically feasible, secure, and not too expensive. b) The metaformat of messages must be agreed. Possible solutions are standards XML or RDF [23,30]. The formats of commonly accessible messages should be fixed (standardized). c) The rules and tool for the client access (browsers). d) Security regulations, tools, and standards.

Properties of WAIS The following requirements must be obviously fulfilled by WAIS. 1. Incomplete, contradictory often obscure, continuously growing and changing requirements. 2. The necessity to integrate third party products as well as legacy systems. 3. Changing, complex, and always growing set of users. 4. The necessity to keep the autonomy of certain subsystems, e.g. a police information system inside the information system of state administration or an information system of a newly purchased company. There are many further reasons for it. The responsibility for the correctness of data and/or operability of the particular systems must be delegated to individuals and organizations producing the data and using them. Their primary interest is to have correct and actual data and an operable system. The interests of outsiders are not so strong. The use of almost autonomous subsystems is therefore preferable. 5. The necessity to collaborate with information systems of changing and unpredictable set of partner organizations (business partners, information systems of the companies residing on the state territory, state administration) 6. The components of the system must be distributed. 7. Distributed access of users. The user interface should be easily modifiable and it should hide the system internal structure. 8. As the number of components is very large it is not feasible to coordinate and/or to confirm the communication agreements in a centralized way. 9. Continuous and seamless modification of the system is crucial. The modification process should not be too centralized, as it would cause unacceptable delays, as the bottleneck would be the central authority. It implies that the message formats (syntax) must be agreed by the communicating parties. The feasible solution is a metatool able to define all the possible variants of the message syntax. A promising solution is XML. The problem is the optimal balance between freedom in the choice of formats and standardization The points 2., 4., 5., and 6. imply that the system must be constructed as a network of large loosely coupled components - being usually (almost) complete applications - interconnected by a powerful middleware. All the components work in parallel way; all are active like the services in human society. The practice indicates that the AC interface should be based on a message passing middleware. The messages are the words in a complex language2.We call such components autonomous. An autonomous component (AC) is an independent activity (process). AC is usually a quite complex software entity. It usually is a complete application providing some gates for other autonomous components. In business a component can be OLAP or a complete warehouse control system. The inner structure of autonomous components is hidden. Autonomous components are therefore used as totally black boxes. The only external knowledge about the AC is its interface defined via a complex language. System is assembled from set autonomous components working almost independently and in parallel way (at any moment all the processes can be active and work like members of a team). Autonomous components collaborate. Their collaboration is based on the exchange of messages. The messages are words of a language having a quite complex syntactical structure (SQL, KQML, XML etc.). It follows from the requirement on parallel execution of components that the exchange of messages must be asynchronous (if a message requiring some service is sent the sender need not wait till the service is finished). Such a system is therefore based on principles similar to the principles in human society where many activities are autonomous and works in parallel [9]. The collaboration of activities is based on the exchange of messages in a complex language. The messages must be transported by a powerful infrastructure - by a middleware. The properties of middleware are crucial. Due to the important role of the middleware we shall call such systems middleware oriented systems (MWOS).

2

The use complex languages on the presentation level of the middleware service.

Middleware oriented systems. The MWOS promises to bring substantially new impulses into the whole SW industry \cite{c17}. New releases of information system e.g. BaaN [4], SAP \cite{c26}, Lawson \cite{c21}, Oracle, etc., are or are intended to be to a high degree MWOS\footnote{The progress of the mentioned systems towards MWOS is clear but quite slow. It is probably due to difficulty to manage new paradigm.}. The following properties of MWOS are obvious: • Any autonomous component (AC) can be easily replaced by another one provided that the interface is not changed. As the interface based on a complex syntax is quite flexible and structured it is not difficult to achieve the interface stability. A new AC can easily be integrated provided that the interface is modified accordingly. There is a great chance to detect the changes in the interface not agreed yet. • If an AC have form of almost independent application, it can be developed almost independently and integrated later (see e.g. Král, Žemlička [18]). It simplifies the integration of third party products as well as of legacy systems. The system is therefore very flexible and truly open. • If an AC have form of almost independent application, it can be developed almost independently and integrated later (see e.g. Král [15,16,18]). It simplifies the integration of third party products as well as of legacy systems. The system is therefore very flexible and truly open. • Middleware orientation enables techniques easing the development of AC even in the case when the previous point is not valid [16]. It facilitates e.g. the incremental development and prototyping. AC's collaborate asynchronously. They tend to be large. The components known from object oriented CASE systems and object oriented methodologies [24,27] collaborate synchronously and tend to be small. The syntax of communication in object oriented systems is quite simple. It is a sequence of method/procedure calls and returned values. The technology of autonomous components (AC) betters the user-oriented properties of the system like stability, scalability, modifiability, etc. The main problem is that AC is a new paradigm requiring a new way of thinking, new team organization and even new commercial practices. It is generally known that objects and their methods tend to be small. The object oriented programs are designed as a pool of objects communicating via method calls. The system is designed as a whole (a monolith) and then possibly decomposed into parts running on different computers (see the famous ATM example in Rumbaugh [24]). The reusability is then difficult due to the necessity to know and use the interface of huge number of small units [8]. The lack of structuring causes that only small units can be used as black boxes. The integration of legacy systems and/or third party products is then an uneasy task. It implies that the enhancement of a customized enterprise information system installed by a vendor like SAP via an integration of a component developed directly for the user in order to achieve an competition advantage is a costly and time consuming task. The communication between objects is synchronous. The object oriented programs tend to be sequential rather than a large collection of activities running simultaneously. I.e. the OO programs tend to be sequential. All these facts imply that big systems are quite difficult to develop, maintain, and modify. Autonomous components tend to be large and if a satisfactory gate to middleware services is provided, they cam be and often they must be used as black boxes of a quite complex functionality. Autonomous components are often complete or almost complete applications, for example warehouse administration subsystem. The practice indicates that the interface should be based on a language possessing a quite complex syntax. It facilitates the integration of legacy systems and third party products. Autonomous components must be supported by a powerful middleware.

Attributes of middleware orientation, an overview. The above example show the communication system cannot be treated as a service which need not be a topic of the requirements specification and/or design. The above given examples indicate that the required software architectures have a lot of common features. Let us summarize the substantial features of communication subsystems supporting the architecture based on the network of autonomous components. 1. Asynchronous peer-to-peer collaboration of autonomous components acting in parallel way like the services in human society. 2. Dynamic change of addressees. System should enable an easy replacement of components. The possibility to find the addressee according to the format (syntactic structure) and/or contents of the message. The middleware services can the mobility and cloning of components. 3. Flexible user interface. The basic properties of the interface should be defined in the system design as a part of middleware specification. 4. Many other functions of the communication system should be specified in system requirements. This is e.g. the case of message routing. 5. Dynamic configuration management. 6. The online administration tools It is important from the methodological point of view that many problems are passed from individual components to middleware (message formats, data security, universal user interface tools etc.). In other words if we use the terminology of OSI standards the middleware must cover not only the link and transport layers but also the presentation layer and some functions of application layer. There is yet another problem. One of the most difficult issue of the requirements specification is the decision what is to automate and what not [18]. Equally difficult seems to be the decision what to solve in components and what to require from middleware.

System integration with autonomous components A feasible implementation of WAIS is middleware oriented (i.e. the properties and functions of middleware influence substantially the functionality and overall quality of the system; the services and properties of the middleware should be therefore stated in requirements specification) and use autonomous components. This solution opens the way to many new powerful techniques [33]. It requires, however, a new way of technological thinking but also new marketing and/or organizational turns and attitudes. Let us start with technological issues. Development with the help of autonomous components requires skills, techniques, and software processes different from the skills, techniques and processes optimal for object oriented technologies. Incremental development is inherent to systems implemented as a network of autonomous components (NAC). The habits, techniques and processes of the customization must be changed substantially. The classical customization starts from a predefined (existing) rather monolithic package that is reduced and/or modified during the customization process. The assembling of the system based on large autonomous components used as black boxes is a quite different paradigm. The possibility that the set of components can change dynamically is not taken into account at all. The middleware issues are assumed to be solved and therefore they are not considered in the requirement specification phase. In the case of NAC a crucial technical as well as a marketing problem is the decision what components should be purchased, what services will be provided by components and what by the middleware (an example are the security issues). Autonomous components enable the integrators not to be too depended on unique package vendor. It depends, however, on the goodwill of the vendor. He must agree to open his system and to enable it to use middleware services. Another issue is how the customer can acquire a competition advantage. It can be obtained via a specific combination of autonomous components, via an unusual use of middleware, by the customization of components and, above all, by the development of some components from scratch. It is an opportunity for small SW firms. The NAC can be changed dynamically. It is an opportunity for the system integrator. He host support the continuous changes of the system. On the other hand the changes of the system are simpler, less

expensive then in the classical or traditional case. The modifications of the system are simpler and they could therefore be performed by customers. The above facts are still rather promises than an everyday reality. Any new technology requires time to be managed fully. We are at beginning. New releases of the packages by SAP, Oracle, BaaN, etc. were planned to be NAC since 1999. Although there is a visible progress this vision was not implemented fully yet. Lawson SW started implementation several years ago with a moderate progress. Nevertheless there is a good chance now that the construction of information systems could be as easy as today assembling of PC’s. The bus architecture enabled even small firms to assemble quite complex computer systems. It facilitated the mass production of computers nowadays. The middleware, XML, and autonomous components can be in software something like the bus architecture in hardware with similar important consequences.

Conclusions. Middleware orientation and autonomous components (MOAK) is a very promising, although not completely novel, paradigm. There is now time for it. Information society must be based on wide area information systems, which in turn must be middleware oriented and must use autonomous components. MOAK is now a feasible solution as there are efficient communication tools, communication protocols, flexible data formats provided by XML and programmable user interface provided by WWW browsers. Autonomous components provide not only the sole known feasible solution for WAIS. Autonomous components increase the possibilities of ‘small’ systems. It is now possible to enhance their functionality by the integration of purchased highly efficient data analysis tools, databases etc. Autonomous components and modern middleware has power to change substantially the whole software industry. It is already clear that it will bring many advantages for customers. It is open, however, what are consequences for software firms (great vendors like Oracle or Microsoft, system integrators, or small programming firms); what software firms will get the greatest advantage. The situation offers many opportunities to system integrators. They need not be too dependent on one vendor. They have several possibilities to offer new solutions to customers. The system integration with autonomous components must be based on a broader collection of skill and a broader knowledge than in the case or traditional system integration strategy. The SW market will be probably more global and more dynamic. There are still many technical and management problems to be solved [18]. The main problem is, however, to change thinking stereotypes. It is a painful, difficult, and long-term process. The duration of it could be several years (compare how much time was necessary to manage object orientation paradigm). In spite of these problems the technology of autonomous components is already usable. Autonomous components are a big opportunity for all, software developers, system integrators and large software vendors provided that they are efficient and clever enough

References [1] Aglets, http://www.trl.ibm.co.jp/aglets/index.html. URL on Aglets, the IBM variant of SW agents [2] Booch, G., Rumbaugh, J., Jacobson, I., The Unified Method for Object-Oriented Development , Version 0.8. Metamodel and Notation. Relational Software Corporation, (1995). [3] Brown, W.J., 1998, Antipatterns. Refactoring Software, Architectures, and Projects in Crisis, John Wiley \& Sons, (1998). [4] BaaN, http://www.baan.com. Home page of BaaN company. (1999). [5] Boehm, B. W., Software Engineering Economics, Prentice Hall, (1981). [6] CACM, The Promise and the Cost of Object Technology. A five years Forecast, Comm. of ACM, Vol. 38. Oct. 1995. [7] CORBA, http://www.omg.com. Home page of OMG Group containing link to various CORBA activities and/or documents and standards, (1999). [8] Finch, L., http://www.ddj.com/oped/1998/finc.htm. So much OO, So Little Reuse, Dr. Dobb's Web Site, May 1998. See also Dr. Dobs Journal, May 1998 [9] Fisher, K., Oliveria, E., Štěpánková, O., Multiagent Systems: Which Research for which Applications. (1999). To appear.

[10] Howlett, D., Lawson Adds Web and Workflow. PCUser, 16 Oct. 1996. [11] Jacobson, I., et al, Object Oriented Software Engineering: A Use Case Driven Approach, second printing, Addison Wesley, 1995. [12] Král, J., Demner, J., Software Engineering, (in Czech), Academia Praha, (1991). [13] Král, J., Software Physics and Software Paradigms. In Proceedings of 10th IFIP Information Processing Congress, Dublin, North Holland, (1986). [14] Král, J., Object Orientation in Information Systems. A useful but not Universal Tool. In Proceedings of ISD96 Conference, Gdansk, (1996). [15] Král, J., Information Systems. Science, Veletiny, 1998. 357pp. In Czech. [16] Král, J., Architecture of Open Information Systems. In Evolution and Challenges in Systems Developments, (Wojtkowski, W.G., Wrycza, S., Župančič, J., eds.), 7th int. Conf. on Information Systems, Bled, Slovenia, Sept. 21-23., 1998. Kluwer Academic/Plenum Publishers, (1999),131-138. [17] Král, J., Middleware Orientation Can Simplify the Development and the Future Use of the Czech State Information System. Proceedings of conference System Integration'99, Prague, (1999). 573-579. [18] Král, J., Žemlička, M., Autonomous components and wide area information systems. Report 10/99 of the department of software engineering, Faculty of Mathematics and Physics, Chrles U., Prague, 18 pp. [19] Lawson Software, Lawson Software Documents, 1997. [20] Lawson Software, http://www.lawson.com, 1999. [21] Moudry, J., Private Communication, 1998. [22] Plášil, F., Bálek, D., Janeček, R., 1998, SOFA-DCUP: An Architecture for Component Trading and Dynamic Updating. Proceedings of ICCDS'98, May 4-6, (1998). [23] RDF, http://www.rdf.com/.Resource Description Framework. A Proposal of the W3C consorcium. [24] Rumbaugh, J., Blaha, M., Premerlani, W., Eddy, F., Lorensen, W., Object Oriented Modeling and Design, Prentice Hall, (1991). [25] SAP, http://www.sap.com. Home page of SAP company. [26] Serain, D., Middleware, Springer V, (1999). [27] UML, http://uml.systemhouse.mci.com/artfac.htm. Home page containing links to the proposed standard of UML, 1999. [28] Warren, I., The Renaissance of Legacy Systems. Method Support for software system Evolution, Springer V., (1999). [29] W3C, http://www.w3.org. Home page of W3C consortium, 1999. [30] XML, http://www.xml.com/xml/pub/ Extensible Markup Language. A proposal of W3C consortium 1999. [31] IEEE Std1042-1987, Guide to Software Configuration Management, IEEE 1993 [32] Kr8l, J., Autonomous components. Promises and challenges of openness. Proceedings aof the DATASEM99 Conference, Ost. 24-26, 1999, Brno. Edited by Masaryk University, Brno, 1999, 191200.