The typical client-server strategy is that ASP pages contain ... 5 Note, this strategy allows the ASP developer to determine what browser features the client ...
Image Management and Dissemination Systems under MS DNA: Towards Rapid Deployment of C3I Applications for Defence Missions Alistair Barros (DSTC), John Hildebrandt (DSTO) and Andrew Goodchild (DSTC) Distributed System Technology Centre (DSTC) Level 7, GP South, The University of Queensland, Brisbane QLD. 4072. AUSTRALIA http://www.dstc.edu.au Defence Science Technology Organization (DSTO) Fernhill Park, Department of Defence, Canberra ACT. 2600. AUSTRALIA http://www.dsto.defence.gov.au
Abstract Rapid deployment of C3I applications for transient operational sites is a time-consuming task involving the integration and adaptation of many custom pieces of software. Microsoft offers a well integrated range of COTS products and resources like document managers and workflow management, collaboration tools, and rapid application development technology which improve the productivity of developing and customising C3I applications. Current perceptions, however, are that the Microsoft platform suits front desk applications only. This paper explores whether Microsoft technologies are functionally mature enough to accommodate loosely-coupled, distributed applications primarily of a backend nature. The test case used is the Image Management and Dissemination system, currently Java/CORBA based, which allow autonomous and potentially worldwide image libraries to centrally register for geospatial image searching and retrieving in defence missions. 1
INTRODUCTION
In order to support the gambit of command, control, communication and information (C3I) requirements for defence missions, a large number of complex and independently developed applications needs to be rapidly integrated, deployed and adapted for specific situations at transient operational sites. Essentially, C3I requirements include collaboration and coordination support, (dynamic) reference to heterogeneous information repositories, document and database management, and feedback monitoring - for mission planning and execution. In principle, a deployment of C3I applications should be a straightforward migration exercise. In practice, however, as described through the United States Department of Defence’s (DOD) Global Command and Control System (GCCS) experiences [GAO, 1998], the C2 legacy proves cumbersome for adaptation in new situations, severely restricting deployment. This problem is especially pronounced in missions having multiple executors where a C2 configuration is formed from heterogeneous and previously unrelated sources (i.e. defence agencies in different countries). As C3I systems plans have increasingly engaged COTS products, with development efforts redirected into component infrastructures to maximise reuse of applications, a new problem has emerged. This is the integration burden resulting from the sheer heterogeneity that a fully component-oriented approach advocates. An open “market” of COTS products requires integration over a diverse set of platforms and architectures which support the products. This has resulted in large amounts of “glue” code, i.e. integration legacy, becoming significant development efforts in their own right. Moreover, a technical integration does not guarantee integration at the application (semantic) level. In an attempt to mitigate just some of these problems the DOD has created the vast bureaucratic effort known as the Defence Information Infrastructure Common Operating Environment [DII, 2000]. A practical way forward for building transient operational sites is to leverage COTS products whose functionality and interoperation is familiar ground for stakeholders of defence missions. In other words, the platform of COTS products should be widely used, and mission deployment, at least theoretically, should be reduced to custom
Page 1 of 12
adaptation of C2 generating tools. Under DII COE, this strategy is framed under experimentation of Microsoft (MS) technologies for the rapid generation and adaptation of C2 systems [CCRP 2000]. The MS platform offers database, document and workflow management, human collaboration, and high-productivity application languages among other tools indispensable for the C2 domain. The hard-wiring of document, workflow and proprietary-specific data management logic in monolithic C2 applications, gives way to thin, pliable application layers which make use of sophisticated services in COTS tools. Importantly MS allows its technologies to be harnessed across the Web through its Distributed Internet Architecture (DNA) [Kir, 1999], its support for extended Markup Language (XML) data exchange through the Simple Object Access Protocol (SOAP) standard. More recently MS has announced the development of the .NET [CDFG, 2000] architectural framework which is designed to support the development of more loosely coupled Internet based applications. Taken together, this set is comparable with competing distributed infrastructure standards in the OMG Group’s CORBA standard and Sun’s J2EE standard (for a comparison of MS DNA services with those from CORBA and Java see [Szy, 1997]). To date, the MS experimentation has been undertaken by the Defence Sciences and Technology Organisation (DSTO) to expose MS utilisation in its C3I infrastructure, EXC3ITE. Early results have demonstrated rapid development of functionally rich C3I component-ware, however the exposition of MS technologies for heavy-duty backend applications remains open; indeed the area presents grave uncertainties for MS assimilation in business and database communities. One of the experiments, which is the subject of this paper, is test out the backend waters through a mission-critical C3I backend application: the IMAD (Image Management and Dissemination) system. IMAD is a distributed application (prototype) deployed on the EXC3ITE architecture. Like a number of other DSTO/EXC3ITE developments, it is Java and CORBA based. IMAD allows geospatial images from a federation of image libraries, managed through heterogeneous database systems, to be queried and appropriately retrieved by clients. The clients range in network bandwidths, compute power and target exploitations of images. IMAD departs from traditional client-server applications in the autonomous nature of image libraries. At any time, image libraries may “choose” to be available or unavailable. This warrants a naming service which allows registration and deregistration of image library servers. IMAD fosters this through a CORBA trader. In general, all IMAD components are trader located, enabling a distributed and flexible interaction of the system. Furthermore, the heterogeneous nature of clients and their image retrievals, has led to the utilisation of a dynamically determined “intelligence” for optimal image transmission from server to client. In this paper, the range of MS technologies pertinent for IMAD are examined. These include n-tier services in: Internet Information Server (IIS), Microsoft Transaction Server (MTS), Active Directory Service (ADS), Microsoft Message Queue (MSMQ) and SQL Server. These technologies are critiqued and applied where possible into a version of IMAD re-architected to support DNA. In doing so, some enhancements are introduced to the current IMAD design for operational transient sites. For instance, the current system coerces the interfaces and image exposition details on potentially world-wide set of image libraries registering with IMAD. Alternatively, the new design seeks preserve the autonomy of image libraries. The structure of the paper is as follows. Section 2 describes the architecture, components and operation of the current IMAD system. Section 3 provides an overview and examination of MS DNA services relevant to IMAD. Section 4 re-architects IMAD under MS DNA, with possible extensions under .NET are commented upon. Finally, section 5 concludes this paper by summarising findings and describing future work.
2
IMAGE MANAGEMENT AND DISSEMINATION (IMAD) BACKGROUND
IMAD, written in Java 1.2.2, provides a number of services, which allow clients to access images from a federation of image libraries, whereby image libraries can dynamically register and de-register themselves in the federation. Each image library comprises image files and a relational database storing image metadata, such as latitudes and longitudes of an image’s bounding box and image file locations. Queries to image libraries are formulated through SQL via JDBC. The architecture of IMAD is described in Figure 4.1. Each image library server is accessed via a GIAS compliant interface1. In general, the middleware infrastructure of IMAD is CORBA based2. Services offered and used by
1 2
GIAS is a CORBA specification from the US National Imagery and Mapping Agency. The default installation requires Visibroker.
Page 2 of 12
IMAD are accessible via a trader3. Recently, IMAD was extended to include multiple traders where the local domain of a trader corresponds to a node in EX3CITE. Each IMAD component registers its service type, node name and service name. When a local trader receives a request for a service type, it passes the request to currently linked remote traders. It then returns object references of its results as well as those from the remote traders.
IMAD services IQS
IDM
CORBA services
Clients
Trader
Image libraries
Level 1
Library 1
Level 2
Library n
Level 3
CORBA Transport Layer Figure 1: IMAD systems architecture Clients access imagery data through specific applications on a client workstation or via web based applets. On system startup, an image4 of the world map is displayed with a latitude and longitude grid display. Selection of an area of interest occurs through a user-specified bounding box. Clients vary in their access needs for images, in their compute and image display capacity, and in the bandwidth between them and the source library server. Accordingly, IMAD classifies clients into three levels. Level 1 clients typically involves “out-on-field”, laptop access over slow RF network links (such as low power satellite transmission, mobile telephony, etc). They have limited display and compute capabilities. As such, client software is pre-installed for Level 1 clients and image transmission typically involves tiling, compression and progressive encoding. Level 2 clients, being LAN based with high-speed connectivity and low compute-power, and requiring images for “backdrop” purposes, are supported through applets. Level 3 clients have high compute and display power for detailed analysis. This involves full exposure of each image being analysed in an image exploitation package where display structures like image analysis cells correspond to image libraries. Level 3 clients have network access with FTP facilities. The IMAD process is two-fold. The image query part determines the image libraries that contain images overlapping the user-specified area of interest. The image dissemination part follows user-specified selection of which libraries to target. The image query part is as follows. A client first contacts the local trader to obtain an Image Query Manager (IQM), essentially a factory object, for connection to an Image Query Service (IQS). Queries formed from input fields of the client interface are then submitted to the IQS. In order to perform the query, the IQS obtains references to all available libraries from the local trader. The IQS then instantiates the QM as a thread which creates threads to send the query to each library. At the same time, the IQS determines quality of transmission information for the libraries via a Transmission Availability Forecaster (TAF) and sorts the library list based on the bandwidth returned. The QM determines "distance" between client and image library using the TAF – acting as a network quality of service facility. The IQS then waits for the QM to return. After the results are returned from the QM, the IQS goes to each the target libraries to obtain required image metadata. The hit-list formed ignores duplicate images (from different libraries). Finally, the IQS inserts the hit-list into a distributed query request object which is returned to the client. The client uses the distributed request object for user display.
3 4
The trader is Java Trader written by DSTO and compliant with the CORBA trader specification. Images are rendered via a JavaBean
Page 3 of 12
The image dissemination part follows when a client requests an image. An image dissemination service (IDS) is determined by a Context Manager, again a factory. The Context Manager appropriates the quality of transmission in an image dissemination service generation, using the likely network bandwidth between the client and the library server, its level, the size of the image (pixels) and the size of the region of interest. The TAF is used to retrieve the bandwidth (since this might have changed since image query time). This is possible through TAF agents located at each library server. Currently FTP and socket-based image retrieval services can be generated. Currently, CORBA based transfer is not considered feasible.
3
MS DISTRIBUTED INTERNET ARCHITECTURE (DNA) TECHNOLOGIES
DNA rests on COM+, which is based on the Component Object Model (COM) and has extended with additional interfaces to support component lifecycle management and distributed transactions. COM/DCOM means that all services are exposed through component building blocks (via interfaces) to the presentation, business and data access layers of applications, including the services which support the layers themselves.
Figure 2: The Windows DNA architecture Recently, inter-process communication in heterogeneous, distributed environments has been enhanced through the protocol, Simple Object Access Protocol (SOAP). SOAP is a transport layer neutral protocol which enables XML encoded messages to be exchanged across a number of mediums including HTTP and email. Commonly SOAP is transported across HTTP, bypassing the usual non-HTTP blocks imposed by network firewalls. DCOM, by contrast, can only practically work on an Intranet since it relies on dynamically assigned ports for remote method invocations. By providing programmatic access to services over the Internet using SOAP the concept of Web Services has been developed. A Web Service offers programmatic access to some backend service via standard web based API’s. Both IBM and Microsoft are supporting the SOAP standard to develop a Web Service environment and market and are shipping products to build Web service based systems. The SOAP standard has been submitted to the W3C to develop a formalised Web XML protocol. To describe the interface provided by a SOAP service the XML based Web Services Description Language (WSDL) has been developed and submitted to the W3C. Once one has a collection of Web services in a large environment the issue of discovering and locating these services will arise. To facilitate this the Universal Description, Discovery and Integration (UDDI) service interface has been developed by both IBM and Microsoft. It is likely that while Microsoft will still support the development of COM+ systems across a single homogonous enterprise the preferred cross enterprise communications mechanism will be via these XML based protocols. 3.1
Presentation layer
At the presentation layer, DNA incorporates a range of user interface types, including HTML, dynamic HTML (DHTML) via Internet Information Server (IIS) and native Win32 applications. Key to supporting these user interface types on different hardware device types is the COM model and client- and server-side scripting, and other client infrastructure services. For Web based clients, Internet Explorer 5.0 supports HTML, DHTML, client-side scripting, Java applets and Microsoft Active X controls. Active X controls (similar but not functionally equivalent to sessions beans under Enterprise JavaBeans) provide a presentation layer componentry including a deployment (container) mechanism.
Page 4 of 12
Client-side scripting allows the style, content and structure of a Web page to be changed without contacting the web server. Languages include those which plug-in to MS scripting architecture, which currently includes VBScript, and Jscript (a JavaScript like language). DHTML, which allows the content of pages to be generated dynamically, provides an open, language-independent object model for HTML. Dynamic content from remote data sources is optimised by passing data between client scripts and server applications, rather than by passing entire pages. Internet Information Server (IIS) Internet Information Server (IIS) 5.0, provides the execution of server-side ASP pages for client-response generation. When IIS (INETINFO.EXE) receives a request, it examines the resource being requested (e.g. ASP page), and if it is available, loads the assigned ISAPI extension (e.g. the ASP parser, ASP.DLL) using the registry’s mapping of resource file name extensions to ISAPI extensions and forwards the request to it. The resource extension processes the space page and then sends it to IIS which in turn returns it to the client. The traditional encapsulation of content generation code within clients and interaction through mechanisms such as the Common Gateway Interface (CGI) or Internet Server Application Programming Interface (ISAPI), is now contained within Active Server Pages (ASP). ASP programming, like client-side programming, combines HTML and a scripting language (IIS provides scripting engines for VBScript and JScipt). More complex logic could still be encapsulated through COM components, of course. The typical client-server strategy is that ASP pages contain server-side code which invoke middle-tier components, which in turn access data resources to obtain the required data. ASP generates the HTML response which in turn is returned to the client5. Given its widespread adoption, XML as an application data exchange mechanism is increasingly complementing HTML6 as a means for enabling server to server communication. Indeed XML forms the basis for business-tobusiness communication in Microsoft' s BizTalk framework7, and as mentioned above, XML transport over HTTP is being pitched as the interoperability standard under SOAP8. When combined with MSMQ messaging, a robust, distributed communications model is possible. At the client end, an XML document is validated against an XML schema and posted to a ASP page9. With the emerging .NET framework the Web based user interface component is referred to a Web Forms which will replace the ASP framework.. Within the .NET framework all code is compiled to MS Intermediate Language (MSIL) and then Just In Time (JIT) compiled before execution. This includes server side web pages and is claimed to result in significant performance improvements.
3.2
Business Logic Layer
The business logic or middleware services layer provides various services, exposed in a unified way through COM, which provides software interaction (“plumbing”) and execution support. Core services are discussed in the remaining section.
Microsoft Transaction Server Run-time execution services, through Microsoft Transaction Server (MTS), combines object brokerage and transactional execution of object updates across distributed sites. As a whole, this means MTS provides services for the creation of an object by a client(s) and its multi-access synchronisation. This includes the polling, allocation and deallocation of shared resources like system processes, threads and memory in the presence of large numbers of requests and large numbers if COM objects. Also included is multi-object update consistency across distributed resources (like database and file managers) including connection management to these resources. In doing so, MTS allows different components to map to different servers for security or fault isolation but currently does not address load balancing across different machines.
5
Note, this strategy allows the ASP developer to determine what browser features the client supports and appropriate the HTTP response accordingly. 6 xHTML is an XML compliant version of HTML 4.0. 7 See http://msdn.microsoft.com/xml/articles/hess061499.asp for an announcement article in MIND magazine. 8 See http://msdn.microsoft.com/msdnmag/nettop.asp?page=/msdnmag/issues/0300/soap/soap.asp&ad=www.msj.com/fea ture2.shtml for an announcement article in MIND magazine. 9 This can be done through XMLHTTPRequest object – which provides the leanest possible HTTP POST forms without compression.
Page 5 of 12
For object brokerage, MTS intercepts object creation (meaning that MTS will be assigned in the registry to be launched when the object is requested). In this way, MTS can maintain the object (execution) context and a context wrapper which allows MTS to sit between the object and its client(s) thereby observing all calls to the object. Resource management is provided via the object context and context wrapper. MTS checks security constraints to prevent unauthorised clients from using components. When a method is invoked on the object, the context wrapper creates the actual object which stays in memory until the object indicates that it can be deactivated. Note, all components using MTS services must be in-process (on the same node as MTS) so that all objects can be created within MTS-managed server processes. Transactional support for components is set through the registry. This signals MTS to create transaction boundaries for update consistency by the component’s operations (MTS uses COM containment to add transaction boundaries and COM aggregation to add transactional interfaces to objects as required). The server also detects references to other objects in update operations and accordingly extends transaction boundaries to these. To coordinate multi-resource updates, MTS uses the two-phase commit protocol through the MS distributed transaction coordinator (MS DTC). The MS DTC uses a COM based protocol, OLE Transactions, to communicate with resource managers for enlisting in transactions, transaction propagation across processes and systems and for two-phase commit. Alternately, X/Open’s XA standard API is used by most TP-monitor technologies for transaction and resource manager communication. MS have adopted OLE Transactions for more of object-based, and therefore multi-resource type based, transactional interface, unlike XA which intended for pure database interaction. Moreover, X/Open do not support recovery initiated by a resource manager. However, the DTC in addition to supporting OLE Transactions, also supports XA, allowing it to coordinate transactions between MTS and other 3rd party resource managers.
Microsoft Message Queue Asynchronous messages enables loose-coupled interaction between applications and components on different machines. This can be achieved through Microsoft Message Queue (MSMQ). MSMQ uses point-to-point rather than wider-grained publish/subscribe messaging. MSMQ messaging works as follows. The calling application opens a queue and sends an XML message. The message queuing software routes the message to its destination. At the destination, an application watches the incoming queue for messages (asynchronously or synchronously). MSMQ API provides functions for creating, opening and deleting queues; locating existing queues and messages in queues; and sending and receiving messages. MSMQ provides an OLE Transactions resource manager so that queues can be transactional, meaning that the act of sending a message to the queue or receiving a message from the queue is transactional. MS bridges to IBM environments are available through the COM Transaction Integrator (COMTI) for IBM CICS/IMS and MSMQ to IBM MQ bridge. These together with the interoperable nature of OLE and XA DTP open Windows DNA to heterogeneous interaction.
Active Directory Service Trader federation and functionality can be addressed through Microsoft Active Directory10, which has been released as part of Windows 2000. Active Directory is a global namespace for locating and regulating access to computing resources, organized into Internet Domain System (DNS) style hierarchies. Shared resources such as servers, printers, shared volumes, applications, user accounts and domains are stored as objects in Active Directory databases. The Active Directory service is based on the X.500 object model and the Lightweight Directory Access Protocol (LDAP). Active Directory objects are globally uniquely identified (by 128-bit number GUIDs) defined through essentially object-oriented schemas, which allow standard classification (classes and attributes), association and inheritance structures. Schemas are extensible permitting application or organizational semantics to be attached to computing resources, e.g. user account objects can have a purchase authority limit. Also, group policy objects, for computer and user objects, permit access control to be administered through the Active Directory Service. This is a major advantage over a CORBA Trader provision (on its own). A further advantage of Active Directory is its integration with other network management resources. Through its support for Windows 2000 operating systems directories, applications level directories such as in Exchange and 10
Active Directory Technical Summary, available at: http://technet.microsoft.com/cdonline/Content/Complete/windows/win2000/win2ksrv/technote/actdirts.htm
Page 6 of 12
Web folders, it can be seen that Active Directory provides a central management and interfaced facility for a variety of sources. Key to this strategy is its incorporation of LDAP as a core protocol. This allows interoperability with other directory services – through Active Directory Service Interfaces (ADSI). MAPI RPC protocols are also being supported. The alignment with the Internet standard, Domain Network Service (DNS), is also managerially significant. DNS allows location and connection of computers on TCP/IP networks (through a domain name to IP address mapping). The DNS hierarchical naming structure consists of a single root domain, underneath which can be parent and child domains. Windows 2000 domains and computers have similarly structured DNS names, and so correspond to Active Directory objects. Thus, it can be seen that Active Directory has been retrofitted with global Internet DNS namespace. This means that Active Directory clients use DNS to locate Active Directory domain controllers. Each Active Directory domain has a single domain controller (multiple controllers in a single domain provide fault tolerance and load balancing - they share the same directory data). The root of the Active Directory namespace (the forest root) provides the LDAP entry point to Active Directory. With the emerging .NET framework the use of the UDDI service is another option for service discovery. If due to requirements for loose coupling and to remove infrastructure constraints from systems participating in an IMAD federation then the use of Web services may be considered rather than COM+ (or CORBA) interfaces. In this case the discovery method employed would be UDDI rather than a mechanism tied to a specific middleware platform. Hence if the IMAD federation is intended to be uniformly based on libraries offering a COM+ interface then the use of Active Directory for discovery would be a likely choice. However if the requirement is for a loosely cross organisational federation where one can not assume the technology base at each location then Web services and UDDI discovery would be a likely path to follow.
3.3
Data Layer
At the data layer, MS’s Universal Data Access (UDA) strategy provides abstracted interfacing and manipulation of structured and unstructured heterogeneous, distributed data resources. A number of components, known as MS data access components (MDAC), implement the UDA strategy. Data access to heterogeneous sources is standardised through the OLE DB11, which is a specification of a set of COM interfaces for data access management by different data providers. Essentially, data providers expose abstract table structures known as row sets which provide generic access to data. The data provider is responsible for mapping row sets to internal data structures be they SQL tables via the Open Database Connectivity access, nonSQL data like mail, video, text typically formatted through XML or directory services data, or proprietary gateway access to mainframe data. Under OLE DB, an object hierarchy is available for flexible (late-binding) access to data sources. OLE DB also allows loose-coupling of data access management functionality through service components: query processors, cursor management and transaction management. This means that a data can be accessed through a data provider but the actual querying, sorting, filtering, navigating etc. could be enabled through separate service components. Similarly, a service component may be used to standardise that service functionality like query processing for all data providers. OLE DB also provides other features for optimal, distributed query processing such as connection pooling, asynchronous support (in the request to execution cycle), and distributed queries (building a result set through a single OLE DB service from multiple data sources). UDA defines an application-level programming interface through Active X data objects (ADO), which is built on top of OLE DB, and thus abstracts its functionality further still. For example, the connection object maps into a unique session to a data source, and it exposes a standard (Execute) method for performing simple data access tasks. Connection objects can also be attached to Command (similar to OLE DB Command objects) and Record set objects (similar to OLE DB Row set objects) which provide other data access functions. The typical route to data is not through OLE DB but through ADO. The flagship database technology under MS is SQL-Server (now in version 7). SQL-Server provides relational functionality, conforming to the ANSI/SQL92 standard. It also provides different forms of distributed processing 11
OLE DB/ADO: Making Universal Data Access a Reality, available at: http://msdn.microsoft.com/library/backgrnd/html/msdn_dbado.htm
Page 7 of 12
including client-server (ODBC, JDBC, OLE DB and particular gateways), replicated server and distributed database management. Under version 7, SQL-Server provides some object-relational support such as binary-large objects (BLOBs) and user-defined types (and methods). In the geospatial domain, SQL-Server has demonstrated itself to be capable of dealing with large volumes of geospatial data in the TerraServer project. The MS TerraServer12 stores aerial and satellite images of the earth in a Microsoft SQL Server database served to the public through the Internet. It is the world’s largest atlas, combining five terabytes of image data from the United States Geodetic Survey, SOVINFORMSPUTNIK, and Encarta Virtual Globe. However, a technical subtlety needs to be pointed out. Physical files associated with database objects are kept under SQL Server’s control. A database administrator can control file systems and disk partitions allocations for tables, indices and their storage spaces. However, the individual physical file mappings are under the SQL Server Data Manager’s control. Thus, it is not clear how a further mapping between physical files and those of dedicated storage resources, e.g. large-scale, active (hierarchical) storage device systems, would be achieved. This means that the benefits of image query through SQL Server query needs to be balanced with the performance problems which will come with accessing large image files, without dedicated storage resource management. 4
RE-ARCHITECTING IMAD UNDER MS DNA
A re-conception of IMAD under MS DNA is depicted in Figure 3. Recall, the first tier consists of client applications, the middle tier covers objects that implement business rules and logic, including the generation of client display, and the third tier covers data providers. Through this proposal, a new version for IMAD is apparent which facilitates desktop tools integration – the key advantage of the MS environment - for transient operational situations. In principle, it is possible to reuse current IMAD client software in an MS environment and use COMCORBA bridges for interoperation with a Java-based middle tier. Such a configuration, however, provides a specific type of integration and adds to the platforms required on deployment. Therefore, a case for a MS re-architecture is investigated.
Client Tier
Middle Tier IQM
Web Client (IIS +ASP)
VB/VC++ Windows Client
Data Tier
IIP & SOAP over HTTPS
DB 1 RDS
OLE -DB
DB 2
IDM
DB n
MTS + MSMQ
Active Directory Service
Figure 3: Architecture of IMAD under MS DNA
Applied to IMAD, all clients including the three default ones (levels 1, 2 and 3) can be placed in the client tier. The Image Query Manager (IQM) and Image Dissemination Manager (IDM) can be placed in the middle tier. Asynchronous communication for component interaction is introduced through MSMQ. Naming and discovery of all components and other resources is introduced through Active Directory, which permits optimised image searching (described below). The various data sources can be placed in the data tier. As described earlier the architecture in Figure 3 assumes a uniform MS DNA technology base across the IMAD federation. If a more loosely coupled technology neutral architecture is required then one may provide Web service (SOAP) interfaces for the middle tier components and use UDDI interfaces for service discovery. Note the underlying implementations could still continue to employ MS DNA components. 12
Microsoft TerraServer, available at: http://msdn.microsoft.com/library/backgrnd/html/msdn_msterra.htm
Page 8 of 12
Client Tier Current IMAD client JavaBeans and Java classes could be wrappered into Active X Controls relatively straightforwardly – the outer interfaces would map into the inner interfaces. The only problem would be local “peculiarities” in the Java interfaces, e.g. if Java objects are present in interface signatures. This would preclude non-Java outer interfaces. Re-implementation of clients under an MS IMAD version can be implemented using a variety of approaches. Level 1 clients can be implemented using a combination of: Internet Information Server (IIS), Active Server Pages (ASP) and ActiveX components (both in the web page and embedded in server side ASP scripts.) Level 2 clients can be implemented as stand alone windows applications using either Visual Basic (VB) or Visual C++ (VC++) and ActiveX components. Level 3 clients can be supported by implementing specialised tools to export data in the appropriate format using either VB or VC++ and ActiveX components. In order to facilitate reuse, there should be a core set of ActiveX controls that can be shared between two or more types of client. Secure communication between the client tier and the middle tier can be facilitated via SOAP, which permits XML transport over HTTPS. The dynamic location and discovery of the appropriate middle tier services can be facilitated by the Active Directory service. Middle Tier The middle tier contains the Image Query Manager (IQM) and Image Dissemination Manager (IDM). MTS is used as an object broker for instantiating the IQM and IDM. To ensure that requests to a data source wrapper do not exceed the capacity of the backend data sources there should be a hard limit set on the number of simultaneous connections (for example limited to the number of connections in the connection pool). Additional requests should be stored in MSMQ. Also, requests in the message queue could be prioritised, e.g. users with more important requests (e.g. senior ranking officers) having their data demands met first. Finally, the message queue can enable work asynchronicity to be built into the system. For example, a mobile user may submit a query for processing, disconnect from the network, and reconnect later to fetch the results. A major enhancement introduced for transient operational sites is centralised searching. A single AD service is proposed for the IMAD federation, which could easily evolve into a hierarchy of ADs for improved performance and resilience, as the number of image libraries significantly increase. A central source of image library searching is undertaken by the IQS, in contrast to its current broadcast mode of searches to all image libraries. To enable this, all image library servers would be registered as objects within an AD schema. Along with the domain name and IP address attributes, all exposed meta-data for the image library would be also be stored in the object’s type definition. Thus, an image object (with the metadata as attributes) would be associated with an image library object which in turn would be associated with an image library server object. Note, the distinction introduced between image library server and image library objects. This provides a number of flexibilities. For example, a number of image libraries (databases) can share the same image library server under this scheme. Also, the image library server and image libraries can be manipulated as separate resources. For instance, an image library can be blocked from access while the server can still continue to be accessed - for asynchronous query (message) forwarding to the server. Under this configuration the IQS and IDM processes would work as follows. All image objects in the AD would be searched to satisfy the user query. Under an object scheme (as opposed to a relational scheme), this would involve a sequential search of all image objects – which would be fast if the AD was cached. All satisfying image objects would be stored in the hit-list together with the image library and server references, which would be displayed for levels 1-3 clients. In the IDM phase, the IP addresses of the servers and image library names would be used for query “connection” purposes. These would be obtained by reinterrogating the AD. Recall, the reason for reinterrogation is that the image library and/or server may disconnect or change locations between the IQS and IDM operations. A central directory for image searching means that image updates in a library would have to cascade to the central directory. Libraries, under strict autonomy, however, should be free to do so “as they please”. Thus, some versioning mechanism could be introduced into the Active Directory to reflect the currency of the updates on an image as well as a image library basis. This would be used to either flag the IQS or the user for local image library searches, to ensure currency of the result sets. For instance, the library might expose the fact it has undergone major updates which are not reflected in the directory. Either the IQS would then expand its search to the image library or it would notify the user (as a status field in the result-set), leaving it up to the prompt the expanded search.
Page 9 of 12
An important consideration for rapid deployment is that not all “imported” image library servers would be expected to be structured similarly , i.e. all having the same meta-data set for describing images. Thus, a flexible, dynamic schema mechanism would have to be applied in Active Directory to allow for different image object types to be associated with the image libraries. All exposed meta-data peculiar to an image library would have to be “fed” back to the user for the searching specification. Such flexibility should, of course, be balanced against the extra search overhead incurred. Data Tier OLE DB OLE DB providers allow rapid injection of heterogeneous image library servers into an IMAD deployment because the data access protocol is more generalised than the relational interfaces of OBDC/JDBC. The assumption is that providers of the sources provide effective maps between OLE DB data structures and their local data structures. Note, that the efficiency of driver bindings with respect to the database interface manager can vary from provider to provider and needs to be tested out. In terms of configuring and using OLE DB, it should be run as its own separate process and accessed via the Remote Data Source (RDS) COM interface. The main reason for doing this is to enable connections to be maintained by the connection pool even after objects instantiated by the IQM or IDM have completed their task. A second reason is that it will mean less overhead as each instance of the IQM and IDM will not need to have their own copy of OLE DB. RDS COM interfaces can be advertised in the Active Directory Service and dynamically located at run time by the IQM or IDM. “Aggregation” of various back-end systems can also be introduced through OLE DB – through its distributed query service. With this service it is possible to query a number of different back end systems with a single query. SQL-Server In the current IMAD architecture the image data is held on a file system. One possibility is to fully exploit SQL Server’s storage and access for both meta-data and images files. The main reason for doing this is simplified management. SQL Server offers a suite of administration and management tools that can make tasks such as backup, replication, linking and federation of data sources much simpler. In particular, the distribution, data export and replication facilities provided in SQL Server allow larger storage configurations of image libraries, allowing clustering of libraries where a set of libraries share a single schema but have separate replicates (like a distributed database) or where separate image libraries have the same physical storage (for management economy). A motivation for clustering is the provision of tight-coupled, transient operational site. Reliance on SQL Server functionality for such situations provides a single point of control for replication/distribution where updates are automatically cascaded. Another benefit is that specialised image content search functions can be seamlessly integrated with standard database queries. Under a separated storage strategy, metadata (database) and image (file) searching will be separate and lower level programming will be required for the integration of “result sets”. A database querying language allows pre-written as well as custom-supplied functions to be specified as part of the query, thereby accommodating a large and typically unforseen query requirements. It should be noted that full query optimisation will not be possible under this approach. Moreover, the querying database image objects is likely to be slower than in direct image file access because certain aspects image querying will be controlled by DBMS’s data manager, e.g. caching some parts of the image object. The main problem of including images into a database is the loss of control at the file system level to fully exploit large-scale hierarchical storage devices. Such file managers have their own schema, and it is not clear how image files can be mapped in a one-to-one fashion into elements of these schemas. DBMSs typically allow larger grains of storage areas like file systems and partitions to be allocated to individual tables/indices or tablespaces/indexspaces. This mapping does not easily lend itself to mapping into more particular file storage schema which provides clustering between image sets, images, tiles and various granularities thereof, and physical elements of storage devices. Clearly, a dedicated provisioning is required by a DBMS, which in turn requires a joint collaboration by database vendors and large-scale storage device vendors. This remains an open issue [VLDB, 1999]. Added to this is the more open issue of image content querying. The same levels of content search cannot be guaranteed for images as with standard database records. Content in an image is subject to uncertainty depending, among other things, on the quality of image and the structural recognition capacity of contained and targeted objects.
Page 10 of 12
As apparent through current research efforts [NRT, 1999], the incorporation of uncertainty in image searching and the provision of general purpose image query languages, still remains open. 4.1
Value-Added Considerations
Dedicated Sites The idea of dedicated sites is to provide sub-IMAD systems for particular operational missions. As with infrastructural, logistical and personnel deployment for missions like East Timor, applications and databases are often pre-fabricated out of existing resources as supporting services. Under this business practice, subsets of image libraries and even search results (using Web folders) could be cached into a cluster of image databases which are updated as their master image libraries are updated – within the scope of the sites (e.g. only image within a geospatial boundary or containing particular geospatial features). For such a configuration, SQL Server masterslave replication strategy could be set up, using a predicate to specify the replication, i.e. site, scope. A further change to IMAD would be to allow for the specification of a site scope. This means that the IQS and IDS would only operate on image libraries in that scope. Such a scope could easily be incorporated into the IMAD Active Directory Schema as a separate objects associated with image library objects or as one of their attributes. Image Data Servers In image information systems both the size of individual images and the number of images in a collection may be very large. To manage this volume of information a number of commercial and experimental systems are being developed to store, manage and disseminate image information. For level 3 clients it is assumed an FTP mechanism is used to transfer data to the client. For other client types one will require a more dynamic mechanism that delivers image data on an as required basis is needed. Some of these systems are described in the following subsections and all support delivery of imagery in a multi-resolution on demand basis. Incorporation of these addresses efficient image transmission in distributed applications over the Internet. Flashpix and IIP The Digital Imaging Group has developed the Flashpix Image format and Internet Imaging Protocol (IIP) for distribution of high-resolution imagery in Internet like environments. This format and protocol support tiled multiresolution imagery and access to data only as required. Imagery servers using IIP are available commercially for distributing high-resolution imagery. Several commercial imagery products have built in support for Flashpix The Flashpix format stores imagery in multiple resolution levels to allow quick access to the image at multiple resolutions over network connections. Within each resolution level the image is tiled to allow fast access to arbitrary regions within the image. Each tile may be stored in uncompressed or JPEG compressed form. IIP enabled The IIP specification supports socket and HTTP connections. The protocol supports requests for regions and resolution levels within the image data. Some processing operations such as affine transformation and contrast enhancement can optionally be supported by the protocol. This protocol allows a URL based request to specify the image to display together with the resolution and region required. Web pages (or other clients) can use this protocol to construct dynamic pages that allow the user to pan and zoom through very large images without excessive bandwidth requirements. An IIP server has been used in this project to allow display of very large images. ActiveX components are available from third party suppliers such as HP's OpenPix. MrSID MrSID is a commercially available wavelet based image format that is available from a single vendor. The use of wavelets avoids the need to store multiple resolution levels and tile the image. This format allows the viewing of very large raster datasets on standard hardware. Viewers and plug-ins for Internet browsers are available and several third party vendors support the format. To create MrSID imagery a commercial product from the developer of this format must be purchased. An image server, browser plug-ins, and JAVA applets are available for using MrSID imagery in Internet environments. ERMapper Image Server The ERMapper Company has also developed a wavelet based compression method and web based delivery protocol. Plug-ins for web browsers and several commercial products are freely available. An image server is available to deliver imagery compressed in their format across the Web on an on demand basis.
Page 11 of 12
Automated, Dynamic Meta-Data Generation In the core model architecture, image searching takes place through a global Active Directory with local schemas at image libraries allowing for more refined searches as well as image dissemination. There is no constraint that these schemas be static, i.e. predefined. Indeed this would be undesirable for the global schema which has to incorporate a variety of metadata exposures from image libraries. A partially dynamic strategy is to allow exposures at fixed times, e.g. as part of the nightly maintenance run of the Active Directory server. Certainly, this strategy follows the static nature of image metadata generation (which is mainly manual). A fully dynamic strategy would be to allow metadata to be generated “on the fly” and exposed at any time to the Active Directory global schema. This solution would become more compelling as image content searches become more feasible, where image scans could be undertaken for pre-conceived searches, and then both local and global schemas would be updated (in two phase commit). From a functional point of view, interfaces at both the image library and Active Directory levels would allow updates to their schemas - the caller requiring the update simply hands over an image update script through the interface. Performance and security issues however loom. Schema updates should at worst block searches to images of the updating image library object. A versioning strategy for schema updates would ensure such an efficiency. A security role model would be require to authorise which users could undertake schema changes. 5
CONCLUSION
In this paper we have shown how Microsoft technologies could be applied to deliver a backend C3I service. This demonstrates that the Microsoft architecture now has the required services to construct backend services. To explore the performance and details of such an implementation the next step will be to develop a more detailed design and a prototype implementation. A description of the components of the DNA architecture relevant to constructing such a service has been provided and an indication of how the .NET architecture may modify the implementation has been offered. One of the risks of the DNA approach is tightly integrated vendor lock-in. Many of the tools used in this paper come from a single vendor. Should any one component prove to incapable of meeting operational or functional requirements, it may be difficult to remove such a component and replace it with another vendors component without significant ripple through affects on other components. In future work we wish to explore decoupling this vendor lock-in by exploiting MS .NET technology. In particular, we will explore how .NET XML web services exposed through SOAP can be complemented with SOAP wrappered J2EE, COM+ and CORBA services. Furthermore, we will examine how these SOAP services advertised in UDDI can be taken advantage of in the rapid deployment of C3I applications. 6
REFERENCES
[CCRP, 2000]
A. Barros, C. Herring, J. Hildebrandt, and M. Rees, “Generating Command and Control Systems in an Hour: The Microsoft Way”, 5th International Command and Control Research and Technology Symposium, Australian War Memorial, Oct 24-26 2000.
[CDFG, 2000]
J. Conard, P. Dengler, B. Francis, J. Glynn, B. Harvey, B. Hollis, R. Ramachandran, J. Schenken, S. Short and C. Ullman, Introducing .NET, WROX Press 2000, ISBN 1-861004-89-3.
[DII, 2000]
Defence Information Infrastructure Common Operating Environment Home Page, http://spider.dii.osfl.disa.mil/dii/
[GAO, 1998]
U.S General Accounting Office, “Defense Information Superiority - Progress Made, but Significant Challenges Remain.”, NSIAD/AIMD-98-257. 08/31/98 (Letter Report), http://www.access.gpo.gov/cgi-bin/getdoc.cgi?dbname=gao&docid=f:n198257.txt.pdf
[Kir, 1997]
M. Kirtland, Designing component-based applications, Microsoft Press, Redmond, Washington, 1999.
[NRT, 1999]
S. Nepal, M.V. Ramakrishna, J. Thom , “A Fuzzy Query Object Language (FOQL) for Image Databases”, Proceedings of the Sixth International Conference on Database Systems for Advanced Applications, April 1999, Hsinchu, Taiwan.
[Szy, 1997]
C. Szyperski, Component Software: Beyond Object-Oriented Programming, Addision-Wesley, Harlow, United Kingdom, 1997.
[VLDB, 1999]
Notes from VLDB’99 industrial stream, Proceedings of the Very Large-Scale Database Conference 1999, September 1999, Edinburgh, Scotland.
Page 12 of 12
