Using WWW and JAVA for image access and ...

MED. INFORM.

(1997), VOL.22,

NO.

4, 291-300

Inform Health Soc Care Downloaded from informahealthcare.com by Moorfields Eye Hospital on 07/18/11 For personal use only.

Using WWW and JAVA for image access and interactive viewing in an integrated PACS E. BELLONt, J. WAUTERSt, J. FERNANDEZ-BAYO*t, M. F E R O N t , K. VERSTREKENt, J. VAN CLEYNENBREUGELt, B. VAN D E N BOSCHf, M . DESMARET?, G . MARCHALT and P. S U E T E N S t t Laboratory for Medical Imaging Research (Radiology & ESAT), Katholieke Universiteit Leuven, Herestraat 49, 3000 Leuven, Belgium f Department of Information Systems, University Hospitals Leuven, Herestraat 49, 3000 Leuven, Belgium

Abstract. We start from the observations that (1) potential benefits from PACS can only be realized if PACS and H I S are integrated into a ‘multimedia medical information system’, and (2) that this also requires integration at the level of the user interface. The user interface should allow integrated interaction with information from different subsystems, of which PACS is one. However, in the real world, different information systems are constructed using different technologies. Moreover, radiological image viewing is a highly interactive task that puts a particular burden on the graphical user interface. We describe our experiences in applying technology emerging around the ‘World Wide Web’ (WWW) for interactive access to an integrated PACS/HIS. We illustrate the use of Web browsers to access new medical services, touch technologies for interactive access to HIS-PACS information, and emphasize the potential of JAVA applets. We argue that JAVA may become an important tool for providing highly interactive user interfaces to larger multimedia information systems. We discuss Web technology in the general context of HIS/PACS integration.

Keywords : Picture archiving and communication systems (PACS); User-computer interface; World Wide Web ( W W W ) ; J A V A .

1. Introduction The purpose of Picture Archiving and Communication Systems (PACS) is to exploit computer technology in the image-based diagnostic or therapeutic process. In order to realize the potential advantages of PACS, the imaging system must be integrated into the overall information system environment, which we will refer to as the H I S (Hospital Information System). This integration is often pursued at the ‘system level’, e.g. by using DICOM (Digital Imaging and Communications in Medicine) for exchanging images inside the PACS or for sending workflow information from H I S to PACS. However, it is in our opinion necessary to extend HIS/PACS integration towards the ‘user level’. This includes the ability to access images simultaneously with reports from radiology or pathology, laboratory results, etc. This includes, also, the ability to retrieve images based on information that is maintained in the HIS rather than in the PACS [l]. As it will become increasingly common for a radiologist to produce quantitative results such as tumour volumes and delineation, it will become

* Present address: SDI-UDIAT. Consorci Hospitalari del Parc Tauli. Parc Tauli s/n, 08208 Sabadell, Barcelona, Spain. 0307-7640/97 $12.00

0 1997 Taylor & Francis Ltd

Inform Health Soc Care Downloaded from informahealthcare.com by Moorfields Eye Hospital on 07/18/11 For personal use only.

292

E. Bellon et al.

necessary to access different image processing services from within the general reporting workstation [ 2 ] . All this requires an integrated user interface to the multimedia information system. It is not straightforward to choose a software technology for building this user interface. It should be kept in mind that in practice the overall information qatem will consist of different subsystems that have been designed with different foci and have been developed by different teams, over different time frames and using different phases of technology. A potential technology for building integrated user interfaces is the one emerging around the World Wide Web (LVLVW, or ‘the Web’). This manifests itself in the increasing use of ‘ browsers ’ for a wide range of applications. T h e WWW-paradigm is platform independent and it promotes integration of different subsystems, even if large geographical distances are involved. Browser technology has very broad support from commercial companies. However, as detailed in $ 2 , most experience is available for applications that do not put large demands on the user interface itself. For large-scale routine use in medical practice, t\vo considerations are important. T h e first is that the user interface must be able to reflect the complex organization of the hospital. Although it is not obvious that the page-based metaphor of the Web is sufficient for this, we only indirectly touch this issue in this paper. A second consideration is that, especially in radiology, image viewing is a highly interactive task. It is not straightforward to present an effective overview of tens of images on a workstation. It must be possible to interactively select a range of pixel values and to map this from black to white on the monitor (a process referred to as ‘intensity windowing’). In this paper, we investigate the applicability of a Web browser as a user interface where integration of iizformation and/or interactivity on images are required. I n $ 3 , we illustrate employment of a browser as a general user interface for accessing results of novel imaging services. In $4, we focus on the need for interactivity in the selection of information, and we illustrate interactive image access for a PACS that is tightly integrated into a HIS. In $ 5 , we focus on interactivity in high-end image viewing, and we illustrate the possibilities of the JAVA programming language. I n $ 4 we discuss WWW/JAITA in the general context of PACS software development and HIS/PACS integration.

2. Current use of WWW to access clinical multimedia information Systems using the WWW for accessing multimedia medical information can be classified into three groups. X first group is directed towards teaching (often continuing education), and presents radiological cases or interactive atlases [3]. T h e user interface can be kept relati\,ely simple here, both with respect to retrieval of information and interactivity in image viewing. A second group of systems are used to experiment with WWW and browser technology to access clinical HIS/PACS databases. Encouraging initial results have been reported [4-7]. T o our knowledge, no large-scale clinical systems are used yet, and image interactivity is most often limited or provided outside the browser technologv. A third group of systems is directed towards teleradiology or tele-cooperation. WWW-based technology is in principle the same whether applied on a local area network inside a single institution (‘intranet ’) or over large distances using the

Inform Health Soc Care Downloaded from informahealthcare.com by Moorfields Eye Hospital on 07/18/11 For personal use only.

Using WWW and JAVA for integrated image access and interactive viewing 293

Figure 1 . A Web browser (right) can be used to access the results of oral implant planning in oral surgery, although the user interface for actually generating the planning data is much more demanding (left). The generic browser technology allows such planning to be implemented as a centralized telematics service for a health care region. (Images courtesy of the department of Periodontology of the Katholieke Universiteit Leuven, which participates in the research project illustrated here.)

Internet. Thus, there are little conceptual differences with the previous group. However, in practice there is more emphasis on the problem of transmission speed and on security and privacy [8]. In [9] an experimental framework is described for tele-cooperation, where the emphasis is not the same as for sheer information retrieval from a distance.

3. A Web browser as a common interface to access pre-arranged data T h e most common operation mode for browsers is to access information on a server computer that is organized into ‘pages’. These pages are sent to the browser on the client computer in the H T M L format. A page may contain links to other pages, which enables the user to access these easily. Important factors with respect to integrated user interfaces are (1) that it is conceptually irrelevant on which server a page is located, and (2) that browsers provide a common way of presenting data on virtually any client computer platform. T h e first factor enables simultaneous presentation of information from different subsystems. T h e second factor encourages developers of information systems to provide a Web interface (possibly in addition to a more specific interface), and thus encourages software development using WWW technology. Constructing a Web interface is relatively easy for those applications where users need primarily to view a set of pre-organized data. Figure 1 illustrates an application we developed for oral implant planning in periodontology. T h e planning itself is performed using a dedicated workstation and a specific user interface [lo]. However, during surgery the extended interactivity of the planning system is no longer needed and any browser can be used to view the surgery plan. T h e same technology can be used in a local area network (intranet) and with larger distances over the Internet. Especially in the context of a centralized planning service a specialized hospital may provide to other institutions, the ‘ standard ’ mechanism for information access and the availability of browsing technology on virtually any computer platform are important advantages. In the working mode illustrated in figure 1, information flow between server and client is unidirectional. Interactivity is limited to the user indicating which page he

294

E . Bellon et al.

or she \\ants to view, probably by activating a link in a previous page. However, this in itself is not sufficient to navigate through patient data in a multimedia hospital information system. I n the latter application, the user must, for example, be able to specify queries. Both a more interactive selection process and more dynamic links between different sources of information are needed.

Inform Health Soc Care Downloaded from informahealthcare.com by Moorfields Eye Hospital on 07/18/11 For personal use only.

4. Interactive multimedia access %loredynamic access to information is provided by the C G I (Common Gateway Interface) extension to HTML, which enables the server to activate a specific program in reaction to forms submitted by the interactive user. This is a prerequisite for most applications that involve database searching, and interactive filling of forms in a browser is very common nowadays in Web-based applications. If the focus is on database queries, there exist numerous tools that help in performing SQL queries nithin the C G I framework. With respect to integration at the user interface level, an important factor is that both selection of information and presentation of information can involve different subsystems, a fact that must not be apparent in the overall user interface. For example, at our institution we consider the PACS a subsystem of the H I S . Selection of patient and examination is performed in the H I S , as this process is the same for images or other types of information. Data presentation is a shared task of PACS and other subsystems in the H I S [ l l ] . T h e demonstrator setup in figure 2 illustrates WIYW-based access to multimedia information in the clinically used HIS and the experimental PAC subsystem at our hospital. H I S as well as PACS had been developed in-house, but without specifically taking W W \ Y technology into account. T h e user first searches for an examination in the H I S , and then instructs the imaging subsystem to display the corresponding images. These images are converted on the fly from the DICOSI 3 format used in the PACS database to the J P E G format that can be inserted in the dynamically generated HTML page. I n our system, HIS and PACS reside on different computers, but images and other data are integrated in the user interface. Other user interfacing technologies could provide this as well (11 but, again, it is an advantage that Web technology has broad support in the information technology community. However, in this form browser technology has limitations that make it impossible to provide highly interactive radiological viewing as described in the introduction. First, the mechanisms for user interaction are fairly limited, as construction of extensive graphical user interfaces xvas not a goal of C G I . This makes it unlikely that one could develop a user interface for efficiently navigating through many images. Second, all processing is performed on the server, while the C P U power in the client is basically wasted. For example, interactive intensity windowing of 1 2 bit image data lvould require the server to continuously calculate new ‘screen images’ out of the dataset on the server and transmit these over the network to the browser. A useful workaround for the latter lack of image interaction is to call external programs from within the \Veb browser. T h u s , the browser may load a D I C O M image series and pass it to the local program for interactive viewing. However, this gi‘ies u p the idea of total platform independence, since the external program must be ported to every platform the browser is to be used on, which in turn hinders development of widely reusable software components.

Inform Health Soc Care Downloaded from informahealthcare.com by Moorfields Eye Hospital on 07/18/11 For personal use only.

Using WWW and JAVA for integrated image access and interactive viewing 295

Figure 2. A simple demonstrator application for image retrieval out of a HIS/PACS using a standard Web browser. H I S data for this application is available in a relational database, PACS metainformation is also maintained in this database, while the images themselves are available in DICOM 3 format in a UNIX file system. The interactive user submits a query from the browser by filling a form (a)and refines his request by choosing from the information returned by he Sybase WebSQL server ( b ) . If the images of an examination are requested (c) these are converted into JPEG and sent to the browser in a new H T M L page (d). Mind that some information in the user interface is provided by the traditional HIS, while the images are provided by the imaging subsystem.

5. JAVA applets to provide more interactivity in image viewing T h e drawbacks mentioned at the end of the previous section may be eliminated by using the JAVA programming language developed by SUN Microsystems (see http ://www.javasoft.com) [12]. JAVA is an object oriented programming language, in the tradition of C , that has been developed to be network-aware. More specifically, JAVA software objects reside on the server but, when downloaded to the browser, execute on the client, within the browser. Usually, such a software object (or ‘applet’) has a graphical interface. T h e browser provides a window wherein the applet can present information and accept user interaction. JAVA extends the principle of WWW browsing in such a way that not only data in standard format are sent to the browser, but data together with the associated program to manipulate it. JAVA supports the paradigm of the network computer (NC), or the ‘Web PC ’ [13]. A complex interactive application, e.g. for desktop publishing, does not reside as a whole on the local workstation but consists of a collection of cooperating software objects located ‘anywhere’ on the network. If it is required to update a spreadsheet in the document that is currently edited, a software object that can

++

Inform Health Soc Care Downloaded from informahealthcare.com by Moorfields Eye Hospital on 07/18/11 For personal use only.

E. Bellori et al.

Figure 3 . An interactive DICOXI image v w n w running as a JAVA applet in a browser, together with user interface elements for patient/examination selection in the HIS and provisions for refining selection in the PXCS. 'The image viewing applet provides image interactivity such as control of intensity window, zooming, and presentation of different slices in stacked or tiled format, though in this erample in a rudimentary fashion.

perform this is loaded from the network. -4 complex graphical application is spread amongst different software objects that each focus on part of the task, while the Web browser glues these objects together. iYith respect to integration at the user interface level in the context of this paper, the important factor is not whether the software objects reside on the local computer or on the network. first essential factor is that this technology actively promotes the notion of a user interface constructed using smaller building blocks (JAVA applets) glued together by the browser. AA second essential factor is that JAVA enables the construction of rather elaborate user interfaces (although, in aiming at platform independence. JAY.\ provides somewhat less flexibility than, for example, the S Windon. graphical system). Figure 3 illustrates an experimental application that exploits both the possibility of integration at the user interface level and the advantage of local processing. As in figure 2 , information from HIS and P.4CS can be accessed simultaneously. Unlike in the user interface in figure 2 , the J.AYX viewing applet can now directly visualize the DICO3,I images of the PXCS archive without ha\,ing to convert them into J P E G format first, and it provides interactive image navigation, intensity windowing, and zooming using the resources of the local workstation. Interactivity in an applet can go beyond interaction inside its window in the user interface. Figure 4 illustrates a pilot project setup to enable a radiologist at the eniergenc\- \yard to ask advice of a distant expert. T h e viewing applets at the two (or inore) sites communicate via the server to stay synchronized. XI1 image manipulations at one site are reflected in the other connected applets. T h e type of platform or browser used a t either end is irrelevant in principle. In our setup, actions such as intensity windowing do not require continuous retransmission of pixels, but merely

Inform Health Soc Care Downloaded from informahealthcare.com by Moorfields Eye Hospital on 07/18/11 For personal use only.

Using WWW a n d J A V A f o r integrated image access and interactive viewing 297

Figure 4. In this setup for tele-cooperation, the JAVA applets in both browsers communicate to synchronize the information both experts perceive. Image navigation and manipulation actions in any browser are transmitted to and reflected in all other connected applets. Any participant can use the pointer to indicate the structure he or she refers to.

exchange of information to trigger local reprocessing. Synchronization of the user interfaces is accomplished by the JAVA applets, not by the browser. T h e browser provides the overall context for the applet and helps by, for example, caching images locally. Currently, an important drawback of JAVA is speed, as the platform independent intermediate program code that is sent over the network must be interpreted locally. This makes JAVA applets between 5 and 50 times slower than natively compiled code. In the example in figure 3, truly interactive intensity windowing using software recalculation to map image pixel values to screen intensities is not yet possible using pure JAVA code. In our experience, the features currently available in the JAVA framework make it somewhat awkward to manipulate ‘radiological’ images with more than 256 pixel values. T h e latter drawbacks are much less important in image management systems for ‘ non-radiological ’ medical images, in particular photographs obtained using visible light. We envisage using JAVA for applications such as endoscopy and dermatology. In these applications, the required level of interactivity in image viewing is lower. WWWIJAVA can, by design, more efficiently use ‘plain colour images’. In many situations irreversible compression can be employed ; WWW/ JAVA technology is designed to take advantage of image compression (in contrast to, e.g. the X Window system).

6. Discussion In this paper we illustrated and put into context the use of Web technology for building integrated user interfaces to navigate through HIS/PACS information. Our motivation was to integrate at user interface level various subsystems of the multimedia information system, potentially combined with image processing or

Inform Health Soc Care Downloaded from informahealthcare.com by Moorfields Eye Hospital on 07/18/11 For personal use only.

298

E. Bellon et al.

telematics services. T h e illustrated examples are from experimental developments, howe\-er. T o enable user interface level integration in large-scale clinical PACS, three requirements must be met. The f i r s t reqitirenient is suitability of LYeb technology for interactive image display. \Ye have illustrated that, though traditional browser principles are too limited to pro\.ide interactive image handling in radiology. JAVA technology may be used for developing more demanding user interfaces, though it is still not straightfonvard to do. J.AV-A has amazingly quickly obtained a strong position in the informatics industry, and most browsers support JAVA applets. JAVA is not the first attempt to provide a tool for exchanging ‘life objects’ over a network. However, JAVA seems to have been launched a t an economically suitable moment and it combines different attractive features. Amongst those are ease of software development, the fact that in the language much importance is attached to security, and the relative ease with which a browser can interpret the platform independent code. T h e main problems arise from the somewhat immature state of JAVA technology and its lower speed. With respect to speed, there is little doubt that solutions will appear. These range from processors specifically designed to run JAVA to compilers that use ‘just in time’ technology to generate native code the moment the applet is loaded. The serorid requirement is availability of a widely accepted framework for composing a complex information system (including its user interface) out of independently developed building blocks. We have mentioned during the discussion of various examples that the browser glues JAVA applets together. However, this in itself is not a strong framework. T h e example applications in this paper were developed within a single laboratory. Integrating commercial objects that have been independentl?. developed is a totally different endeavour. Even in the example applications in this paper, user interface level integration of different applets with each other, and with the brolvser, is somewhat limited: each applet comes with its own tiny user interface, relatively independent of the user interfaces of other applets in other windows, instead of adding functionality to one consistent overall user interface. 3Iuch effort is currently devoted to developing frameworks that enable construction of large applications out of a set of software objects that cooperate over a network. An influential architecture for distributed objects is CORBA (Common Object Kequest Broker Architecture). \Yhen the emphasis is on software reuse by integrating relatively weakly coupled components, the notion of a ‘ component architecture’ arises, and integration can be pursued at the user interface level. Influential component architectures that aim at giving the interactive user a consistent user interface in which software building blocks of different vendors can cooperate are OLE/XctiveX (llicrosoft proprietary) and CORBA/OpenDoc (originated b y Apple and further developed by a consortium of manufacturers, though with a rather insecure future a t the moment). We refer to [14] for a detailed discussion. T h e use of J.AYA applets in a browser must be situated within these dei.elopments. JAVA allows objects on different computers to cooperate (or more basically to coexist) and JA17A addresses integration at the user interface level. Dea\.elopment of complex information systems will be encouraged by the fact that J.41rAI has recently been enriched with its own methods of remote object cooperation,

Inform Health Soc Care Downloaded from informahealthcare.com by Moorfields Eye Hospital on 07/18/11 For personal use only.

Using WWW and JAVA f o r integrated image access and interactive viewing 299 and that integration between JAVA and CORBA has been made possible. Development of multi-vendor user interfaces will be encouraged by the recent emerging of ‘Java Beans’ as a component architecture for JAVA. The third requirement to come to clinical multi-vendor systems is the willingness of PACS/HIS manufacturers to provide building blocks that can be combined into an overall high-end user interface. WWW/ JAVA has the attractive feature of platform independence and vendor neutrality. However, this issue is not so much about the choice of technology or architectural framework, but rather about attitude. There still is a difference in attitude amongst many imaging companies on the one hand, and providers of more general hospital information systems on the other hand. Many of the former tend to propose a PACS as a self-contained system with its own user interface that is not meant to be part of a more general user interface. If they use the term ‘integration’,they refer to information exchange at the level of DICOM or HL/7, not to composing a complex interactive application out of different components. H I S vendors more naturally regard PACS as an extension of existing systems, but often lack specific image related know-how. Probably, the next generation of successful PACS products will emerge from companies that are able to combine expertise in imaging and the possibilities to acquire user requirements in the real environment, with new visions in software integration.

Acknowledgements Most of this research was performed with support from the Flemish Government in a project of the Flemish Institute for the Promotion of Scientific-Technological Research in Industry (IWT) (project ‘Adding images to an integrated multimedia patient record ’). We acknowledge the support of Agfa-Gevaert NV, Belgium.

References 1. BELLON,E., FERON, M., VAN DEN BOSCH,B., BOGAERT, J., HOUTPUT,W., VERSCHAKELEN, J., LAUWERS, P. SUETENS, P. and MARCHAL, G., 1996, Integrating digital ICU viewing into the global working environment. European Journal of Radiology, 22, 221-227. 2. BELLON, E., FERON, M., MAES,F.,VANHOE,L., DELAERE, D., HAVEN, F., SUNAERT, S., BAERT, A. L., MARCHAL, G. and SUETENS, P., 1997, Evaluation of manual vs semi-automated delineation of liver lesions on C T images. European Radiology, 7 , 432438. 3. RICHARDSON, M. L., 1995, A World-Wide Web radiology teaching file server on the Internet. American Journal of Roentgenology, 164, 479-483. 4. LOWE,H. J., ANTIPOV,I., WALKER, W. K., POLONKEY, S. E. and NAUS,G. J., 1996, WebReport: a World Wide Web based clinical multimedia reporting system. Proceedings of the A M I A Annual Fall Symposium, 26-30 October 1996, Washington, DC (Philadelphia: Hanky & Belfus), pp. 314318. 5. BROWNING, G. C., LIANG, Y., BUCKWALTER, K. A,, KRUGER, R. A. and AISEN,A., 1996, World Wide Web interface to digital imaging and communication in medicine-capable image servers. Journal of Digital Imaging, 9, 178-184. 6. MASCARINI, G., RATIB,O., TRAYSER, G., LIGIER,Y., and APPEL,R. D., 1996, In-house access to PACS images and related data through World Wide Web. European Journal of Radiology, 22, 218-220. 7. RATIB,O., MASCARINI, CH., TRAYSER, G., REVILLARD, C . , LIGIER,Y.and APPEL,R., 1996, The “ intranet ” concept applied for convenient access to PACS image data and computerized medical record. Proceedings of the 14th International EuroPACS Meeting, 3-5 October 1996, Heraklion, Crete, pp. 31-33. 8. LI, X., VALENTINO, D. J., So, G. J., LUFKIN,R., and TAIRA, R. K., 1996, A World Wide Web Teleradiology PACS. Proceedings of the SPZE, 10-1 5 February, Newport Beach, California, abstract and poster nr. (Washington: SPIE), pp. 2711-53.

Inform Health Soc Care Downloaded from informahealthcare.com by Moorfields Eye Hospital on 07/18/11 For personal use only.

300 Using IZ'U'W a n d J A V A f o r integrated image access and interactive viewing 9. CHROSAKI, C., Z.ABVLIS, S . and ORPHASOLDAKIS, S., 1996, XIaintaining Image Annotations in I?Cnet. Proceedings of the I J r h International EirroPACS a14eeting, 3-5 October 1996, Heraklion, Crete (Philadelphia: Hanley & Belfus), pp. 141-145. 1 0 . VERSTREKES,K., \'.AS CLEYSESBREVGEL, J., MARCHAL, G . , NAERT,I . , SUETENS, P. and VAN STEENBERGHE, D., 1996, Computer assisted planning of oral implant surgery : a three-dimensional approach. The International Journal of Oral and Maxillofacial Implants, 11, 806-810. 1 I . BELLOS,E . , V.AS CLEYVESBREL-GEL, J.. \-.AS DES BOSCH,B., >I.AEs, F., FERON, M., HOUTPUT, W., DELAERE, D . , YASKTGIERDES, >I., MARCHAL, G . , SVETENS, P. and BAERT,A . L., 1994, An integrated environment aimed at exploiting new possibilities of digital imaging technology. Proceedings of the I2th International EuroPACS Meeting, 22-24 September 1994, Geneva, Switzerland, 1 2 . COSIERFORII, R.. 1997, Software engineering. I E E E Spectrum, 34, 65-69. 13. HALFHILL, T. R., 1996. Inside the \Veb PC. Byte, 21, 44-56. 14. ORF.AL.I, R.. HARKEY, D. and ED\V.ARDS, J., 1996, The Essential Distributed Objects Suroioal Guide (New York: IViley).