A review of intelligent content-based indexing and browsing of ...

Health Informatics Journal http://jhi.sagepub.com

A review of intelligent content-based indexing and browsing of medical images L.H.Y. Tang, R. Hadka and H.H.S Ip HEALTH INFORMATICS J 1999; 5; 40 The online version of this article can be found at: http://jhi.sagepub.com/cgi/content/abstract/5/1/40

Published by: http://www.sagepublications.com

Additional services and information for Health Informatics Journal can be found at: Email Alerts: http://jhi.sagepub.com/cgi/alerts Subscriptions: http://jhi.sagepub.com/subscriptions Reprints: http://www.sagepub.com/journalsReprints.nav Permissions: http://www.sagepub.com/journalsPermissions.nav

Downloaded from http://jhi.sagepub.com at PENNSYLVANIA STATE UNIV on February 6, 2008 © 1999 SAGE Publications. All rights reserved. Not for commercial use or unauthorized distribution.

A review of intelligent content-based indexing and browsing of medical images L.H.Y. Tang, R. Hanka and H.H.S. Ip

Dr Rudolf Hanka MA, MSc, PhD, FRSS, MIEEE Head, Medical Informatics Unit Director of the Centre for Clinical Informatics University of Cambridge IPH Robinson Way Cambridge CB2 2SR Tel: 01223 330303 Fax: 01223 330330 Email: [email protected]

Physicians are beginning to be able to gain access, through the Internet, to the world’s collections of multimedia medical information such as MRI (magnetic resonance imaging) and CT (computer tomography) image archives, videos of surgical operations and medical lectures, textual patient records and media annotations. New techniques and tools are needed to represent, index, store and retrieve digital content efficiently across large collections. In this review, we trace the development of visual information systems for healthcare and medicine from Picture Archiving and Communications Systems (PACS) to the recent advances in content-based image retrieval, whereby images are retrieved based on their visual content similarity – that is, colour, texture, and shape. Medical images, unlike consumer-oriented images, pose additional challenges to content-based image retrieval, in that visual features of normal and pathological images are typically separated by only subtle differences in visual appearance. Intelligent image retrieval and browsing therefore requires a combination of prior knowledge of the medical domain, image content and image annotation analysis. To this end, we also overview the I-Browse project, conducted jointly by the Clinical School of the University of Cambridge and the City University of Hong Kong, which aims to develop techniques which enable a physician to search over image archives through a combination of semantic and iconic contents.

Lilian H Y Tang BEng, MEng Honorary Research Associate, Department of Computer Science, City University of Hong Kong

INTRODUCTION

Professor Horace H S Ip PhD, CEng, CPhys, MIEE, MinstP, MIEEE, MHKIE Professor & Head, Department of Computer Science City University of Hong Kong, Tat Chee Avenue, Kowloon, Hong Kong Tel: (852) 2788-8641 Fax: (852) 2784-4916 Email: [email protected]

Medical Informatics Unit, University of Cambridge IPH Robinson Way Cambridge CB2 2SR Tel: 01223 330319 Fax: 01223 330330 Email: [email protected] Correspondence to: Lilian H Y Tang

The advent of the Internet has opened up the possibility of globally accessing medical information from disparate sources. As physicians are beginning to be able to gain this access, there is a clear need for new tools and techniques to efficiently represent, index, store and retrieve this digital content.

Health Informatics Journal (1999) 5, 40-49

Visual data such as videos and images play an important role in medical diagnosis and training. The ability to express spatial and visual queries and to search for relevant visual information from image archives poses a major challenge for the development of advanced image indexing. Searching technologies must not only be robust but also be capable of scaling up to large collections across wide networks.

Research issues in medical image retrieval With the advent of advanced medical imaging modalities such as CT, MRI, ultrasonograms, etc., images obtained by these new techniques as well as by conventional means (such as histological slides) have become indispensable in medical diagnosis. At the same time, with the development of digital libraries, global communication infrastructures and image databases, large collections of medical images depicting various pathological cases can be accessed conveniently and remotely through high-speed networks. The availability of these large collections in recent years has shifted the problem from information availability to accessibility, that is, how to find the images we want easily and quickly. Content-based image indexing and retrieval aims to automate some of the manual process associated with indexing and retrieving large collections. However, since medical images of a given class (e.g. histological images, chest Xrays, or cardiac ultrasound images) are very similar and differ only in minute detail, it is expected that current content-based indexing techniques based solely on specific image characteristics, e.g. texture, colour, shape, may not be sufficiently precise for medical images. Investigating the feasibility of medical image indexing based on several conceptual levels of image content seems the key to developing an intelligent medical information system. Technologies must also be developed to understand and easily search across databases, handling variations on several levels of abstraction, for example, primitive image features and semantic meanings as well as high-level concepts, and combine the different levels of conceptual and iconic features when necessary. An advantage of indexing images based on multilevel contents rather than solely low-level features such as texture and colour is that it would readily provide the basic framework required for ‘semantic interoperability’ when one tries to search through not only one but a federation of image collections from different disciplines. Achieving this will require breakthroughs in image description as well as in image and object retrieval protocols.

Downloaded from http://jhi.sagepub.com at PENNSYLVANIA STATE UNIV on February 6, 2008 © 1999 SAGE Publications. All rights reserved. Not for commercial use or unauthorized distribution.

A review of intelligent content-based indexing

Research issues should include the definition and use of images and computational techniques for detecting objects from images; the integration of heterogeneous collections with disparate semantics; the organization and management of images; and algorithms for automatic rating, ranking, and evaluating perceptual properties. For such goals, some forms of semantic switching will be required in order to search for images among these collections on the basis of conceptual similarity rather than syntactic (keywords) similarity. Such a system may sometimes place a human being in an interactive loop; someone who helps improve the system performance by refining the query through relevant feedback. In addition, not only one feature detector will be developed even for one set of data, as different levels of image features and different types of knowledge should be combined for image retrieval.

VISUAL INFORMATION SYSTEMS FOR MEDICAL IMAGES The technologies for image processing, communications and medical data management are converging. The development of a picture archiving and communication system (PACS) has acted not only as a bridge between these divergent areas, but has also improved healthcare by providing a unified infrastructure for managing and co-ordinating the acquisition, archiving, communication, and display of multimodality image data and video.

Picture archiving and communication system (PACS) Since 1982, numerous efforts have been made in the area of PACS after the term was proposed during a conference of the Society of Photo-optical Instrumentation Engineers (SPIE). In PACS, unlike traditional data management methods, digital images can be copied and exchanged without loss of information, allowing different clinicians to consult them independently or co-operatively. Furthermore, images can also be retrieved faster, facilitating disease diagnosis based on the correlation of images from different radiological modalities. PACS research came from but also later contributed to developments in related areas such as high-bandwidth communications, database management, and multimedia integration. It has accordingly had an indirect impact on medical informatics

41

research, e.g. in the areas of telemedicine, clinical databases, computer-aided instruction, and patient simulation [1]. Although many PACS projects have been developed, making certain progress in some important aspects of the complex overall system, few of them have been fully accepted by the user community. This is not only because of technological factors such as the limitation of network and image processing techniques, but also economic factors as the equipment for PACS were so expensive that few hospitals and departments could really afford it. Recent technological advances in computers and networks, the development of lower-cost medical equipment, and the presence of recognized and implemented standards, have helped to encourage the adoption of PACS. PACS is beginning to be a cost-effective solution for many hospitals. One example of PACS that makes use of high-performance computers was introduced by Friscione et al. [2]. The PARCS system, which is actually a mini-PACS, is expected to follow a clear and seamless upgrade path towards a complete PACS. This system aimed to solve the limitation of network and I/O bandwidth saturation by parallel computing method. In order to facilitate image acquisition, as well as image visualization and parallel processing, the system exploited a Parallel Databse MIMD distributed memory machine to build a fast and scalable archiving server, with multiple network connections to client workstations. When developing PACS in hospitals, it is expected that the database can be accessed by clinicians in different wards and departments. In order to provide a general user interface to integrate different types of document in a platform independent manner, one alternative way to conventional database query and retrieval software is to use tools such as Internet browsers and the World Wide Web (WWW) server. One project which makes use of this idea is the development of a hospitalwide PACS (complete PACS) at the University Hospital of Geneva, and several archive modules have become operational since 1992 [3]. This PACS is intended for the wide distribution of images to clinical wards. For security reasons, strict access control has been put into its interface, that is, before allowing a user to navigate through its patient data records, the system will first check user identifications and access rights. Researchers have long been aware that PACS systems should evolve into intelligent databases, which incorporate high-level domain knowledge, computer vision and het-

Downloaded from http://jhi.sagepub.com at PENNSYLVANIA STATE UNIV on February 6, 2008 © 1999 SAGE Publications. All rights reserved. Not for commercial use or unauthorized distribution.

42

Health Informatics Journal

erogeneous data management methods. Taira et al. [4] made much progress towards this by integrating more intelligent features into their existing PACS, such as general decision making, study routing, comparison image retrieval, and fault tolerance algorithms. They also provided a user interface that allows radiologists to operate a PACS workstation within their own knowledge domain. In summary, PACS research issues may be divided into the following aspects: ● ● ● ● ● ● ● ● ●

Image database Multimedia and Internet applications Networking DICOM Security Large-scale PACS Workstation performance Clinical applications PACS quality assurance and evaluation

Among the items listed above, medical image database system is one of the core components of any PACS systems.

Medical image database systems Medical images play a central role in patient diagnosis, therapy, surgical planning and longterm follow-up for outcome assessment as well as medical reference and education. Medical informatics systems are shifting the focus of medical imaging from the generation and acquisition of images to the post-processing, storing, analysing and management of images. This is why medical image database systems are growing so rapidly, at the same time changing the practice and research of the medical domain. Recent progress in medical image database systems such as PACS and other healthcare systems have improved the quality of patient care and healthcare, disease diagnosis and therapy, and advanced the frontiers of biomedical knowledge. Many fundamental challenges in image information processing still remain unresolved. Research is expected to proceed in multidisciplinary areas that combine the expertise of database and image processing with medical domain knowledge. Traditional medical database management systems are designed to manage and query large amounts of textual information such as patient data, diagnostic reports, pathological examinations and laboratory tests. With the rapid growth of digital imaging modalities that are producing a broad spectrum of multimedia data types, medical database management is becoming increasingly complex. In addition to textual data, radiological and pathologic images are

also important elements of a patient’s record. This raises even more difficulties in managing such data set with different data types and different medical domains since no corresponding generalized image processing tools exist that enable a user to extract key features reliably from any image. How to manage these sophisticated data in the medical database for efficient clinical applications is a challenging research issue. Arya et al. [5] described a database prototype for querying and visualizing 3D spatial medical images, QBISM (Query By Interactive Spatial Multimedia). This project focuses on the brain-mapping requirements for multimodality relationships across multiple subjects. The anatomy and physiology coordinates using rendered imagery and statistical output are represented in their data structures. Medical researchers can ask ad hoc queries over numerous 3D patient studies, and receive fast responses from the system despite the large space requirements of even a single study. One of the projects on medical information systems at the National Library of Medicine in the USA, is to develop a prototype system for archiving data and client software so as to enable a broad range of end users to access the database, retrieve, display and manipulate text and image data [6]. The system, which makes use of the Internet infrastructure, is built on the concept of using industry standard technology to provide access to metadata in an integrated design. They apply the system on medical X-rays and associated data, and a collection of digitized cryosection images, CT and MR, taken of cadavers as part of the National Library of Medicine’s Visible Human Project. Another NLM project, which is carried out in collaboration with the National Centre for Health Statistics and the National Institute for Arthritis and Musculoskeletal and Skin Diseases, is to build a system for collecting radiological interpretations for a large set of X-ray images acquired as part of the data gathered in the second National Health and Nutrition Examination Survey [7]. The system is able to deliver the images via the Internet to workstations to be interpreted for the degree of presence of particular osteoarthritic conditions in the cervical and lumbar spines. The collected interpretations can then be stored in a database. This system uses a client/server database architecture supported by the distributed server processing of client requests; a customized image transmission method for faster Internet data delivery; distributed client workstations with high-

Downloaded from http://jhi.sagepub.com at PENNSYLVANIA STATE UNIV on February 6, 2008 © 1999 SAGE Publications. All rights reserved. Not for commercial use or unauthorized distribution.

A review of intelligent content-based indexing

resolution displays; image-processing functions with an online digital atlas; and relational database management. Lowe et al. [8] described the Image Engine project which was also supported by the National Library of Medicine. As a first step towards implementing a multimedia electronic medical record system, Image Engine is a multiuser, object-oriented, client/ server database system for the storage, retrieval, integration and sharing of a wide range of biomedical images. Moreover, it uses agent technique to real-time link medical images and associated data stored in external database systems. Medical image database includes the following research components: system architecture, data acquisition and modelling, feature extraction, content-based retrieval, query languages, and user interface. Among these components, content-based retrieval attracts attention from every area of applications involving images [9]. The content of images, however, is not a very explicit concept. Different systems and applications have their own interpretation. Roughly described, it can be divided into two types, low-level (iconic) content such as texture, shape, colour, contour etc., and high-level content such as the (semantic) meaning of the objects and their spatial relationships.

Content-based retrieval in medical image databases In a conventional medical image database, most of the indexing and retrieval operations have been based on the patient identity, date, and type of examination, image number or other information contained in the image record. However, the information of higher abstraction inherent in the images is far different from the kinds of representations that are suitable for textural information. Moreover, as the use of multimedia in healthcare extends, if image databases can be organized and retrieved based on image content, more information could be utilized. It also makes possible the fusion of multimedia information extracted from different media sources. For many applications, it may not be possible to solve the complex problems of medical information database using image contents alone. Different applications make use of different content-based engines to tackle their problems. This depends on what kinds of image are to be analysed in an application. Although some research uses mathematical methods such as fractal analysis to detect and measure clinical features for medical images [10][11], most research exploits traditional

43

image content such as texture, shape, colour, contour as well as other image representation techniques (for example wavelet) to index and retrieve images. A number of such techniques have been proposed in the literature. Congiu, del Bimbo and Vicario [12] described a prototype which exploits a special indexing and querying language to support retrieval by content from a database of cineangiographic reports through visual iconic interaction. In their system schema, qualitative static and dynamic shape descriptors were introduced to provide a symbolic representation of the evolution of deformable shapes, and in order to facilitate content-based queries, a language based on interval logic was presented. Robinson, Tagare, Duncan, and Jaffe [13] proposed a similarity measure and an indexing mechanism for the non-rigid comparison of shapes. They used this nonrigid mapping between curves to compare oriented shape using KD-trees algorithm. They did experiments with the data on a database containing 86 cardiac MR images. Texture feature analysis has been among the major tools in image analysis field for decades. Petrosian and Homan [14] studied whether texture information contained in clinical EEG could be reliably used to distinguish between different interictal, preictal, ictal, and postictal stages, and predict epileptic activity in patients before an actual seizure occurs. Although the data were obtained from only one patient and the researchers restricted themselves to the direct neighbouring samples, the results showed that signal texture information can be used to distinguish different abnormal patterns. Wavelet feature vectors have also been used as a measurement for retrieving images from image databases [15] [16][17]. Image reference databases (IRDBs), as the multimedia counterpart of printed anatomy or pathology atlases, are a recurrent research topic in medical imaging. Most IRDBs require that users have prior knowledge of the field when using the system therefore such systems cannot actually meet education purpose. Bucci, Cagnoni and De Dominicis [18] proposed a content-based search engine for tomographic image databases based on the Kahrunen-Loeve transform. They aimed to integrate the content-based search engine into a radiological image database for reference and education as well as computer-assisted diagnosis purposes. While some of the research projects employ one specific image processing technique to tackle their various problems, a number of research projects are dedicated to the

Downloaded from http://jhi.sagepub.com at PENNSYLVANIA STATE UNIV on February 6, 2008 © 1999 SAGE Publications. All rights reserved. Not for commercial use or unauthorized distribution.

44

Health Informatics Journal

integration of several measurement methods into their systems. CANDID [19] is one such example. CANDID (Comparison Algorithm for Navigating Digital Image Databases) enables content-based retrieval of digital imagery from large databases using a queryby-example method. The system uses the concept of global signature that is derived from various image features such as localized texture, shape, or colour information to retrieve medical images. When manipulating image content such as shape, colour, texture or spatial relationships, conventional keyword-based indexing techniques are no longer adequate to support users’ needs. Chang [20] developed the active index concept to content-based medical image retrieval. With an active index, the retrieval system can dynamically handle ‘smart images’ that respond to accessing, probing and other actions. Khan and Yun [21] present a completely different mechanism for facilitating image retrieval by content. Instead of relying on symbolic representation of image content, they incorporated a computing paradigm called multidimensional holographic associative memory computing (MHAC), which allows post-learning dynamic specification of attention on the pixel fields in the sample. It is based on direct visual similarity and does not require intermediate modelling of meaning which might not be interpreted properly, but rather allows the user to attach their own meaning dynamically during query. Its computational mechanism is based on the principles of optical holography. Content-based retrieval is not just a database issue, it is also closely related to the understanding of a human being’s own mechanisms of perception, interpretation, and representation. Automatic interpretation of images reflects this procedure. However, due to the complex nature of the image content, when implementing the interpretation, researchers have to simplify the problem, such as only computing a symbolic description of certain aspects or content of the image by labelling a set of objects such that specific constraints are satisfied. Deruyver and Hode [22] introduced their approach to this by focusing their interest on knowledge representation based on semantic graphs for understanding images. The aim is to detect the significant anatomical cerebral regions. Since the different parts of the brain always have the same spatial relations, even if the distances may differ between different brains, with this perception the anatomic knowledge can be described by a semantic graph. The control algorithm is

based on arc consistency that Mohr and Henderson [23] have proposed. Petrakis and Faloutsos [24] proposed a query by example method to handle approximate searching by image content in medical image databases. In their method, image content is represented by attributed relational graphs holding features of objects and relationships between objects. It works with the assumption that a fixed number of ‘labeled’ or ‘expected’ objects (e.g., ‘heart’, ‘lungs’, etc.) are common in all images of a given application domain in addition to a variable number of ‘unexpected’ or ‘unlabeled’ objects (e.g., ‘tumor’, ‘hematoma’). An R-tree index structure is used in the database. Most of the early clinical PACS systems provided an infrastructure for acquiring, storing, displaying, and processing radiological image data. Recent systems pay more attention to the integration of high-level intelligence into all aspects of its operation. Furthermore, providing access to image databases through a WWW front-end has received considerable attention. There are already several projects investigating how to maintain online demonstrations of content-based retrieval through a standard WWW interface. Tang and Chiang [25] used Internet resources to provide a cost-effective, userfriendly method of accessing a medical image database. In the study, the prototype Intelligent Medical Image Retrieval (IMIR) system, as a Hypertext Transport Prototype (HTTP) server, permits a user as an Internet client to search, identify, retrieve and review medical images from the archives using general Internet browsers. The intelligent retrieval engine is expected to be able to map free text search criteria to the standard terminology used for medical image identification. Another example of content-based retrieval on the Internet is a network of severs I2Cnet [26], which provides content-based query services through a WWW server. To represent and retrieve medical images by content, I2Cnet uses description types. One of the types, AttributeMatch, supported by I2Cnet, is designed to capture the knowledge of medical experts in queries by using a similarity criterion that can be tailored to user preferences. When researchers investigate capabilities for a medical image management system, they expect that the system will not only handle clinical and administrative queries but also reduce clinicians’ repetitive work, and provide assistance to them in difficult diagnoses or with unfamiliar diseases. One interesting piece of work towards this goal by Goldbaum and Moezzi et al. [27] is the STARE

Downloaded from http://jhi.sagepub.com at PENNSYLVANIA STATE UNIV on February 6, 2008 © 1999 SAGE Publications. All rights reserved. Not for commercial use or unauthorized distribution.

A review of intelligent content-based indexing

(STructured Analysis of the REtina) system, a sophisticated image management system that will automatically diagnose images, detect key features in them, annotate image contents, and retrieve similar images. In order to generate annotation, the objects in the images are segmented and grouped into correspondent classifications. Some of the image contents are deduced by reasoning. Bayesian networks that learn from image examples of each disease are exploited for inferencing. Another study in a similar application domain can be seen in Gupta and Moezzi et al. [28] which focuses on developing primitive features for ocular fundus image content-based retrieval. Databases may contain multitype information including medical images like X-rays, MRI, tomography as well as patients’ clinical identity and examination reports. When both the alphanumeric data and information inherent in the images are required to be retrieved, content-based retrieval cannot be solved only with a matching process based on image analysis. Tchounikine [29][30] and Petrakis and Faloutsis [24] introduced the semantic aspects in the retrieval procedures in medical information systems and propose a data model for describing the semantic contents of images. The image is viewed as pairs of isosemantic regions and signals in respect of an anatomic and a pathological model. Such representation is used to index the pictorial part of the medical record. In the past, medical imaging researchers and the medical informatics community have been poorly incorporated. Today medical images are expected to merge into an integrated database which can gather, manage, and use multimodality image information for archiving, communicating, classifying, organizing, and retrieving medical images. If such integrated database systems are developed and expand, they will facilitate the correlation of massive multimedia clinical data and advance medical knowledge. Wong and Huang [31] implemented an integrated medical image database system at UCSF. In order to index images by content, the medical image database system was based on a three-tiered client server architecture: client medical workstations, database application servers, and a hospital-integrated picture

Table 1 The basic facilities in I-Browse Input

Output

Matching

image content textual query image content image+text

textual annotation images images images

content->features->text text->features->images features->invariants->images combination of above

45

archiving and communication system (HIPACS). They combined knowledge-base techniques with content-based retrieval to move the database toward a real practical system. As PACS systems and other medical information systems accumulate more and more medical data, many new research issues and applications will arise. Medical image database research has yet to mature, however. A major problem is that the data representation and organization in medical information systems lacks proper and efficient concepts and a comprehensive knowledge representation scheme to present a common basis of the heterogeneous information so that the information in the database systems may be stored, shared and accessed more easily. Particularly for the higher-level data types and the relationships between high-level and low-level data, suitable concept formalization methods to facilitate the construction of large-scale information systems have not yet been developed.

I-BROWSE: INTELLIGENT BROWSING OF MEDICAL IMAGES The I-Browse project is a collaborative project between the Department of Computer Science, City University of Hong Kong and the Clinical School of the University of Cambridge, UK. A major aim of the research is to identify and to develop algorithms and techniques that would form the basis of an intelligent medical visual information system capable of supporting medical diagnosis or training in the histological image domain. Since histological images are complicated and the differences between them quite subtle, current content-based indexing techniques which seek to identify images based solely on their specific image characteristics, e.g. texture, colours and shapes, are not sufficient to meet the users’ searching requirement. New techniques must be developed to understand and search across the database, combining the variations in several information levels, i.e., primitive image features, semantic meanings, as well as concepts. This multilevel contentbased indexing and browsing would readily help to capture a user’s conceptual request and reduce variations. The system is expected to support several modes of query and retrieval as well as the automatic annotation of images. Table 1 gives the basic facilities that will be implemented in the system. The following are some research themes and questions that not only need to be inves-

Downloaded from http://jhi.sagepub.com at PENNSYLVANIA STATE UNIV on February 6, 2008 © 1999 SAGE Publications. All rights reserved. Not for commercial use or unauthorized distribution.

46

Health Informatics Journal

tigated in I-Browse, but also need to be asked for in a pure content-based retrieval system. The research questions are: 1. What constitutes the ‘relevant information content’ of an image in the specific context of the application? 2. How much meaning can a user attach to a query to specify a search for the desired images? 3. How may an extensible system be developed? A user may eventually require retrieval based on a set of properties not initially captured by the system. 4. How to retrieve images which partially or possibly satisfy a query or an imprecise query in a ranked order? There is a difference between ‘recognition’ and ‘browsing’. Recognition has to be 100% correct (i.e. always the best match) among very similar images, but browsing only requires that the one you want is among, say, the top 10 best matches (which is slightly easier). In medical application, it would be useful to find out which ‘group’ of images a pathologist always considers together when carrying out a certain task, then find ways of describing that group of images based on their image characteristics and conceptual similarity. 5. How to recognize multiple dimensions inherent in the images? Different properties present themselves differently depending on how we view the object or how we prepare the tissue slide (in the case of histological images). How to convert these from medical terms to visual features measurement? It is necessary to define useful image features by which the supervised information and feedback visual feature h1

visual feature detector

hypothesis generator

semantic analyser

h2 hn

KB1

annotation

visual content

classifier index constructor semantic index

iconic index

Image database

Fig. 1

An architecture of I-Browse.

6.

7.

8.

9.

objects can be identified or the content of the image could be illustrated. How efficient and accurate is the retrieval and browsing process? What would be a reasonable waiting time for the users? The organization and the content of the index structure play an important role in this aspect. It is generally desirable to establish an ordering among index terms in order to be able to cut the search process. If the aim is to support real time answer to queries, the adopted indexing techniques should be scaleable with respect to the database size; the search must be efficient enough to support real time, which should be defined according to users’ time requirement. How to integrate the ranked hypotheses obtained from a retrieval engine and derive a final reasonable result. How to integrate intelligent features into the system, for example, pre-fetching and intelligent agents that can analyse a query object, determine the dominant features of that object, and then decide which among the several available retrieval techniques is most promising for handling that query. How to provide mechanisms for refining queries using context and user models?

To answer some of these questions, we propose a system architecture to support annotation of medical image and the intelligent browsing of images.

An architecture for supporting intelligent indexing and retrieval of images The I-Browse architecture is shown in Fig. 1. The I-Browse architecture combines disparate techniques that complement each other. In visual feature detector, under the supervision of the semantic analyser, a number of visual feature detectors will be used to extract the necessary primitive features such as shape, texture, layout etc. from the input or query images. The extracted image features will be passed to a hypothesis generator together with their correspondent probabilities which tell how likely it is that the measurements represent specific histological meanings. In hypothesis generator, with the help of a knowledge base, KB0, and a suitable reasoning mechanism, a set of hypotheses and their correspondent probabilities (or mass functions) will be generated and sent to the semantic analyser. In concert with another knowledge base (KB1) they will confirm or refute the hypotheses, and generate high-level descriptions for the

Downloaded from http://jhi.sagepub.com at PENNSYLVANIA STATE UNIV on February 6, 2008 © 1999 SAGE Publications. All rights reserved. Not for commercial use or unauthorized distribution.

A review of intelligent content-based indexing

images. In addition, it will produce feedback to the visual feature detector about which features need to be further confirmed by what kinds of specific feature detectors. It is here that natural language processing technique would be deployed. After several interactions between visual feature detector and semantic analyser, with the result obtained from the semantic analyser, we can further classify the images according to certain criteria such as the function organs or disease patterns etc. via a pattern classifier. Since we will make use of several levels of features to do the indexing and retrieval work, we expect there are at least two kinds of index, one for low-level (pure content-based or iconic) retrieval and browsing, and another one for high-level (semantic or conceptual) retrieval and browsing.

(a)

(b)

Medical domain: GI tract To give the research its practical significance, the above architecture is being applied to the intelligent retrieval and browsing of histological images obtained from the whole gastrointestinal tract (GI tract). The gastrointestinal system is primarily involved in reducing food for absorption into the body. This process occurs in five main phases within defined regions of the gastrointestinal system: ingestion, fragmentation, digestion, absorption and elimination of waste products. The gastrointestinal system is essentially a muscular tract lined by a mucous membrane that exhibits regional variations in structure reflecting the changing functions of the system from the mouth to the anus [32]. Although each region of the GI tract has specific functions, there is a general structure common to all segments. Understanding of the structure will greatly aid in an appreciation of the variations of this central theme that reveal the distinctive features of each organ of the GI tract. We focus on five areas along the tract, i.e., oesophagus, stomach, small intestine (small bowel), large intestine (large bowel) and anus. The GI canal may be identified by its tubular nature and the division of its wall into four distinct layers: 1. 2. 3. 4.

mucosa; submucosa; muscularis externa; serosa.

(c)

(d)

(e)

Fig. 2 shows the basic structure of mucosa (M), submucosa (SubM), lumen and muscularis mucosae (MM), which belongs to mucosa, of the above five areas. Fig. 2 Basic mucosa and submucosa structure in GI tract (a) oesophagus; (b) stomach; (c) small intestine; (d) large intestine; and (e) anus. Downloaded from http://jhi.sagepub.com at PENNSYLVANIA STATE UNIV on February 6, 2008 © 1999 SAGE Publications. All rights reserved. Not for commercial use or unauthorized distribution.

47

48

Health Informatics Journal

Histological images have their own image characteristics which are quite different from other types of medical images. They are 2-D images produced by cutting across a 3-D object, and the interpretation for the images depends on many factors, such as magnification, cutting angle, staining and slide preparation etc.. The reason for selecting the GI tract is that it has a fairly limited range of appearances that are histologically distinct, and there are conditions that set admixtures of these appearances that provide interesting challenges for our system. Further detail of this work can be found in [33].

CONCLUSIONS We have reviewed the recent advances in medical image database and retrieval research. In particular, we have highlighted the recent exploitation of the communication infrastructure provided by Internet and the World Wide Web for medical image retrieval and medical information gathering and distribution. The convergence of medical database, image analysis and content-based indexing and retrieval has opened up new research avenues for PACS as well as medical visual information systems. Medical images, unlike consumer-oriented images, pose additional challenges to content-based image retrieval in that visual features of normal and pathological images are typically separated by only very subtle differences in visual appearance. Intelligent image retrieval and browsing requires a combination of prior medical domain knowledge, image content and image annotation analysis. This in turn calls for the development of computational mechanisms and architecture that support hypothesis generation and reasoning, iconic feature extraction from images, and semantic contents generation and analysis. The I-Browse project [33] is one project that works towards this aim.

Acknowledgments The I-Browse project is funded by the Hong Kong Jockey Club Charities Trust. Ms Lilian Tang is part-supported by an Overseas Research Studentship.

References [1] Shiffman S, Shortliffe E H. Biomedical imaging and the evolution of medical informatics. Computerized Medical Imaging and Graphics 1996; 20: 189-92. [2] Friscione M, Marenzi L, Punzi S, Stofella P. PARCS: Parallel Picture Archiving and Communication System. High-

Performance Computing and Networking. April 1994: 441-45. [3] Mascarini C, Ratib O, Trayser G, Ligier Y, Appel R D. Inhouse access to PACS images and related data through World Wide Web. SPIE Proceedings: Medical imaging 1996. PACS design and evaluation: engineering and clinical issues. 1996; 2711: 531-37. [4] Taira R K, Breant C M, McNitt-Gray M, Sinha S, Huang HK. Add intelligence to PACS. SPIE Proceedings: PACS design and evaluation. 1992; 1654: 477-84. [5] Arya M, Cody W, Faloutsos C, Richardson J, Toga A. A 3D medical image database management system. Computerized Medical Imaging and Graphics 1996; 20: 269-84. [6] Thoma G R, Long L R, Berman L E. A client/server system for Internet access to biomedical text/image databanks. Computerized Medical Imaging and Graphics 1996; 20: 259-68. [7] Long L R, Ostchega Y, Goh G H, Thoma G R. Distributed data collection for a database of radiological image interpretations. SPIE Proceedings: Storage and retrieval for image and video databases V. 1997; 3022: 228-37. [8] Lowe H J, Walker W K, Polonkey S E, Jiang F, Vries J K, McCray A. The image engine HPCC project. A medical digital library system using agent-based technology to create an integrated view of the electronic medical record. Proceedings of the Third Forum on Research and Technology Advances in Digital Libraries – ADL ’96. 1996: 45-56. [9] Wong S T C, Huang H K. Editorial. Computerized Medical Imaging and Graphics 1996; 20: 187-88. [10] Cross S S. Fractal in pathology. Journal of Pathology 1997; 182: 1-8. [11] Veenland J F, Grashuis J L, van der Meer F, Beckers A L D, Gelsema E S. Estimation of fractal dimension in radiographs. Medical Physics 23 (4): 585-94. [12] Congiu G, Del Bimbo A, Vicario E. Iconic retrieval by contents from databases of cardiological sequences. Proceedings of the Third IFIP 2.6 Working Conference on Visual Database Systems. 1995: 158-74. [13] Robinson G P, Tagare H D, Duncan J S, Jaffe C C. Medical image collection indexing: shape-based retrieval using KD-trees. Computerized Medical Imaging and Graphics 1996; 20: 209-217. [14] Petrosian A, Homan R W. The analysis of EEG texture content for seizure prediction. IEEE Engineering in Medicine and Biology 16th Annual Conference, 1994: 231-22. [15] Stark H G, Laufkotter G P. Image indexing and contentbased access to databases of medical images with wavelets. SPIE Proceedings: Wavelet applications in signal and image processing III. 1995; 2569: 790-800. [16] Murthy R, Kaya K, Bilgutay N M. Multiresolution techniques for ultrasonic imaging. IEEE Ultrasonics Symposium 1994; 3: 1793-96. [17] Impagliazzo J, Greene W, Huynh Q. Wavelet image compression algorithm for side scan sonar and teleradiology. SPIE Proceedings. 2491: 162-72. [18] Bucci G, Cagnoni S, De Dominicis R. Integrating content-based retrieval in a medical image reference database. Computerized Medical Imaging and Graphics 1996; 20: 231-41. [19] Kelly P M, Cannon M. Query by image example: the CANDID approach. SPIE Proceedings: Storage and retrieval for image and video databases III. 1995; 2420: 238-48. [20] Chang S-K. Active index for content-based medical image retrieval. Computerized Medical Imaging and Graphics 1996; 20: 21929. [21] Khan J I, Yun D Y Y. Holographic image archive. Computerized Medical Imaging and Graphics 1996; 20: 243-57. [22] Deruyver A, Hode Y. ‘True’ three dimensional image labeling: semantic graph and arc consistency. Fifth International Conference on Image Processing and its Applications. July 1995: 105-108. [23] Mohr R, Henderson T C. Arc and path consistency revisited. Artificial Intelligence, 1986; 28: 225-33. [24] Petrakis E G M, Faloutsos A. Similarity searching in medical image databases. IEEE Transactions on Knowledge and Data Engineering. 1997; 9: 435-47.

Downloaded from http://jhi.sagepub.com at PENNSYLVANIA STATE UNIV on February 6, 2008 © 1999 SAGE Publications. All rights reserved. Not for commercial use or unauthorized distribution.

A review of intelligent content-based indexing [25] Tang Y-K, Chiang T T. Intelligent retrieval of medical images from the internet. SPIE Proceedings: Medical imaging 1996. PACS design and evaluation: engineering and clinical issues. 1996; 2711: 440-48. [26] Orphanoudakis SC, Chronaki C E, Vamvaka D. I2Cnet: Content-based similarity search in geographically distributed repositories of medical images. Computerized Medical Imaging and Graphics 1996; 20: 193-207. [27] Goldbaum M, Moezzi S, Taylor A, Chatterjee S, Boyd J, Hunter E, Jain R. Automated diagnosis and image understanding with object extraction, object classification, and inferencing in retinal images. Proceedings. International Conference on Image Processing. 1996; 3: 695-98. [28] Gupta A, Moezzi S, Taylor A, Chatterjee S, Jain R, Goldbaum I, Burgess S. Content-based retrieval of ophthalmological images, Proceedings. International Conference on Image Processing. 1996; 3: 703-706.

49

[29] Tchounikine A. Creation and content-based retrieval in a radiological documentary record. Proceedings of the Third Basque International Workshop on Information Technology (BIWIT ’97) Data management systems. 1997: 111-17. [30] Tchounikine A, Amghar Y, Flory A. Semantic interrogation for a radiological documentary record. Proceedings. Tenth IEEE Symposium on Computer-Based Medical Systems. 1997: 98-102. [31] Wong S T C, Huang H K. Design methods and architectural issues of integrated medical image data base systems. Computerized Medical Imaging and Graphics 1996; 20: 285-99. [32] Wheater, P R, Burkitt, H G, Daniels V G. Functional Histology. Edinburgh: Churchill Livingstone, 1987: 203. [33] Tang L H Y, Hanka R, Ip H S H. A system architecture for integrating semantic and iconic content for intelligent browsing of medical images. SPIE Proceedings: Medical Imaging 98, San Diego, CA; 1998.

Downloaded from http://jhi.sagepub.com at PENNSYLVANIA STATE UNIV on February 6, 2008 © 1999 SAGE Publications. All rights reserved. Not for commercial use or unauthorized distribution.