May 25, 2000 - experience has suggested that simple techniques fail to provide accurate results .... Differential Pulse Code Modulation (DPCM);. ⢠Hierarchical ...
25-05-00
Diagnostic Classification of Leg Radiographs D.N.Davis and B. Sharp Artificial Intelligence Group, School of Computing, Staffordshire University. January 1997 Abstract Medical imaging is a powerful and increasingly popular tool. This project addresses two major problems: data storage and transmission; and automated diagnosis and classification. We suggest that by addressing the latter issuue using advanced reasoning tools and clinical models (necessary in even the most deceptively obvious modailities), medical images (and subregions within them) can be categorised according to medical standards. Past experience has suggested that simple techniques fail to provide accurate results; and that to cope with the wide diversity of image information within any one medical modaility and subdomain (e.g. leg radiograophs) an adaptable system that combines evidence from multiple sources is required. The outcome from using such a system will provide a means for indexing large numbers of images. Furthermore, we suggest that the clinical classification of an image is of importance in categorisng image sub-regions and so providing a means by which differential information retaining compression techniques can be used, alleviating many problesm related to dtaa storage and image transmission. These problems need to be addressed as we move towards a technology dependent medical domain using Picture Archiving and Communication systems and Telereadiology techniques for diagnosis, conference and teaching purposes.
Background Medical imaging is a powerful and useful tool of use to radiologists and consultants, allowing them to improve and facilitate their diagnoses. In the USA, X-ray images represent 70% of the total amount of radiological images; the remaining 30% consisting of more newly developed image modalities, such as Computed Tomography (CT), Magnetic Resonance Imaging (MRI), Ultrasonography (US), Positron Emission Tomography (PET), Single Photon Emission Computerised Tomography (SPECT), Nuclear Medicine (NM), Digital Subtraction Angiography (DSA), and Digital Fluorography (DR). These latter technologies directly provide digital images, while Xray films have to be digitised, using for example, high definition laser scanners. A similar proportion of analogue to direct digital medical images is found in the UK (and
1
25-05-00
Europe). Although the proportion will decease with time, as the more recent technologies are used, there still exist a considerable number of X-ray films (and corresponding digitised images) and an increasing number of medical images overall. There are a number of reasons why digital images are preferred in the medical domain. Firstly a digital image is readily accessible for processing (enhancement, volume rendering, etc.) which facilitates their analysis and can improve the clinical diagnosis. Secondly, there is the ease of storage and transmission. This is particularly apt, given the increasing awareness of Picture Archiving and Communication Systems (PACS), which provide a full-digital environment for image manipulation, specifically acquisition, processing, storage, communication and display. PACS provides the means to improve the extraction of medically pertinent information across databases of images and allows telediagnosis or remote teaching. Extending PACS to include multi-media facilities and diagnostic or teaching aids is also a growing possibility [Oakley et al, 1994; Natarajan et al, 1996]. However, there are a number of problems associated with this increasing volume of data. Here we highlight just two. As the number of digitised images grows, it will become increasingly difficult to manage them (particularly in terms of meaningful storage and retrieval terms) unless clinically useful means of categorising the image subtypes are found; we return to this theme latter. Furthermore the sheer volume of data has already exceeded the capabilities of the most recent and efficient technologies, such as optical disks for storage or Fibre Distributed Data Interchange networks for communications. Table 1 highlights the storage and transmission bandwidth requirements for the frequently used radiological images. For instance, an optical jukebox can contain about 1 to 2 Tetrabits, while a medium-sized hospital will produce around 20 Tetrabits of image data per year. The transmission of a full X-ray, via a 100Mbps network with an effective transfer rate of 20Mbps, will take 10 seconds, when medical applications will require a time shorter than 2 seconds. For this reason, some means of compressing digital images, without loss of medical detail, is another area of interest.
Modality
Image size (pixels)
Grey level (bits)
Image size (bytes)
Images per Examination
CT MRI DSA DF US SPECT PET CR Digitised Film
512*512 256*256 1024*1024 1024*1024 512*512 128*128 128*128 2048*2048 2048*2048
12 12 8 8 6 8 or 16 16 12 12
512 Kb. 128 Kb. 1 Mb 1 Mb. 256 Kb 16. or 32 Kb 32 Kb. 8 Mb 8 Mb
30 50 20 15 36 50 62 4 4
Examination size (bytes) 15 Mb 6.3 Mb 20 Mb. 15 Mb. 9 Mb 0.8 or 1.6 Mb 2 Mb 32 Mb 32 Mb
Table 1: Sizes of radiological images In summary, we contest that for future improvements in the use of PACS (and similar medical image systems), the suitable medical categorisation and compression of digitised images will be of great importance. We believe that the proposed work
2
25-05-00
addresses both these issues. We hope to show that techniques applicable to one particular subtype of digitise X-ray film may generalise across other subtypes of similar diagnostic intention with similar characteristics.
Image Compression Image compression provides a means of reducing storage load and transmission bandwidth, and is therefore of great interest [Wong et al, 1995]. There are two broad types of compression algorithm of use for (medical) images: reversible (or lossless) compression; and irreversible (or lossy) compression. Each of the two methods has its advantages and drawbacks, and the choice between them raises important legal and regulatory questions. Another technique, clinical compression, reduces the number of images in examinations which require usually large number of images (for example MRI or DSA); there are no algorithms to perform this task, and requires manual control of the examination by clinically adept personnel. Lossless compression introduces no artefact in the images, nor is there any loss of detail or grey level changes; the decompressed image is exactly the same as the original. However because of its main drawback, a low compression rate (between 2 to 10), the compressed image is still large. Current and advanced lossless techniques include: • Differential Pulse Code Modulation (DPCM); • Hierarchical Interpolation (HINT); • Difference Pyramid (DP); • Bit-Plane Encoding (BPE); • Multiplicative Autoregression. The choice between these methods has to be made with regard to compression rate, algorithm complexity and ease of implementation. Some algorithms are also particularly suited to certain types of image (CT, MRI etc.). All these techniques can be made lossy through the introduction of a quantisation stage (see figure 1), but this gives little improvement in the compression rate.
Source Image
Image Transformation (decorrelation)
lossless Entropy Coding lossy
Compressed Image
Quantisation
Table Specification
Table Specification
Figure 1: General image compression framework Irreversible (or lossy) compression techniques provide a much greater compression rate factor (between 10 to 100), depending on the algorithm and the desired quality of the
3
25-05-00
decompressed image. Their major drawback is their irreversibility with the compression causing a loss of detail and sometimes the alteration of grey level distribution. The most promising (lossy) algorithms (and their compression rates) include: • Discrete Cosine Transform (DCT) [20 to 50]; • Full-Frame DCT (FFDCT) [20 to 35]; • Lapped Orthogonal Transform (LOT); • Subband Coding [10 to 40]; • Vector Quantisation (VQ) [up to 10]; • Quadtrees [up to 10]; • Fractal Compression. The indicative compression rates given are as measured for certain classes of image without loss of “clinical sensitivity” (according to Receiver Operating Characteristics), but are dependent on the quality required for the compressed image. The introduction of irreversible compression techniques in medical imaging raises the problem of defining a legal standard to assess the quality of compressed images which allows clinically correct diagnoses. At the moment, such standards do not exist because of the lack of objective clinical references and validation tests. For this reason, lossless compression is preferred for primary diagnosis images, while irreversible compression is used for teaching and reviewing reports, and for archiving images with a known diagnosis.
Aim of the project Many modalities (of medical image) carry a lot of redundant information, for example ground or medically unimportant areas, especially digitised X-ray film. Irreversible compression on these areas, with lossless compression on medically relevant areas, would provide significant storage and transmission gains (see figure 2). This requires finding a means to categorise image areas into, at least, three regions: ground, medically irrelevant and medically relevant, with the latter being labelled with medically important areas. To do this accurately, we also need to find a way of categorising the medical nature of images, as depending upon the nature of the image some areas that may normally be described as medically irrelevant can have a bearing on the medical diagnosis (and prognosis).
Background Source Image
Developed Image Analysis System
High ratio/loss
Medically Irrelevant
Low ratio/loss
Medically Relevant
Lossless Compression Stage
Figure 2: Selective Image Compression Process
4
Classified and Compressed Image
25-05-00
We aim to identify means by which this can be performed using one category of X-ray film (of the leg). The techniques and knowledge used in analysing this class of image may be sufficiently generic for them to be used on the wider class of X-ray film images contained background, soft tissue (muscle, cartilage, etc.) and hard tissue (or bone).
Image Categorisation Given that we can develop segmentation algorithms and a system capable of providing initial hypotheses about the image areas, we can then use more advanced techniques to classify the medically important areas of the image, and so provide an overall classification for the image. We would expect the system to initially provide three classifications: healthy; fractured or diseased; and unclear. The fractured or diseased case can be further classified, for example into: hairline; splintered; clean; and severe. Problematic images (i.e. the unclear categorisation) can be passed onto either a clinician or a more advanced aspect of the system capable of reasoning using past cases and medical models to provide further clarification and minimise the number of unclassified images. The proposed system should then make use of the image classification to refine its categorisation of the image areas in all but the case of a healthy diagnosis; this proposed use of feedback, to refine the image segmentation, is further described below.
Proposed Approach To solve the two highlighted problems for this application, we propose to develop a computer system that combines image models, clinical knowledge and reasoning capabilities. In doing so, we will be building on our experience in building an earlier intelligent medical image interpretation system [Davis and Taylor, 1991; Davis and Forsyth, 1994]. This system was capable of re-evaluating earlier hypotheses about the location of objects within an image on the basis of subsequent results, and so modifying its proposed solutions as they developed. Before proposing how such a system may be applied to this domain, we need to consider the various stages in categorising images and image areas. To provide accurate image interpretation, we need first to produce high resolution digitised images (2048 by 2048 pixels by 12 bits) of X-ray films (to be provided by Stafford District General Hospital); a high resolution laser scanner is requested for this purpose. An extended storage system will be required to store the digitised radiographs in sufficient number to facilitate some of the proposed statistical based techniques.
5
25-05-00
Figure 3: Example digitised X-ray film of the chosen image domain showing multiple fractures. The first step in classifying the images is to perform a ground-figure separation (primary segmentation). Initial experiments suggest that segmentation based on thresholding may be sufficient for some images. However, as X-ray film quality is affected by operator error and machine variation, with fluctuations in image capture factors such as exposure time and X-ray beam energy fluctuations, we can expect some inconsistency in the nature of the background. This suggests that a more adaptable approach that includes thresholding and model-based techniques, will be required. Secondary segmentation involves the differentiation of image areas within the figure aspect of the digitised X-ray film, primarily the separation of soft and hard tissue. In some images, showing just the upper or lower leg, this may be straightforward. However as many X-ray films of (possibly broken) legs incorporate knee joints plus occasionally leg extremities (for example ankle, foot and hip), we need to develop an adaptable method capable of operating within the full range of expected imaged body parts. This process may be further complicated in images showing extreme fractures, by the dispersion of hard tissue through splintering. One promising approach is to develop model-based vision techniques [Cootes et al., 1991] allied to a adaptable means of using them, whether a rule-based segmentation system [Davis and Taylor, 1991] or active contour models (or snakes) [Davis et al., 1995]. We can develop geometric and appearance models for the expected major structures to be found in images of the leg. However as the major structures will vary across patients and with age, we may need a number of possible models, and adaptable ways of using them. The outcome of the secondary segmentation process will be image areas labelled as medically irrelevant (for example soft tissue) and clinically relevant (typically hard tissue or bone).
6
25-05-00
Figure 4: Radiograph showing multiple fractures to fibula and less damage to the tibia. Having identified the clinically relevant areas, the next step will be to identify the critical areas, i.e. areas depicting fractures (if any). Depending on the nature of a fracture this may be relatively straightforward (for example a single severe or clean fracture causing a misalignment of the bone), or more complex (for example splinter fractures) or indeed very hard to spot (hairline fractures). Beyond these three cases, exist many more (for example multiple fractures as shown in figure 3, crushed bones etc.). We will need to investigate what type of image cues (for example, whether related to appearance or geometric structure or both) are of use in identifying these regions. There are a wide variety of possible approaches and image cues available, including geometric, texture and intensity distribution based information [Sonka et al., 1993]. We hope that much of the groundwork, for this particular part of the project, can be achieved using postgraduate and exchange students. One possible way of using these cues, to correctly label the image subregions, is through the application of an appropriate classifier system (for example, neural nets). Artificial neural networks are now a well known technology capable of producing accurate classification results in many domains. More recently they have been applied to image domains with some success [Tsao et al, 1993; Greaves et al., 1995]. We would not expect this module to be totally reliable and therefore expect to develop a (clinical) model reasoning module to clarify the situation for unclear situations. Such a classifier system may only be appropriate as a technique for determining the presence of crushed or other less obvious fractures on the basis of hypotheses generated by other parts of the system. For example, fractures to the fibula are usually accompanied by damage to the neighbouring parts of the tibia; though an obvious fracture to the tibia is not always present. Figure 4 shows just such a situation. The fibula is obviously fractured in two places, with displacement of the now separate parts of the hard tissue causing a geometric misalignment; there seems to be now obvious displacement of hard tissue to the tibia with the lower of the two fractures. However, the tibia local to the upper fracture to the fibula appears to have a hairline fracture, with a morphological abnormality to the upper (image) surface, but no geometric displacement. The developed system needs to be able to generate hypotheses about such areas (as likely sites of damage); the classifier can then be used to confirm or negate the hypothesis.
7
25-05-00
Step 1
Background
Source Image
Primary Segmentation Figure
Step 2
Medically Irrelevant
Secondary Segmentation
Figure
Medically Relevant
Step 3
Feature Extraction
Medically Relevant
Categorise
Step 4
Unclear
Diseased
Classifier
Unclear
Healthy
Reasoning Module
Figure 5: Possible Image Classification Process Figure 5 shows how these four processes may work serially. However, it is suggested that the stepwise combination of these four processes will prove to be a too rigid approach to work in all cases, as other work on the segmentation of X-ray images has shown [Manos et al., 1993; Efford, 1993]. Furthermore, it lacks the means by which the categorisation of an image can affect the segmentation within other parts of that image. For example, fractures to the tibia are usually accompanied by damage to the neighbouring parts of the fibula, typically a fracture; hence, finding a fracture the main bone (the tibia) should cause the system to reappraise its segmentation of the fibula. So while the stepwise combination of the four processes may provide a routine method for image classification and diagnosis in the more straightforward cases, such a strategy may fail where a more adaptable (blackboard system) architecture (such as that developed in [Davis and Taylor, 1991; Davis and Forsyth, 1994]) would succeed. This type of artificial intelligence system can combine all the capabilities listed above in such a manner that progressive (and opportunistic) steps towards a final image (and image region) classification can modify earlier results (hypothesis refinement). As the system will require the integration of a number of well defined (almost independent) modules, we can develop the blackboard architecture used in the earlier work to make use of more recent advances in artificial intelligence architectures. One promising avenue is the use of agents within a tightly coupled multi-agent system (for example, [Hayes-Roth, 1990]).
8
25-05-00
Accesses only source image and hypotheses about figure and ground
Accesses and refines hypotheses about all figure subregions and provides final image classification
Primary Segmentation
Accesses hypotheses about figure and medically relevant and irrelevant
Source Image Ground Accesses and refines hypotheses about medically critical (and relevant) regions
Reasoning Modules
Figure
Secondary Segmentation
Medically Irrelevant Medically Relevant Healthy
Feature Extraction
Accesses hypotheses about medically relevant to generate further information
Medically Critical Set of hypotheseses (initially empty) to be refined by the system and its modules
Diseased Categorise
Unclear
Classifier
Accesses information about medically relevant to initialise hypotheses about medically critical areas
Figure 6 : A Tightly Coupled Multi-Agent Image Analysis System Figure 6 shows one possible architecture that combines this approach with the processes shown in figure 5. Each process has limited accessibility to a central set of hypotheses about the image (for example its classification) and its subregions (ground, figure, medically irrelevant, relevant and critical). The hypotheses, their changes and the contributing evidence, can be centrally stored (on a global data object or blackboard). As these hypotheses are seeded (or modified), those modules which are allowed access to them can opportunistically access, seed and refine hypotheses they contribute towards. Each time a hypothesis is modified, a global updating is allowed, allowing the different semiautonomous modules (or agents) to collaboratively refine the set of hypotheses about the source image. As we found in the earlier work [Davis and Taylor, 1991], such collaborative reasoning processes require a means of stopping; possible alternatives include stopping when no hypotheses can be further modified or when a specific module (for example one of the reasoning modules) is satisfied that a consistent and full set of hypotheses about the image have been generated. It is at this stage, that the system can then categorise the image (according to some useful clinical metric) and pass the labelled subregions to the appropriate compression techniques.
Project Objectives 1. Gathering of suitable (leg) X-ray films to form a database of images. 2. Development of image segmentation techniques to provide initial image segmentation of image areas (Deliverable D1) 3. Investigation of suitable visual (and spatial) cues to provide reliable classification of medically relevant areas (Deliverables D1 and D2).
9
25-05-00
4. Development of classifier system (for example artificial neural net) to make use of cues found useful in objective 3 (Deliverable D2). 5. Development of clinical models of use in categorising images, including modules for reasoning about unclear cases (Deliverable D3). 6. Development of existing image interpretation architecture [Davis and Taylor, 1991] to provide framework for integrating above objectives in a cohesive system (Deliverable D4). 7. Integration of compression techniques for segmented and categorised radiographic images (Deliverable D5). 8. Comparison of classification results for developed system with experienced radiologist (Deliverable D5).
References [Cootes et al., 1991] T.F. Cootes, D.H. Cooper, C.J. Taylor and J. Graham, A trainable method of parametric shape description. Proceedings of the British Machine Vision Conference,BMVC91, Glasgow, pp 54-61, 1991. [Davis and Taylor, 1991] D.N. Davis and C.J. Taylor, A blackboard architecture for medical image interpretation. Proceedings Of SPIE90:Medical Imaging V, pp. 421432, San Jose, California, February 1991. [Davis and Forsyth, 1994] D.N. Davis and D. Forsyth, Knowledge-based cephalometric analysis: A comparison with clinicians using interactive methods. Computers and Biomedical Research, vol 27, 1994. [Davis et al., 1995] D.N. Davis, K. Natarajan and E. Claridge, Multiple Energy Function Active Contours Applied to CT and MR Images of The Brain. IEE IPA95, Heriot-Watt University, July 4-8 1995. [Greaves et al., 1995] I.D. Greaves, J. Davies and P.B. Musgrove, Area identification of bone marrow smears using radial basis function networks and the HSI colour model., IEE IPA95, Heriot-Watt University, July 4-8 1995. [Hayes-Roth, 1990] B. Hayes-Roth, Architectural foundations for real-time performance in intelligent agents. Real Time Systems, May, 1990. [Efford, 1993] N. Efford, Knowledge generation techniques for the model-driven segmentation of hand and wrist radiographs. SCIA93: Proceedings of the Eighth Scandinavian Conference on Image Analysis, Norway, pp. 251-256, 1993. [Manos et al., 1993] G. Manos, A.Y. Cairns, I.W. Ricketts and D. Sinclair, Automatic segmentation of hand-wrist radiographs, Image and Vision Computing, vol. 11, no. 2, 100-111, March 1993. [Natarajan et al., 1996] K. Natarajan, D.N. Davis and E. Claridge, The symbolic atlas of the brain: handling non-visual information associated with neuroanatomy. Medical Informatics, vol. 21, no. 1, 1-22, 1996. [Oakley et al., 1994] J.P. Oakley, D.N. Davis, R.T. Shann and L. Hugeville, A Database Management System For Vision Applications. British Machine Vision Conference, BMVC94, September 1994.
10
25-05-00
[Sonka et al., 1993] M. Sonka, V. Hlavac and R. Boyle. Image Processing, Analysis and machine Vision. Chapman and Hall, 1993. [Tsao et al., 1993] E.C.-K. Tsao, L. Wei-Chng and C. Chin-Tu, Constraint satisfaction neural networks for image recognition, Pattern Recognition, vol. 26, 553-567, 1993. [Wong et al, 1995] S. Wong, L. Zaremba, D. Gooden and H.K. Huang, Radiological image compression - A review, Proceedings of the IEEE, vol. 83, no. 2, 1995.
11
