Automated extraction of image segments from ... - Semantic Scholar

Automated extraction of image segments from clinically diagnostic hand-drawn geometric shapes R.M. Guest, M.C. Fairhurst, J.M. Potter* Electronic Engineering Laboratory, University of Kent, Canterbury, Kent, UK, CT2 7NT. *Nunnery Fields Hospital, Canterbury, Kent, UK. E-mail:{rmg,mcf}@ukc.ac.uk Abstract Simple geometric shape drawing tasks are commonly used to diagnose and monitor patient performance for a range of clinical and neuropsychological conditions. Assessment relies upon observing the presence of components within a drawn image. Application of assessment criteria has been shown to vary amongst trained raters. An algorithm is presented to automatically extract the components from the static image of shape drawing responses. Specifically, images taken from a group of patients with visuo-spatial neglect and control subjects show the accurate identification of horizontal, vertical and diagonal components. Examples of performance metrics based on the features extracted from the component analysis show clear differences between neglect and control responses which are able to detect differences in performance more sensitive to the standard number of component assessment.

diagnosis and recovery monitoring throughout their stay in hospital, accuracy, consistency and repeatability in assessment are of importance. The automated extraction of line information from images is widely documented [7,8] and some attempt has been made to apply this to other Neuropsychological tests such as the Rey-Osterrieth Complex Figure [9]. At present, these techniques have not been applied to the standardised assessments of visuo-spatial neglect. This paper presents a method of automatically segmenting a hand-drawn image into horizontal, vertical and diagonal components and reporting their location within the drawing. More importantly, it will be shown that this information can be used algorithmically to establish the presence of constituent shape components and their spatial relationship and hence produce a series of standardised performance scores based on the outcome of the drawing.

1. Introduction

2. Diagnostic Task and Data Capture

Hand-drawn images are used widely as clinical and neuropsychological diagnostic tasks for a variety of conditions including Parkinson's disease [1], dyspraxia [2] and visuo-spatial neglect [3]. Typical task configurations include the copying of simple geometric shapes such as squares, crosses and cubes and the completion of images, for example forming a mirror image of a given shape [4]. Assessing performance on each individual side of the visual field is particularly important for the diagnosis of visuo-spatial neglect - a condition which can occur following a stroke whereby subjects fail to respond to stimuli to one side of their visual field. Assessment of test responses relies on the application of a set of marking criteria based on the presence (or absence) of drawing components [5]. Performance assessment has been found to be subjective, with disagreement occurring between trained assessors each applying their own interpretation to marking criteria [6]. As these tests contribute to a patient's

Two figure copying and one figure completion test responses are used to demonstrate the extraction algorithm adopted. For the figure copying task, model shapes (a cube and a constructional cross) were printed individually in the top horizontal centre of two separate A4 sized (267 x 210 mm) paper overlays. The test subject was required to copy the shape directly below the printed image. For the figure completion task, the right hand side components of a representational image of a house, split vertically, were positioned at the horizontal centre of an overlay. The test subject was required to draw the mirror image of the provided model. Position data was captured in real time using a Wacom WD1212 graphics digitisation tablet (spatial accuracy of 6.25 lines/mm) sampled at a rate of 100Hz. The test overlays were placed individually on the surface of the tablet and the test subject asked to copy the image using a cordless biro ink pen. Features were extracted by analysing the stored response files.

Figure 1 shows the static drawing responses from an age matched control subject (a-c) who was physically fit and with no history of cerebral or cardiovascular disease, and a neglect subject (d-f). Figures a, b, d, e show responses from the figure copying task while figures c and f show the drawn left hand components of the house completion task.

many horizontal as vertical neighbours), then the neighbourhood is extended to 5 x 5 and the process repeated, zeroing the directional accumulators and inspecting only pixels round the edge of the extended neighbourhood. Figure 4 shows a 3 x 3 and 5 x 5 neighbourhood around an active pixel and the assignment of neighbourhood directions.

Obtain Binary Drawing Response

a)

Skeletonise Image

b)

Directional identification based on neighbourhood

c)

e)

d) Horizontal Elements

Vertical Elements

Diagonal Elements

Connected Path Joining

Connected Path Joining

Connected Path Joining

Connected Component Labelling

Connected Component Labelling

Connected Component Labelling

f)

Figure 1. Drawing responses made by age matched and neglect subjects

3. Algorithmic Component Extraction Component Position Information

Figure 2 shows an overview of the component extraction process. The first stage involves assessing pixels within the drawn image and establishing, on an individual basis, whether they form horizontal, vertical or diagonal components. Figure 3 shows this procedure. Static binary images of test responses, X={y(s)}, size Q x R, are skeletonised to attain the simplest structural representation of the image. Active pixels, defined as y(s)=1, where s is the location within the image, signify a drawn section. A 3 x 3 neighbourhood for each active pixel within the image is assessed and the number of horizontal, vertical and diagonal neighbour pixels around the edge of the neighbourhood are accumulated. The 'winning' direction of neighbourhood pixels of an active pixel at position (i,j) within X is assigned to a second matrix Z, size Q x R, at position (i,j). If no winning direction is found (for example if the active pixel has as

Figure 2. Component extraction algorithm overview

From the image Z, three separate images, H, V and D, can be formed containing horizontally, vertically and diagonally assigned pixels. Pixels are assigned to H, V and D such that a single pixel can not be a member of more than one directional bitmap:

H ∩V = ∅ , H ∩ D = ∅ , D ∩V = ∅ and

H ∩V ∩ D = ∅

These images can then be assessed independently to heuristically find joined pixels which are overlooked by the initial directional assessment. End points, defined as pixels with only one neighbour within images H, V and D, are located. Figure 5a shows a section of a vertical directional image, V, with two end points located in black (the line continuation pixels are shown in grey). An attempt is made to establish a connected path between pairs of end points within a specified Euclidean distance and, in the case of the horizontal and vertical images, within an appropriate directional channel. The original image X is used to assess the connectivity between pairs of end pixels. The corresponding section from X to that shown in Figure 5a is identified in 5b. This shows that the two end points are clearly connected via an 8 neighbourhood connectivity. A path-following algorithm [10] is used to establish the connectivity between pairs of end points. If connectivity is established then the shortest connected path is copied from X to the directional image under investigation. In this way the original image representation is maintained rather than a synthesised (straight line) connection between identified end points.

Having identified connected components with images H, V and D, positional information can be extracted which is necessary for an analysis of figure drawing composition. Following an 8-neighbourhood connected component labelling [11], a list is formed of components detailing the top left and bottom right corners of the area occupied by the component, the centroid location and the component length.

D D H D D

D D H D D

V D D V D D H H V D D V D D

Figure 4. 3 x 3 (and 5 x 5) neighbourhood directional assignments

Skeletonised image

Locate drawn pixel (i,j) (Neighbourhood = 3)

Find winning pixel direction

a) Majority direction found ?

No

Increase neighbourhood size by 2

Yes

Separate horizontal, vertical and diagonal bitmaps

Yes

Assign to directional bitmap (i,j)

No

Last pixel ?

Figure 3. Neighbourhood directional bitmap creation

b) Figure 5. a) End point identification showing active area, b) original image of section showing connected path

4. Examples To demonstrate the effectiveness of the drawing component segmentation algorithm, Figure 6 shows the separate horizontal, vertical and diagonal images from the original drawings shown in Figure 1. For the cross image and, in the case of the neglect subject’s house and cube drawing, no diagonal components were detected. Table 1 shows an example of a generated list of component information from the cube image shown in Figure 1a. A series of features can be extracted from the component list of the drawn image, N, to provide a series of performance metrics. By comparing the component data with a model list, M, which contains spatial and length data about the image which is copied by the test subject, an accuracy measure can be obtained which can quantify overall drawing "quality". This metric can be used to provide a diagnostic indicator and to monitor test subject performance over time. As an example of the diagnostic capability of the algorithm, three features are defined and the results presented from 5 neglect and 5 stroke control patients (Table 2) copying a cube image. Data was captured from stroke patients at Nunnery Fields Hospital, Canterbury. All patients had evidence of a right CVA (Cerebral-vascular disease) confirmed by CT scan and were diagnosed for neglect using clinical assessment and the Rivermead Behavioural Inattention Test (BIT) [5]. A BIT score below 130 indicates the presence of neglect. The maximum score attainable is 146. The following novel features are defined within the computer-based extraction system to facilitate the automated assessment of drawing responses:

The third feature, spatial difference, measures the mean sum of the differences between defined distance measures within the drawn and model images. Figure 7 shows the six distances between horizontal, vertical and diagonal components of the cube. Control Responses Horizontal

Vertical

Diagonal

Vertical

Diagonal

a)

b)

c) Neglect Responses Horizontal

d)

The first feature counts the number of component omissions within a drawing, providing a simple performance accuracy measure. The second feature, length difference, is defined as: n

∑ ( N (c length difference =

n

e)

) − M (c n ) )

1

n

where: n = number of components in drawn image N(cn) = length of nth component within drawn image M(cn) = length of corresponding nth component within model image

f) Figure 6. Horizontal, vertical and where applicable diagonal image bitmaps formed from the images shown in Figure 1 If a component is omitted then the relevant distance is not computed. Individual distances (d1 to d6) are calculated by finding the mean distance between drawn

points on an specific horizontal axis for vertical lines (and conversely vertical points for horizontal lines).

a b

Horizontal

Line q

Line r

c d e

d2 d1

Figure 8. Distance calculations

Distances d1 to d6 are defined:

Vertical P

d4

∑ q( x) − r ( x) dn =

1

p

d3

where :

q(x) = position of a first drawn point on selected axis r(x) = position of a second drawn point on selected axis p = number of distances measured. Diagonal

Spatial difference is defined: d5 n

d6

∑ ( N (d spatial difference =

n

) − M (d n ) )

1

n

where:

Figure 7. Cube spatial distance measures Where only one drawn point exists at a particular point on a given axis then a distance is not computed. Figure 8 shows 5 distances (a-e) between two vertical lines, q and r. Points a and e are not computed as two drawn points are not located. Distances are calculated between drawn points at every position on the vertical axis which are occupied by the drawn lines.

n = number of components in model image N(dn) = distance n within drawn image M(dn) = distance n within model image

Table 2 shows the results of applying these features to the 5 neglect and 5 stroke control patient’s cube responses. Subjects Control 1 and Neglect 1 produced the responses that are shown in Figures 1a and 1d respectively. As can be seen from the results in Table 2, there are clear quantifiable differences in all features between the neglect and control subjects.

Interestingly, the two neglect subjects that draw responses which contain the correct number of components (Neglect 3 and Neglect 4), produce length and spatial differences which are indicative of a neglect response, thereby showing that these features are more sensitive to neglect that the traditional component-based analysis.

5. Conclusions This paper has described a method of segmenting components contained within static hand-drawn responses from a geometric figure copying and completion task. It has been demonstrated, using a range of drawings collected from control and visuo-spatial neglect subjects, that the algorithm accurately extracts component information which can be used to form a performance metric on the overall drawing performance. The automatic identification of components and their relevant positional information allows a standardised assessment of drawn images and enables testing repeatability throughout the rehabilitation process. The main benefit for drawing assessment across a range of clinical and neuropsychological conditions which use figure drawing tasks is the removal of subjectivity in marking scheme application and hence more accurate diagnosis across a range of test subjects. A range of feature extracted from component list provide quantifiable performance metrics for the analysis of drawings. Although the data used within this study was captured using an on-line system, the algorithm can be used to assess drawn responses scanned from conventional pencil-and-paper testing methodologies and therefore can be used with existing data.

Acknowledgements The authors acknowledge the support of the South Thames NHS R&D Project Fund.

References [1] Vinter, A., Gras, P., Spatial features of angular drawing movements in Parkinson's disease patients. Acta Psychologica, 100, pp. 177-193, 1998.

[2] Beery, K., The VMI developmental test of visualmotor integration, Modern Curriculum Press, Cleveland, 1989. [3] Doricchi, F., Incoccia, C., Galati, G., Influence of figure-ground contrast on the implicit and explicit processing of line drawings in patients with left unilateral neglect. Cognitive Neuropsychology, 14, pp. 573-94, 1997. [4] Ericsson, K., Forrsell, L.G., Holmen, K., Viitanen, M., Copying and handwriting ability in the screening of cognitive dysfunction in old age. Archives of Gerontology and Geriatrics, 22, pp. 103-121, 1996. [5] Wilson, B., Cockburn, J., Halligan, P., Development of a behavioural test of visuospatial neglect. Archives Of Physical Medicine And Rehabilitation, 68, pp. 98-102, 1987. [6] Sword, J., Potter, J., Deighton, A., Guest, R., Donnelly, N., Fairhurst, M., Inter-rater reliability in the Rivermead Behavioural Inattention Test, Age & Ageing, 28 (supp. 2), pp. 60, 1999. [7] Lui, W.Y., Dori, D., From rasters to vectors: Extracting visual information from line drawings, Pattern Analysis and Applications, 2 (1), pp. 10-12, 1999. [8] Arias, J.F., Kastruri, R., Efficient extraction of primitives from line drawings composed of horizontal and vertical lines, Machine Vision and Applications, 10, pp. 214-221, 1997. [9] Amara, M., Courtellemont, P., de Brucq, D., Model selection in on-line handwriting, In proc. 3rd European Workshop on Handwriting Analysis and Recognition. [10] Yu, D., Yan, H., An efficient algorithm for smoothing, linearization and detection of structural feature points of binary image contours, Pattern Recognition, 20, pp. 57-69, 1997. [11] Haralick, R. M., Shapiro, L. G., Computer and Robot Vision, Volume I. Addison-Wesley, pp. 28-48, 1992.

Table 1. Component positional data from a cube drawing (shown in Figure 1a) Direction Horizontal Horizontal Horizontal Vertical Vertical Vertical Diagonal Diagonal Diagonal

Top Left 35,16 18,22 20,69 16,25 84,23 99,18 21,17 79,16 83,64

Bottom Right 98,19 78,23 82,70 18,68 87,67 104,62 33,23 101,23 100,70

Centroid 67,18 48,23 51,70 17,47 86,45 102,40 27,20 90,20 92,67

Length 66 61 63 44 45 45 15 28 18

Table 2. Performance metrics extracted from cube drawing tasks Subject

Age 78 70 83 77 84

BIT Score 143 140 145 146 144

Component Omissions 0 1 0 0 0

Length Diff. (pixels) 5.22 6.5 6.24 6.44 3.55

Spatial Diff. (pixels) 2.83 6.4 5.24 6.33 3.67

Control 1 Control 2 Control 3 Control 4 Control 5 Neglect 1 Neglect 2 Neglect 3 Neglect 4 Neglect 5

85 75 81 85 72

80 101 121 79 67

3 1 0 0 2

90.5 16.12 17.55 11.22 13.57

40.25 13.8 13.67 13.17 8.25