Correlation based system to assess the completeness and correctness of cognitive stimulation activities of elders J. A. González-Fraga*, A. L. Morán, V. Meza-Kubo, M. Tentori, E. Santiago Facultad de Ciencias, Universidad Autónoma de Baja California, Km 106 Carretera Tijuana-Ensenada, Ensenada, B.C. 22860, México ABSTRACT During a cognitive stimulation session where elders with cognitive decline perform stimulation activities, such as solving puzzles, we observed that they require constant supervision and support from their caregivers, and caregivers must be able to monitor the stimulation activity of more than one patient at a time. In this paper, aiming at providing support for the caregiver, we developed a vision-based system using an Phase-SDF filter that generates a composite reference image which is correlated to a captured wooden-puzzle image. The output correlation value allows to automatically verify the progress on the puzzle solving task, and to assess its completeness and correctness. Keywords: Correlation filters, Vision-Based stimulation system
1. INTRODUCTION The World Health Organization (WHO) has reported that Alzheimer disease (AD) has risen to the sixth place in the list of leading causes of death [2]. It also reports that there are nearly 18 million people worldwide suffering from AD and it is projected to double to 34 million by 2025 due to population aging. For this reason the WHO has called to pay special attention to the risk factors of the disease and to seek preventive measures to be taken to postpone its appearance [3]. Currently, there is no cure for AD [4]. But in the absence of a cure, a multidimensional therapeutic approach that includes both pharmacological and non-pharmacological interventions is recommended. Non pharmacological interventions are aimed at optimizing cognition, behavior and function of subjects with dementia [5]. One of these non-pharmacological interventions is Cognitive stimulation. The aim of cognitive stimulation is to stimulate and maintain existing cognitive abilities, with the intent to maintain or improve cognitive functioning and reduce dependence of the patient, and this by working on the remaining skills of the person and avoid the frustration of the patient. Cognitive stimulation is used to treat individuals with mild to moderate dementia in sessions of active stimulation through activities that aim to improve and maintain overall cognitive functions (e.g. memory, language, attention, concentration, reasoning, abstraction, arithmetic, praxis and gnosis) [6]. Recently published research works provide evidence on the fact that a person who participates often in cognitive stimulation activities reduces the risk of suffering AD, or improves his/her cognitive behavior [7], [8]. To this end, and in order to provide technological support for cognitive stimulation activities, several projects have introduced the use of computers and mobile devices (e.g. cognitive stimulation through web – desktop computer [9] and the Orienting Tool – PDA [10]). _____________________________ Further author information – J. A. G-F.:
[email protected], phone: +52-646-174 4560 ext 113; fax: +52-646-174 4560
Applications of Digital Image Processing XXXII, edited by Andrew G. Tescher, Proc. of SPIE Vol. 7443, 74430P · © 2009 SPIE · CCC code: 0277-786X/09/$18 · doi: 10.1117/12.826673
Proc. of SPIE Vol. 7443 74430P-1
The aim of this work is proposing the use of alternative interfaces which are more suited to the abilities of elders. Our proposed vision-based system for cognitive stimulation activities uses an SDF filter that generates a composite reference image which is correlated to a captured wooden-puzzle image. The output correlation value allows to automatically verify the progress on the puzzle solving task, and to assess its completeness and correctness. The rest of the paper is organized as follows. Section II introduces and motives our proposal to use tangible interfaces as an alternative to the use of mouse and keyboard interfaces. Section III describes the tangible vision based context-aware system for the assessment of progress and correctness of cognitive stimulation activities. Section IV describes awareness of the correctness and completeness of elder’s puzzles solving. Finally, section V presents our conclusions.
2. UNDERSTANDING COGNITIVE STIMULATION ACTIVITIES Based on the results from an observational study at a residence for elders with cognitive decline, we identified a set of design insights that designers of a system that aims at providing support for cognitive stimulation through tangible and ubiquitous computing should consider. These insights include: (i) Patients should be aware of the correctness and progress of their activity; (ii) Caregivers should be able to monitor the stimulation activity of more than one patient at a time; (iii) Caregivers should have awareness of whenever they are required to change the materials to a patient; (iv) Caregivers should have awareness on the activity level and on the correctness of the activity of patients, in order to detect and intervene whenever is necessary to encourage or provide feedback to them. Concerning insight (i), the cognitive decline that characterizes these users, generates in them anxiety and doubts regarding their capacity to perform cognitive stimulation activities. In this sense, having a mechanism that automatically provides them with information that encourages them to continue performing the activity and that provides them with feedback regarding their progress on that activity becomes necessary. Concerning insights (ii-iv), cognitive stimulation sessions at the residence are usually performed in groups, so that a single caregiver must supervise more than a single patient. For this reason, monitoring and notification mechanisms should be provided so that the caregiver is able to obtain an awareness level on the activity of the patient, and on the use of material. This would allow the caregiver to provide the patient with feedback, encouragement and help whenever is required. To address these opportunity areas, we next propose the design of a system that automatically assess the completeness and correctness of the elder’s activity.
3. VISION-BASED ALGORITHM AND TANGIBLE PUZZLES The completeness progress of solving the tangram is measured by means of the correlation via Fourier transform. Correlation is used as a similitude measure between two images: the target tangram configuration and the incomplete tangram arrangement. We use a Phase-SDF filter to construct the composite target image. 3.1 Design of the Phase-SDF filter Basically SDF filters use a set of training images, which are descriptive and representative of expected distortions of an object. The ECP-SDF filter produces Equal Correlation Peaks (ECP) on the output in response to the training images. Let {ti(x,y); i=1,2,…,N} be a set of (linearly independent) training images each with d pixels. The SDF filter function h(x,y) in the spatial domain can be expressed as a linear combination of a set of reference images as:
h SDF = R (R T R ) −1 u ,
(1)
where R denotes a matrix with N columns (number of training images) and d (number of pixels in each training image). The i-th column is the ti(x,y) image in vectorial form. A column vector is obtained from each image by rearranging the rows of the matrix, from left to right and from top to bottom. The (i,j)-th element of the matrix S=(RTR) is the value at the origin of the cross-correlation between the training images ti(x,y) and tj(x,y), and the vector u is set to the unit for each training image (the desired output on the correlation plane). Setting all the values of u to the same value, the filter is able to detect patterns belonging to the same class. [11].
Proc. of SPIE Vol. 7443 74430P-2
We construct a Phase-SDF filter, applying the concept of the Phase only filter to the composite image in equation 1. The puzzle tangram is composite from seven geometric pieces: two big triangles, a medium sized triangle, two small triangles, a square and a parallelogram (see Figure 1). Let t(x,y) be the image representing the proposed configuration of the tangram. The target configuration of the puzzle is split into 7 independent images. These images are used to train the SDF filter: 7
t ( x, y ) = ∑ t i ( x, y ) ,
(2)
i =1
Note that none of the parts of the target tangram configuration was centered. Each piece of the training tangram will contribute to the total correlation at the same point.
Fig. 1.. Tangram puzzle.
3.2 Vision-based algorithm The setup of the vision system consists of a work table, a web camera (see Fig. 2), and a recognition system based on a correlation filter. The camera is fixed at 80 cm below the work table; it captures the progress of the puzzle every 5 seconds. The captured image is processed with MATLAB routines. The algorithm can be summarized as follows: 1. Choose the tangram configuration to work with. 2. Design an SDF filter h(x,y), using the independent pieces of the puzzle as training images by means of equation 1. 3. The image captured is called t(x,y), then we compute their Fourier Transform S(u,v), and also compute the phase – SDF filter H(u,v). 4. Compute the correlation values. 5. The progress of the activity is computed according to the maximum correlation value on the output. In others words, the composite image is compared with the captured image of the puzzle. Correlation measures the similitude between the two images. The correlation value is used like a clue in order to solve de puzzle. As an example, Fig. 3 shows the solution of a tangram configuration captured by the system, a similar image is showed to the user to work it. Fig. 4 shows the correlation values and the associated captured image of the puzzle in progress. The higher the progress in solving the puzzle, the higher correlation value is found.
Proc. of SPIE Vol. 7443 74430P-3
System verifying the progress on the puzzle solving task
Tangible tangram Work-table
Camera
Fig. 2. Setup of the vision system.
Fig. 3. Example of a tangram configuration..
Fig. 4. Correlation values obtained with the Phase-SDF filter (synthesized with a similar image as in Fig. 3) and the test images of the incomplete puzzle b-g
Proc. of SPIE Vol. 7443 74430P-4
4. AWARENESS WIDGETS The results of our study, along with our vision algorithm, can be applied in the design of widgets that provide awareness of the correctness and completeness of elder’s puzzles solving. Here we discuss two of those widgets. 4.1 The Puzzle Progress Gauge During a cognitive stimulation session a common technique used by caregivers is puzzle solving (Fig. 5b). Elders enjoy solving puzzles; however, due to their decline, they frequently have problems in solving them or maintaining their engagement in such activity. For instance, elders might start selecting one piece repeatedly trying to fit it into an incorrect region of the puzzle. Since caregivers have to oversee the activity of several patients, they frequently missed this type of events and as a consequence, elders lose interest in the session, thus resulting in its abandonment. To cope with these, we designed a Puzzle Progress Gauge (Fig. 5a) aiming at keeping the elder engaged in solving the puzzle by providing awareness on the correctness and completeness of the puzzle being solved. The Puzzle Progress Gauge has a situational bar (i.e. the horizontal bar in Fig. 5a) that indicates how similar the composite image is with that of the wooden puzzle being manipulated by the elder. When the situational bar is positioned in the blue zone of the Puzzle Progress Gauge, it indicates that the puzzle being solved is correct – opposite as when it’s positioned in the red barcodes (i.e., the red zone). The situational bar distance for the origin indicates the level of correctness or incorrectness. The higher the situational bar, the higher the level of correctness of the wooden-puzzle being manipulated. With this bar an elder can have a sense on how well or wrong he is doing, and based on this, he might decide to change the location of some pieces of the puzzle in order to increase its level of correctness –this gauge can then be also used to provide hints each time a piece is collocated.
Fig. 5.. (a) The Puzzle Progress Gauge (b) A caregiver using the mobile AwarePuzzle (c) A notification received by the caregiver in her mobilePhone
4.2 The Aware Puzzle As discussed, interactions are very important during a cognitive stimulation session, as they allow caregivers and elders to maintain the continuity of their activity. However, the intention of some of these elder’s interactions might not be perceived by caregivers; resulting thus, in lost of interest or attention of some of them. For instance, a caregiver might miss when an elder has achieved a goal, such as solving a puzzle; missing the opportunity to congratulate him. Our algorithm by automatically assessing the completeness level of the puzzle being solved could opportunistically inform when a notification must be sent for a caregiver to encourage an elder – thus providing awareness of the patient’s needs. For instance, when an elder successfully finishes a puzzle and the algorithm detects this event, a notification might be shown in her mobile phone. Fig. 5b and 5c illustrates the Aware Puzzle – running in a mobile phone. In Fig. 5b, a caregiver uses his mobile phone to consult messages or assign contextual information to messages, while in Fig. 5c she consults a notification in her mobile phone. Three types of messages might be informed by the system: • Encouragement, in this case a message is sent to the caregiver notifying a positive event related with problem solving. For instance, as when an elder correctly completes solving a puzzle.
Proc. of SPIE Vol. 7443 74430P-5
• Promote engagement, in this case when an elder has been inactive for more than a lower threshold of time, a message is sent to the caregiver requesting him to promote the engagement of the elder. For instance, if the elder has been inactive for up to 30 sec, this kind of message is sent to the caregiver. • Notify a request for assistance, in this case when an elder’s period of inactivity is greater than a higher threshold (e.g. five minutes), or whenever a patient has explicitly request it, a message is sent to the caregiver for him to look at the patient and provide the required assistance (Fig. 5c).
5. CONCLUSIONS In this work we present the proposal of a system to assess the completeness and correctness of tangible cognitive puzzles in an automatic manner by means of computer vision. We use an algorithm that computes the correlation level of a composite image generated by a Phase-SDF filter that is compared with a real-time obtained image of the puzzle being solved by the elder. The algorithm’s output is used to provide feedback to the patient and to notify the caregiver regarding the progress, or lack of it, in the performance of the stimulation activity. Based on these results, and on those of an observational study of a cognitive stimulation session, we informed the design of an awareness system that provides assistance for the execution of the activities that occur at the stimulation session, both for the patient and for the caregiver. Future work aims at performing an evaluation of the proposed system.
REFERENCES [1] [2] [3] [4] [5] [6] [7] [8]
[9]
[10] [11]
Wan, H., M. Sengupta, V. A. Velkoff, and K. A. DeBarros, U.S. Census Bureau, Current Population Reports, P23209, 65+ in the United States: 2005, U.S. Government Printing Office, Washington, DC, 2005. Chicot, J., S. Rajkumar, P. Tanyakitpisal, V. Chandra. Alzheimer Disease: The brain killer. Available: http://www.searo.who.int/LinkFiles/Health_and_Behaviour_alzheimers.pdf Population Division, DESA, United Nations. DEMOGRAPHIC DETERMINANTS OF POPULATION AGEING. Available: http://www.un. org/esa/population/publications/worldageing 19502050/ Stevens, L., Alzheimer Disease. JAMA. 2001;286(17):2194 Woods, R. T., Non-pharmacologial techiniques. In: N Qizilbash, LS Schneider, E Cui, P Tarior, H,Brodaty J. Kaye, T. Erkinjunyi. Evidence-based Dementia Practice. Ed: Blackwell Science Ltd. Oxford 2003: 428-446. Olazarán, J. et al. Benefits of cognitive-motor intervention in MCI and mild to moderate Alzheimer disease. NEUROLOGY 2004;63:2348-2353 Ball, K. D. Berch, K. Helmers. Effects of Cognitive Training Interventions With Older Adults: A Randomized Controlled Trial. JAMA. 2002;288(18):2271-2281. Spector, A., L. Thorgrimsen, B. Woods, L. Royan, S. Davies, M. Butterworth, and M. Orrell. Efficacy of an evidence-based cognitive stimulation therapy programme for people with dementia: Randomised controlled trial. The British Journal of Psychiatry, Sep 2003; 183: 248 – 254. Tárraga L, M Boada, G Modinos, A Espinosa, S Diego, A Morera, M Guitart, Balcells, O L López and J T Becker. A randomized pilot study to assess the efficacy of an interactive, multimedia tool of cognitive stimulation in Alzheimer’s disease. J. Neurol. Neurosurg. Psychiatry 2006;77;1116-1121. Wu, M., Baecker, R., and Richards, B. (2007). Designing a Cognitive Aid for and with People who have Anterograde Amnesia, In J. Lazar, (Ed.), Universal Usability, John Wiley & Sons, 317-356. B.V.K. Vijaya-Kumar, “Tutorial survey of composite filter designs for optical correlators”, Applied Optics, 1992 vol. 31, pp. 4773-4801
Proc. of SPIE Vol. 7443 74430P-6
