LApp - Open Lab, Newcastle University

LApp: A Speech Loudness Application for People with Parkinson’s on Google Glass Róisín McNaney, Ivan Poliakov, John Vines, Madeline Balaam, Pengfei Zhang, and Patrick Olivier Culture Lab, School of Computing Science, Newcastle University, UK {r.mcnaney; ivan.poliakov; john.vines}@ncl.ac.uk ABSTRACT

Reduced vocal volume in Parkinson’s is extremely common and can have significant social and emotional impact. We describe the development and evaluation of LApp—an application for Google Glass to help people with Parkinson’s (PwP) monitor their speech volume and cue themselves to speak louder when necessary. Our findings highlight enthusiasm for using the application both at home as a volume training tool and in public social settings as a situated cueing device. We contribute insights to the literature on how eyewear technologies can provide assistance to people with health conditions and offer insights for the design of future self-monitoring and management applications on Google Glass. Author Keywords

Parkinson’s disease; Wearable technology; Google Glass; Augmented reality; Self- monitoring ACM Classification Keywords

H.5.m. Information interfaces and presentation (e.g., HCI): Miscellaneous. INTRODUCTION

Parkinson’s is a complex neurodegenerative condition presenting in a heterogeneous range of motor and nonmotor symptoms. Reduced vocal loudness is a common problem for people with Parkinson’s (PwP) and is a frequently treated aspect of voice within speech and language therapy (SLT) for this group. Approximately 90% of PwP will experience a voice problem at some point. Distress, embarrassment and social isolation are highlighted as impacting factors [8] and with worldwide figures placing the prevalence of Parkinson’s at around 5 million people [9], this is an issue deserving great attention. The overall aim of SLT treatment targeting loudness is to maintain a louder voice in daily life, but this can be difficult once the input of the therapist is removed. One aspect of voice Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]. CHI 2015, April 18 - 23 2015, Seoul, Republic of Korea Copyright is held by the owner/author(s). Publication rights licensed to ACM. ACM 978-1-4503-3145-6/15/04…$15.00 http://dx.doi.org/10.1145/2702123.2702292

loudness issues in Parkinson’s that makes the maintenance of therapy gains particularly difficult is that PwP have an impaired perception of how loud they are speaking—i.e. they will feel as though they are shouting when speaking at a normal level, or speaking at a normal level when speaking quietly [10]. Lee Silverman Voice Treatment—a training program involving the intensive and constant stimulation of a loud voice through a set of simple tasks [10]—is regarded as the gold standard for treating vocal loudness problems in PwP. However, only around 20% of SLT practitioners use this treatment in the UK due to the amount of time it takes to complete [8]. Technology could prove beneficial in this context as it offers the potential to aid the self-directed monitoring and management of speech volume. For example, [5] proposed the use of a digital game to help PwP practice volume related tasks at home. While offering a home practice method, this does not address the difficulties PwP have in maintaining their volume within external settings. In this sense, the implementation of a technology to support in-situ prompting of volume, at times it is needed in everyday conversation, could prove beneficial. Our prior research [7] highlighted how platforms like Google Glass can provide a usable and acceptable assistive device for PwP, providing opportunities to deliver discreet and in-situ cues to aid the monitoring and management of Parkinson’s symptoms. We build on this prior work by describing the development and evaluation of LApp; a Glass application to aid in the monitoring and management of voice loudness issues experienced by PwP. The application was designed through workshops with 7 PwP, 6 of whom then used the application in a short field trial. We contribute to the emerging literature on how wearable technologies such as Glass can provide real-time in-situ prompts and cues related to conditions such as Parkinson’s, and provide recommendations for the design of monitoring and management applications for future work to build upon. THE STUDY

Our study was conducted in 3 stages: (i) an exploratory design workshop; (ii) iterative development of LApp and pilot testing; and (iii) real-world deployment of LApp on Glass. We describe each phase in the following sections. Phase 1: Exploratory Design Workshop

The first phase involved an exploratory design workshop with 7 PwP (Table 1). Participants characterised the heterogeneous nature of Parkinson’s in relation to their

symptoms. Our participants were aged between 60 and 81 and ranged from 5 to 15 years since diagnosis (YSD). They also had a diverse range of volume-related problems from mild to severe on the Voice Handicap Index (VHI)—a patient rated questionnaire used to gauge the emotional, physical and functional impact of voice problems [4]. We also used the Hoehn and Yahr scale [3] (H&Y)—a widely used scale for describing Parkinson’s, rated by a clinical member of the research team, ranging from 0 (no signs) to 5 (needing wheelchair or bedridden)—to assess the severity of their overall condition (see Table 1). Four participants had previously engaged in a SLT program to help treat their volume and two were awaiting therapy. Our workshop lasted 3 hours and had 3 aims; (i) to discuss personal experiences of loudness issues and the impact of this has on PwP; (ii) to discuss experiences of undergoing SLT and the strategies currently being used by participants to overcome volume issues; and (iii) to scope the space for using digital technologies to help prompt loudness in daily life. The workshop was audio recorded and transcribed. We conducted an inductive thematic analysis on the transcribed data using methods outlined by [2], by coding data at the sentence to paragraph level and drawing out themes across the data set, which we summarise below. Speech Volume, Confidence and Socialisation

Our participants expressed how significant an impact volume issues had on confidence and socialisation. Much frustration and embarrassment came from continuously being asked to speak louder by others:“I’m constantly being got at by people telling me to speak louder” (Michael), particularly when speaking to strangers:“I find I have to rephrase things and I get lost and then I get embarrassed, especially if it’s a strange situation” (Robert). Gill described her experience of being dismissed by retail staff in public: “I find that people don’t think you have the intelligence to understand what they’re talking about […] he said “I’ll just get your son to talk to me”. I just couldn’t get my words out for him to understand that I understood.” It is perhaps unsurprising then that participants withdrew from social situations: “I shy away more than I used to” (Robert).

Name Susan Neil Jill Michael Jerry David Robert

YSD Age VHI 13 79 37- mild 7 60 41- moderate 6 67 32- mild 15 81 78- severe 8 73 64- severe 5 80 27- mild 10 60 33- mild Table 1: Participant details

H&Y 3 2 1 4 3 3 2

taking deep breaths prior to speaking, or using stock phrases to “cue me back in” (Robert) to a louder voice. However, it was clear that these strategies can leave individuals feeling uncomfortable: “when you’re in the pub you raise your voice because there’s background noise but in a quiet place I would feel like I was shouting” (Robert). Susan, Jill and Neil explained how they would practice their loud voice prior to speaking (e.g. when making a telephone call) but would start loudly and “trail off” (Neil). Three of our participants struggled with performing their SLT strategies in the first place. Robert and Neil explained how they would forget their strategies in day-to-day life, while Michael just “didn’t see the point” of them at all. For some, being able to simply monitor their volume to aid self-understanding was important: “I don’t know how to gauge the loudness of my voice, whether it’s too quiet or too loud. I don’t have any standard outside which is the equivalent of the people listening to me” (Michael). Prompts were considered useful for helping to understand how volume changes over the space of a conversation: “say I was to speak for 10 minutes, it’s a long time, I start tailing off so I need something to bring me back to realise my volume” (Robert). During the workshops different types of cues were suggested by participants (visual, haptic and auditory). There was some disagreement in relation to what would be preferred, with some suggesting audible cues that only they could hear, and others preferring visual or haptic cues they could respond to. Participants came to an agreement that cue preferences depended on context, but all felt comfortable with the idea of using a ‘traffic light’ style visual cue.

Another concern was a feeling of being ignored. Susan explained how: “coming over in the taxi he didn’t know the way and I tried to tell him but I got no answer, I gave up in the end.” Michael also described: “frequently during a conversation with other people I can’t get myself involved, I’m just not speaking loud enough to make an impact”. It was clear that these feelings of dismissal or invisibility had significant impacts on the motivation to interact with others. There was an overriding sense of ‘giving up’: “you lose the thread of what you’re trying to say, so you just shut up”(Jerry). This was echoed by Jill: “I try to think speak louder, but when you get into your spiel people are constantly saying speak up, so now I don’t speak at all”.

Phase 2: Development, Piloting and Iteration of LApp

Monitoring and Managing Loudness

To test initial perceptions of this cueing method and ensure feasibility and safety we ran a small user study with 8 participants (5 males, 3 females with ages ranging from 20-

Participants discussed their experiences of SLT and the strategies they used to manage loudness issues, for example

The second phase involved the design and development of the Loudness monitoring and self-management application (LApp). Our initial design drew upon the ‘traffic light’ idea presented by participants in phase 1. This used a red/amber/green coloured system visualised as an orb on the Glass display which fluctuated accordingly with the speaker’s vocal volume (Figure 1). The orb projected to the limits of a white line representing a conversational speaking volume (60dB (SPL)[1]). The orb was green when speech was loud enough and changed to red or amber when volume was too quiet (40dB under target) or nearing an appropriate level (20dB under target) respectively.

Figure 1: First (left) and second (right) iterations of LApp cue.

40) who did not have Parkinson’s or volume difficulties. Participants were asked to use LApp for approximately 30 minutes while carrying out a series of social interactions of their choosing in a range of settings with varying background noise. All participants provided very similar feedback about LApp which indicated some required changes. While they found the application easy to access and use, the orb-based feedback mechanisms required too much continuous attention to monitor its size. Instead, participants advocated the use of a cue that could be monitored better with peripheral vision. To this end we redesigned the cue as a large ‘thumbs up’ symbol that could be more easily seen peripherally and to provide positive reinforcement that appropriate volume levels were being met. When volume reduces the symbol gradually disappears. Participants additionally had difficulty reaching the 60dB target at times, so we added a feature that allowed users to set their target volume. On reflection, we realised that it may have been difficult for PwP to reach this target and, given that the aim of our application was to promote increased vocal loudness not reach a standardised dB level, we felt that having users set their own goals that they felt comfortable with would be more appropriate for maintaining functional gains in day-to-day life. Phase 3: Field Trial of LApp

The final stage of our study was a series of three day deployments of LApp on Glass. We aimed to explore PwP experiences of using LApp in real world contexts, to see how they familiarised themselves with and used the device, and to promote discussion related to the design of in-situ prompts to monitor and self-cue loudness in speech. We deployed the LApp on Glass with 6 of our original participants (David was unable to take part). Participants were visited in their homes and given a demonstration of how to use LApp. A researcher helped them sample a loud voice that was comfortable for them, to be used as the target volume that they would aim for when using the application. LApp was designed to support self-directed management of volume and, in this sense, participants were advised that they could make a specific decision not to raise their voice if the situation required. Participants were provided with an instruction manual for reference and were assured they could contact a researcher at any time for support. They were asked to use the application as much as possible during the period they had the system, within normal, everyday situations. At the end of the trial, each participant was interviewed in their own home for approximately 40 minutes. Each interview was transcribed and thematically analysed in the manner described in phase 1.

Name

Times Accessed

Susan Neil Jill Michael Jerry Robert

4 5 3 3 18 11

Target dB (TdB) 45 45 51 48 55 52

Above Target (AT) 8 5 9 13 28 32

Below Target (BT) 4 12 11 2 5 6

Table 2: Averages of usage data from 3 days LApp trial Results

Participants varied in the ways they used LApp. While Neil, Robert, Jerry and Jill took Glass out and about with them to different places, Susan and Michael preferred to use it solely at home. All participants used Glass while talking to friends at some point during the trial. Table 2 displays average figures for; number of times LApp was accessed, target dB level participants were aiming for (TdB), times voice reached above TdB but dipped before triggering the cue (AT), and times voice reached below TdB during cue presentation but increased prior to its disappearance (BT). Being Cued by LApp – Conversing and Practicing

All of our participants engaged enthusiastically with LApp during the trial and found benefits in the feedback the system provided. Jerry immediately found that he was increasing his volume to reach the LApp target: “it was good for getting your voice up….you had to speak up to get the [icon] on so I found that very good”. All used LApp for its designed function—as a way to provide in-situ cues related to the loudness of speech. Jill found great comfort in the fact that she was able to confirm her voice was loud enough during a conversation with a friend: “Going into the conversation, when [friend] said ‘pardon’ twice to me. I had the [thumbs up], so it gave me confidence to know it wasn’t me, it was him […] every time that someone says pardon to you it knocks your confidence a little bit […] so it gives you confidence to know that you are not always the one at fault”. Similarly, Michael used LApp while socializing with friends, with his wife noting: “you were speaking quite clearly when you had it on […] it was lovely to hear you”. Michael’s wife noted improvements in the loudness and overall quality of his speech even just for the short period of this study. Some participants felt that the real benefit of LApp however was as a practice aid. For example, Neil explained how “reading aloud was good practice I found” and could see this as a benefit for him in the future: “if I was in a situation where I was being told to repeat myself more than I liked I would go away and practice”. However, he also reported: “when I’m wearing it I’m conscious of the fact that I have to perform”. Frustrations around Feedback

As noted earlier, perceptions of vocal loudness can be impaired in PwP, leading to difficulties in registering how loud the voice actually is. During the first session, a researcher helped each individual to set a target volume on LApp which was comfortable for them to reach, yet loud enough for the researcher to hear. Despite this, several

participants had difficulty reaching and maintaining the target volume once the researcher had left. Robert and Michael found that the LApp “didn’t work” or “just couldn’t pick up my voice” at points during the trial. Interestingly, during the final interviews (and for Robert during an additional session, where a researcher visited his home following complaints about the application “just not working”) LApp functioned as expected: “when you came round and it was working, 5 minutes later it stopped working and since then I’ve just had a series of disasters, it just doesn’t work” (Robert). On demonstration Robert was trailing off in terms of his loudness during extended vocalisation and it was observed that, despite speaking at a sufficient volume, he was speaking in short bursts producing a disjointed voice: “in the end I was shouting at the thing…thumbs up!...icon!”. Considering that LApp was computing average dB over a 2 second time space his overall level was therefore not sufficient to hit his volume target (see Table 2). Since short rushes of speech are common feature in Parkinson’s, this issue warrants careful consideration for a future applications aiming to support a wider range of PwP. Problems with Glass

Finally, some of the participants experienced frustrations with Glass. While there was an appreciation of the potential of LApp, some found it difficult to set and reset their target volume. The primary issues participants faced related to tremor in their hands and arms which led to difficulties navigating Glass’ menus: “I’d touch it occasionally and set it off on a chain of events” (Neil). However, a more immediate challenge was that the system lost battery charge very quickly. This caused particular angst on occasions when the device was being worn while out and about: “I put them on and took them to the garden center. I felt great, didn’t seem like anyone was looking at me, I felt comfortable, I could see my [thumbs up] and then all of a sudden it went off” (Jill). While the participants understood that the system was a prototype, it highlighted the potential dangers of in-the-wild cueing if the technological platform is still immature. DISCUSSION

While our ambition here was to develop LApp to be an insitu self-management technology, we saw how being reliant on a system like this can lead to considerable frustration if things go wrong. Given the noted issues of personal distress and embarrassment associated with Parkinson’s [8], this emphasizes the importance of well-tested applications and a non-trivial level of technical support (e.g. an accessible troubleshooting guide or access to a person). Researchers should also be aware that negative feedback could cause frustration with users, even if this is an accurate portrayal. To this end, having easy methods of altering settings would be beneficial. Furthermore, while there were some frustrations with using LApp out and about, some of our participants placed great value on home practice of their voice using the device. This countered our primary envisioned use-case of LApp. Taking a social shaping of technology perspective (e.g. [6]) then this should be of no

surprise—in this case, the features of LApp facilitated new ways of understanding one’s speech, appropriated in an unexpected (yet rather obvious) manner by PwP. As such, this highlights that tools to support the development of voice management (such as [5]) should not be entirely dismissed in favor of tools for in-situ, real time, feedback. Furthermore, in order for Glass to be more usable to PwP there should be consideration into the sensitivity of touch gestures. The provision of longer time frames for accessing features, as well as ability to lock down Glass to a singular application could aid functionality in this sense. Our participants also had issues with battery life so future applications should consider this, particularly for systems aiming to provide continuous in-situ prompting. CONCLUSIONS

Our study of LApp was with a small sample of PwP and over a very short trial period. However, our findings were promising and demonstrate worth in exploring the development and longer-term trial of applications to support the monitoring and management of vocal loudness through Glass. However, future work is needed to refine the application further so that it is easier to change volume settings and to improve Glass’ limited battery resources through software optimisation or additional hardware. ACKNOWLEDGEMENTS

This research was funded through the RCUK Digital Economy Program SIDE (EP/G066019/1). REFERENCES

1. 2.

ASHA. Noise. Audiology Information Series 7976. (2011) Braun, V, & Clarke, V. Using thematic analysis in psychology. Qual res in psych, 3, 2(2006), 77-101. 3. Hoehn M, & Yahr M. Parkinsonism: onset, progression and mortality. Neurology 17, 5 (1967), 427–42 4. Jacobson, B. H. et al. The voice handicap index (VHI) development and validation. Am Jour Speech-Lang Path 6, 3 (1997), 66-70. 5. Krause, M., et al. A digital game to support voice treatment for Parkinson's disease. In Proc of CHI'13, ACM, 445-450. 6. MacKenzie, D., and Wajcman, J. (eds.). The social shaping of technology (2nd Ed.). Open University Press, Buckingham, 1999. 7. McNaney, R. et al. Exploring the acceptability of google glass as an everyday assistive device for people with Parkinson's. In Proc of CHI 2014, ACM, 2551-2554. 8. Miller, N., et al.National survey of speech and language therapy provision for people with Parkinson's disease in the United Kingdom: Therapists’ practices. Int Jour Lang Comm Disord, 46, 2 (2011), 189-201 9. Olanow C. W. et al. The scientific and clinical basis for the treatment of Parkinson’s disease. Neurology 72, 21 (2009), 1-136. 10. Ramig, L. et al. Intensive voice treatment (LSVT®) for patients with Parkinson's disease. Jour Neurol, Neurosurg Psych, 71, 4 (2001), 493-4.