assess the intensity of neurological disease symptoms

Wearable sensor-based data analysis for neurological disease symptoms evaluation utilising quantitative approach. Authors: Lt.Col. Mariusz Chmielewski, Ph.D. Eng. Michał Nowotarski, M.Sc. Eng. Military University of Technology Cybernetics Faculty Institute of Computer and Information Systems 22nd International Conference on Circuits, Systems, Communications and Computers (CSCC 2018), Majorca, Spain, July 14-17, 2018 Military University of Technology www.wat.edu.pl

Cybernetics Faculty www.wcy.wat.edu.pl

Wszystkie prawa autorskie i intelektualne zastrzeżone – zespół iSULIN & Wojskowa Akademia Techniczna. Materiały wykorzystane do wytworzenia materiału są własnością intelektualną jego twórców. Kopiowanie, powielanie i modyfikowanie materiału bez zgody twórców jest zabronione.

Presentation outline • • • •

• • •

Research domain and main assumptions Parkinson’s Disease symptoms and means of their recognition Developed Process of Clinical Trials – PD clinical trials (requirements) Solution – smartphone application for supporting medical diagnostics and survey assistance (PATRON + IQPharma CTA) • Survey – sensor tests – data aggregation and quantitative assessment • Utilised biometric techniques – inertial data, surface electromyography • Selected wearable for patient’s assistance • Mobile assistance application and server-side reporting services Selected gestures their significance in advanced PD diagnostics Preliminary laboratory tests & clinical tests results Analytical services assessing pharmaceuticals in clinical trials

Military University of Technology www.wat.edu.pl


2

Research Thesis Research Domain and main asumptions

Provide a quantitative method for PD neurological symptoms (not only tremor) assessment using mobile biometric tools Assumptions: •

Provide method for objectively assessing the medical treatment effectiveness.

•

Inertial data taken from limbs can be used to assess the intensity of neurological disease symptoms.

•

Electromyography can be used to identify intensity of tremors and distinguish real essential tremors (lat. tremor essentialis, tremor senilis)

•

Analysis of gestures can be used for bradykinesia and rigidness assessment

•

Utilisation of screen interactions for reflex assessment Domain: development of biomedical sensor data analitical tools Military University of Technology www.wat.edu.pl


3

Parkinson’s Disease – symptoms with respect to presented method Mechanisms of PD disorders

PD Symptoms (asymmetric): 1. Muscle rigidity - Rigidity is the inability of the muscles to relax normally. Most people with the disease develop some degree of rigidity, or stiffness of limbs. This rigidity is caused by uncontrolled tensing of muscles and inhibits your ability to move about freely. 2. Tremor – (shaking) begins in the hands and arms, although it can also occur in the jaw or foot. Tremor typically involves the rubbing of the thumb against the forefinger, and is more apparent when the hand is at rest, or you are under stress. 3. Bradykinesia - slowing down of movement and the gradual loss of spontaneous activity 4. Changes in walking (gait) - This commonly includes the inability of a person to swing their arms naturally while walking, taking short shuffling steps, 5. Mental effects – lack of coordination and degredeted reflexes

Initial clinical symptoms in Parkinson disease include the following: • •

Neurotransmitter - Dopamine (DA, contracted from 3,4-dihydroxyphenethylamine)

• • • • • • • • • •

Tremor A subtle decrease in dexterity; for example, a lack of coordination with activities such as playing golf or dressing (about 20% of patients first experience clumsiness in one hand) Decreased arm swing on the first-involved side Soft voice Decreased facial expression Sleep disturbances RBD, in which there is a loss of normal atonia during REM sleep: In one study, 38% of 50-year-old men with RBD and no neurologic signs went on to develop parkinsonism [22] ; patients “act out their dreams” and may kick, hit, talk, or cry out in their sleep Decreased sense of smell Symptoms of autonomic dysfunction, including constipation, sweating abnormalities, sexual dysfunction, and seborrheic dermatitis A general feeling of weakness, malaise, or lassitude Depression or anhedonia Slowness in thinking



5

Clinical tests – UPDRS-based surveying application of sensor data for treatment assessment Analog clinical tests: • Available at the clinical location • Require physitian assistance • Require additional resources • Require additional processing • Not synchronised • ….



6

Clinical tests – constraint server-controlled surveying for treatment assessment



7

Clinical trials assistance toolkit – proces flow with eventdriven behaviour delivered by server-side interactions Wybranie jednej pozycji z listy

Wybranie opcji bezwzględnie

Dodatkowe opcje w trybie administratora

Wybranie pozycji ze znakiem przeglądaj

Kreator harmonogramu (opcja względnie )

Kreator harmonogramu (opcja bezwzględnie )

Pierwsze uruchomienie aplikacji lub po wylogowaniu użytkownika

Logi zdarzeń Ostrzeżenie przed wprowadzeniem nowego harmonogramu

Przyznawanie wymaganych uprawnień

Pierwsze logowanie

Szczegóły logów zdarzeń

Zalogowanie jako administrator

Historia Zalogowanie jako zwykły pacjent

Ekran podsumowujący zebrane dane sensoryczne (ankiety lub badania)

Dodawanie nowego sensora

Admin Menu

Admin Dashboard Automatyczne logowanie

Ustawienia pacjenta

Ustawienia administratora

Logowanie Wybranie przycisku Zarejestruj się

Zastosowanie odpowiednich filtrów

Wybranie i przytrzymanie logotypu aplikacji

Rozwinięcie dodatkowych opcji

Dashboard

Harmonogram dnia

Notyfikacja badania sensorycznego uruchamia badanie

Wywołanie zdarzenia przez firebase

Wybranie wykresu kołowego

Menu

Uruchomienie głównej notyfikacji

Okno dostępu do trybu administratora

Rejestracja użytkownika

Wybranie osi czasu

Filtr ankiet

Filtr leków

Filtr badań sensorycznych

Ustawienia badania sensorycznego

Raporty dzienne Dodatkowe ustawienia badania sensorycznego

Dostępne ankiety Szczegóły wybranego dnia

Raporty dzienne w postaci diagramów kołowych

Badanie z użyciem myo

Notyfikacja badania ankietowego wywołuje badanie

Firebase

Badanie bez użycia myo

Notyfikacje

Badanie ankietowe

Łączenie z sensorem

Notyfikacja leków generuje powiadomienie

Powiadomienie o lekach Badanie sensoryczne



8

Electromyography (EMG) Amplitude: Frequency:

1-10 mV 20-2000 Hz



9

Concept of Disease symptoms recognition and symptoms intensity evaluation •

•


Remote assessment of tremors and dyskinesias • Recording and evaluation during medical survey • Actigraphy - inertial data taken from forearm or wrist – MYO, smartwatch • Muscle tonicity - myography data taken from forearm – MYO • Heart rate - data taken from wrist – smartwatch • On-demand medical examination triggered by the clinical test system or a physician Easily accessible tremor examination at home for patients • Evaluation of symptoms at home to recognise current state (ON/OFF) and to assess medicine intake • Evaluation of dyskinesias • Identify the schedule for optimal drug intake


10

Evaluation of Parkinson’s Disease symptoms Tremors & Diskinesias •

Evaluation of disease progress based on: • Frequency and intensity of Parkinsonian tremors (sensor-based) • Occurance of ON and OFF phases (time, duration, intensity) • Sleep cycle schedule – evaluation of sleep quality • Analysis of the drug effect on the occurrence of dyskinesia • Self-assessment surver analysis – patient’s comfort level • Analysis of survey completion – patient’s compliance



11

Solution – PATRON smartphone application for supporting medical diagnostics and test survey recording

Selected multi-sensor devices for patient’s assistance Mobile application Smartphone

SmartWatch

SmartBand

Motorola 360

MYO

PPG (photopletysmography), accelerometer, gyro, magnetometer

8 segments Surface EMG, accelerometer, gyro, magnetometer

Android device

PPG, accelerometer, gyro, magnetometer,



12

Medical Survey assessing patient’s state and efficiency of neurological drugs



13

[end of the interval]

Signal Analysis Algorythm – sEMG+ACC

Count number of zero crossings and maximum amplitude for ACC data of the interval

1 Identify an active hand and recalibrate algorithm

Evaluate mean waveform length, mean absolute value, median, mean and peak frequencies for ACC data of the interval

Choose sensors [smartphone, smartwatch, Myo] and examination time

Receive data for the hand

[selected sensor: smartphone or smartwatch]

[selected sensor: Myo]

Count number of zero crossings and maximum amplitude for EMG data of the interval

Store received data for the current time window [current time > examination time]

[interval in progress] [end of the interval] Count number of zero crossings and maximum amplitude for ACC data of the interval

[current time < examination time] Save calculated values in the internal storage

Evaluate mean waveform length, mean absolute value, median, mean and peak frequencies for EMG data of the interval

Show calculated values on the display

Evaluate mean waveform length, mean absolute value, median, mean and peak frequencies for ACC data of the interval

1

Military University of Technology [selected sensor: www.wat.edu.pl[selected smartphone or sensor: Myo] smartwatch]


14

Sensor Examination on demand assessment of patient’s state (ON/OFF) and dyskinesia

User Test Layout Test Controller Myo Controller Reasoner onClick(Button PrepareTest) startConnecting() connectToMyo(Myo) sendMessage(Message) showMessage(Message)

onClick(Button StartTest) startTest() startSendData()

1. Examination configuration [time, data sources, side] 2. Instructions before examination 3. Examination procedure [30 s – 2 min] 4. Recommendation – rule-based classification 5. Examination inspection & review

loop [testTime < timeLimit]

addData(Value, Sensor)

addData(Value, Sensor) updateChart(Value, Chart)

calculate() sendResult(Message) showResult(Message)



15

Sensor assisted examination assessing intensity of selected symptoms Sensor examination 1st person perspective


Sensor examination Sensor configuration and usage


Essential tremors identification MYO sensor implementation



17

Analytical algorithms and methods of biomedical signal analysis

•

Inertial data analysis • Feature extraction • FFT analysis [2-8 Hz] • Spectrum analysis • Hamming’s window

•


Electromyography (forearm) • EMG accumulated signal analysis • Feature extraction – 2-20 s timewindow (depending on specific symptom and gesture) • Muscle activation function • Signal Power distribution • …. Cybernetics Faculty www.wcy.wat.edu.pl

18

Selected gestures – PD symptoms correspondences      

Test #1: Hand held steady and still Test #2: A gentle hand shake Test #3: Fist clenched with all strength Test #4: Waving an open hand Test #5: Deflection of hand to back Test #6: Waving a clenched fist

All tests were conducted for 2 minute duration, while signal features were calculated in 20-second intervals. For each test case (Fig.3), results for one the most representative interval are presented. Tests were performed by the paper authors: males, 30-40 years old, physically active with low body fat and without any neurological, muscle disabilities. Examinations were repeated for each tester 15 times with 5 min delays to exclude muscle system fatigue. Similar procedures have been applied for reflex evaluations, to supplement gathered data with, a diagnostic factor describing the effects of long term, multiple, tedious sensor-fused examinations.



19

Quantitative biosignal analisis - Selected signal features

Sensor signal features • Inertial data • Electromyography data Extensions in development • ANN classificator for patient’s state • Recognition of ON-OFF state • Evaluation of % intensity of tremors



20

Selected gestures – PD symptoms correspondences

Accelerometer Axis

X

Y

Z

Accelerometer Axis

X

Y

Z

Accelerometer Axis

X

Y

Z

No. of zero crossings Mean waveform length [s] Mean absolute value [m/s2] Maximum amplitude [m/s2] Median frequency [Hz] Mean frequency [Hz] Peak frequency [Hz]

182 0.10 0.32 1.01 8.85 14.53 0.20

126 0.15 1.30 4.22 6.29 7.79 1.19

189 0.10 0.29 1.02 12.20 14.92 0.41


394 0.04 0.95 4.37 25.00 23.65 0.47

318 0.06 0.25 1.07 24.39 21.50 0.89

326 0.06 0.17 0.87 24.39 21.94 0.33


275 0.07 0.09 0.85 23.26 24.43 2.51

274 0.07 0.07 0.42 23.26 21.19 0.00

225 0.08 0.04 0.22 12.99 22.91 0.00

Electromyograph Electrode

1

2

3

4


1

2

3

4


1

2

3

4


187 0.10 0.01 0.07 10.64 19.91 0.00

178 0.10 0.03 0.18 11.11 17.04 0.53

185 0.10 0.05 0.44 13.16 17.72 0.61

189 0.10 0.02 0.14 15.62 19.09 0.28


151 0.12 0.19 0.45 10.10 27.23 0.60

147 0.12 0.14 0.45 10.42 18.81 0.54

141 0.12 0.06 0.41 10.31 14.31 0.59

142 0.12 0.14 0.45 10.10 17.41 0.53


172 0.11 0.02 0.09 13.33 19.86 1.94

172 0.11 0.08 0.40 12.20 21.23 0.59

179 0.10 0.16 0.45 11.76 26.25 0.44

161 0.12 0.15 0.45 12.66 16.09 0.58

Electrode

5

6

7

8

Electrode

5

6

7

8

Electrode

5

6

7

8


178 0.10 0.01 0.10 12.05 15.13 5.23

167 0.11 0.01 0.04 10.42 13.16 0.00

157 0.12 0.01 0.06 10.42 14.80 0.00

175 0.11 0.01 0.02 10.53 16.09 0.00


138 0.13 0.20 0.45 9.71 16.43 0.36

146 0.12 0.15 0.45 10.64 19.37 0.63

147 0.12 0.14 0.45 11.36 14.90 0.75

155 0.11 0.18 0.45 11.49 20.46 0.62


171 0.11 0.07 0.31 12.05 22.07 0.65

168 0.11 0.02 0.11 11.76 21.20 0.80

162 0.12 0.01 0.08 11.76 18.53 0.00

162 0.12 0.01 0.06 11.49 17.28 0.00

Compared to the previous test case, it is possible to observe the increase in the mean absolute value and the maximum amplitude for the accelerometer. It can also be noticed that hand-shaking was accomplished primarily with the forearm muscle located under the 3rd electrode. For that specific channel, sEMG values are significantly higher.

Based on the signal from the accelerometer it can be observed that during the tests, a slight hand tremor had been registered. A significant change in sEMG signal was recorded. Signal features for all sensor channels show higher values compared to previous test cases. For each sEMG channel, values are similar, which shows that testers were able to activate full set of forearm muscles.


Based on the maximum amplitude, it is possible to observe high intensity tremors. The accelerometer signal correlated with sEMG demonstrates that the bending of the hand to the back engages muscles located under the electrodes 2, 3, 4. Obtained values from these channels vary compared to the other electrodes. Cybernetics Faculty www.wcy.wat.edu.pl

21

Selected gestures – PD symptoms correspondences

Accelerometer Axis

X

Y

Z


145 0.13 0.33 1.36 8.40 11.96 0.40

67 0.28 1.15 3.00 3.57 3.63 1.26

133 0.14 0.47 1.52 6.41 7.68 1.61


1

2

3

4


183 0.10 0.01 0.07 15.15 14.08 0.00

187 0.10 0.02 0.13 15.62 17.08 0.58

177 0.10 0.11 0.44 10.42 19.21 0.79

174 0.11 0.03 0.26 12.50 16.28 0.63

Electrode

5

6

7

8


178 0.10 0.02 0.09 12.50 15.43 0.28

184 0.10 0.01 0.11 10.53 17.54 0.00

164 0.11 0.01 0.06 10.53 15.35 0.00

179 0.10 0.01 0.09 13.51 17.84 0.80


1

2

3

4


150 0.12 0.10 0.45 9.90 25.98 0.44

173 0.11 0.09 0.40 12.50 21.68 0.80

187 0.10 0.03 0.16 12.66 17.71 0.44

167 0.11 0.06 0.31 10.64 22.75 0.30

Accelerometer Axis

X

Y

Z

Electrode

5

6

7

8


131 0.14 1.82 6.77 6.49 9.83 0.42

96 0.19 3.08 8.66 5.00 5.61 0.41

96 0.20 3.19 8.38 5.00 5.37 0.45


175 0.11 0.04 0.19 12.66 28.89 0.40

169 0.11 0.05 0.45 10.75 21.25 0.57

183 0.10 0.05 0.45 12.99 20.29 0.45

153 0.12 0.07 0.41 10.20 14.35 0.53


Based on the signal from the accelerometer, it is clearly visible that the hand movement was carried out primarily in one axis. The curves calculated for the Y axis are much larger than for other axes. This is consistent with expectations as the researcher moved his hand from side to side in a straight line. The analysis of the electromyograph signal shows that the hand waving was carried out mainly by the muscles located under the electrode number 3.

The accelerometer data prove, that the hand was moved mainly around two axes. This was due to the fact that the hand was rotating in the air. The signal from the electromyograph shows that the muscles were stressed during the test, because the values for each channel are close to those obtained during the Test #3.


22

Conclusions and further research Research Domain and main asumptions

Provide a quantitative method for PD neurological symptoms (not only tremor) assessment using mobile biometric tools Conclusions and achievements • Constructed analytical method provides objective assessment of medical treatment effectiveness (based on surveying and sensor data analysis working concurrently). • System is electronic guidance tool with time-constraint support for monitoring and auditing drug intake, surveying and sensor tests • The method evaluates the intensity of neurological disease symptoms. • Electromyography has proven to be a useful technique for tremor and complex gestures and exercises analysis • Analysis of gestures can be used for bradykinesia and rigidness assessment • Prepared mechanisms support also screen interactions for reflex assessment supplementing the area of application for the method Extensions further development: • Gesture segmentation and partition • Gesture pattern representation with respect to its varying duration • Extend already constructed ML methods for arm/hand movement recognition Military University of Technology www.wat.edu.pl


Domain: development of biomedical sensor data analitical tools

23

SUMMARY Lt.Col. Mariusz CHMIELEWSKI, PhD, Eng. Simulation and Decision Support Tools Department Institute of Computer Science Cybernetics Faculty Military University of Technology Warsaw, Poland Research Team of Modelling, Simulation and Computer Based Decision Support in the Conflict and Crisis Situations

[email protected] https://www.researchgate.net/profile/Mariusz_Chmielewski/

Thank You for attention - Questions please Wearable sensor-based data analysis for neurological disease symptoms evaluation utilising quantitative approach. Authors: Lt.Col. Mariusz Chmielewski, Ph.D. Eng. Michał Nowotarski, M.Sc. Eng. Military University of Technology www.wat.edu.pl

Cybernetics Faculty www.wcy.wat.edu.pl 24