Improved healthcare has resulted in increased life expectancy and an aging population, as well as an increase in chronic illnesses, which are now the leading causes of mortality. People with chronic illnesses require long-term care and dedication to take their medications as recommended. Poor medication adherence is a frequent problem in patients with chronic diseases or the elderly, and it can lead to hospitalization and health complications, increased mortality and morbidity, and, as a result, increased healthcare costs. In such a scenario, mobile- and e-health applications could support the patients and their caregivers in managing the problem of medication mal-adherence. Wearable technologies, such as smart wristbands, incorporate inertial units, such as accelerometers and gyroscopes, which may be used in automatic medication adherence monitoring, offering the possibility to classify hand gestures correlated to the action of intaking a medication. Human Activity Recognition (HAR) is the process of recognizing activities by evaluating data collected by sensors integrated with wearable devices. Home behavior analysis, video surveillance, gait analysis, and gesture recognition are examples of successful HAR uses. The aim of this study was to implement a machine learning method to automatically classify eight common hand gestures (drink, eat, pour water, open a bottle, type, answer the phone, comb hair, cut) using the inertial sensors integrated into a commercial smart-wristband, posing specific attention to the drinking gesture, which is correlated to the action of intaking a pill medication. Thirteen subjects took part in the study, providing their written informed consent to perform the protocol approved by the Ethical Committee of Politecnico di Milano. Each subject was asked to wear the MetaMotion R (mbientlab Inc) wrist device, integrating a tri-axial accelerometer and a tri-axial gyroscope. The experimental protocol consisted of two phases: • Phase 1: each subject was asked to perform each gesture twice to create a training set, and then to repeat each gesture twice (test set). • Phase 2: each subject was asked to drink 15 ml of water in one sip, then 30 ml in two sips, then 45 ml in three sips, and finally 60 ml in four sips (test set). Between successive gestures, the hand was maintained still on the table. The timing was defined by the operator. Signals from the inertial sensors embedded in the wristband were acquired (fs = 50 Hz) using a smartphone application developed ad-hoc. The acquired signals were preprocessed in order to remove artifacts. Also, the hand idle portions were removed. Afterward, signals were segmented using the sliding window approach (two different window sizes: 2- and 3- second), and morphological, time, and frequency domain features were collected from each segment to create the input to the tested machine learning models: K-nearest-neighbor (KNN), Support Vector Machine (SVM), and Random Forest (RF). Specifically, two classification problems were considered: multi-class classification of the eight activities, and binary classification of the 'drinking' gesture versus 'non-drinking'. In the multi-class classification problem, the three tested classifiers allowed to obtain comparable results, with the 3-second windowing approach outperforming the 2-second windowing. In this case, KNN provided 76% f1-score and 85.5% balanced accuracy, SVM provided 76% f1-score and 73% balanced accuracy and RF provided 75% f1-score and 82.4% balanced accuracy. In the binary classification, comparable results were obtained with 2-second and 3-second windowing, reaching high average values of precision (92%), recall (91%), f1-score (92%), specificity (90%), and balanced accuracy (90.9%). Specifically, the drinking gesture was classified with acceptable precision (75%) and f1-score (80%), high values of recall (91%), specificity (93%), and balanced accuracy (90.9%). In conclusion, experimental results showed that hand gesture classification with machine learning from wrist accelerometers and gyroscopes signals can be performed with reasonable accuracy in laboratory settings, paving the way for a new generation of medical devices for monitoring medication adherence.
I recenti progressi nella medicina hanno causato un aumento della aspettativa di vita e un invecchiamento della popolazione, accompagnato da un aumento delle malattie croniche, che rappresentano oggi la maggiore causa di morte. I pazienti cronici necessitano di cure a lungo termine, oltre ad un costante impegno nel rispettare le cure prescritte. Tra i pazienti cronici, e in particolare nel caso di pazienti anziani, è diffuso il problema della bassa aderenza alla terapia farmacologica. Questo può compromettere l’effettiva riuscita del trattamento, aumentando il rischio di ospedalizzazione per complicazioni e gravando sulla spesa del settore sanitario. In questo scenario, applicazioni di mobile- e e-health possono supportare i pazienti e i loro caregiver nella gestione di tale problema. I dispositivi indossabili sono attualmente considerati tra i metodi più accurati ed efficaci, grazie alle piccole dimensioni, al peso leggero, al basso costo e alla facilità di utilizzo. Il paziente può infatti indossare un dispositivo come uno smartwatch per monitorare le attività quotidiane, tra cui l’aderenza alle prescrizioni. L’obiettivo di questo studio è quello di sviluppare un metodo di machine learning per la classificazione automatica di 8 gesti comuni (bere, mangiare, versare l’acqua, aprire una bottiglia, digitare alla tastiera, rispondere al telefono, pettinare i capelli e tagliare un foglio) utilizzando i segnali acquisiti tramite i sensori inerziali integrati in una smart-wristband, con particolare attenzione verso il riconoscimento del gesto “bere”, correlato all’assunzione di terapie in forma di pillole. 13 soggetti hanno preso parte allo studio, fornendo il proprio consenso informato per il protocollo approvato dal Comitato Etico del Politecnico di Milano. I segnali di movimento sono stati acquisiti tramite il dispositivo MetaMotion R (mbientlab Inc), integrante un accelerometro e un giroscopio triassiali. Il protocollo di acquisizioni comprendeva due fasi: • Fase 1: ogni soggetto ha effettuato ciascun gesto 2 volte (training set), e poi ha ripetuto ciascun gesto almeno due volte in ordine casuale per un totale di 20 gesti (test set). • Fase 2: a ciascun soggetto è stato richiesto di bere 15 ml di acqua in un sorso, 30 ml di acqua in due sorsi, 45 ml di acqua in tre sorsi, 60 ml di acqua in quattro sorsi. Tra gesti successivi, è stato richiesto al soggetto di mantenere la mano ferma sul tavolo per alcuni secondi. I segnali dei sensori inerziali integrati nella band sono stati acquisiti (fs = 50 Hz) tramite una applicazione per smartphone sviluppata appositamente.tali segnali sono stati preprocessati al fine di rimuovere eventuali artefatti, e le porzioni in cui la mano era mantenuta ferma sono state eliminate. I segnali sono quindi stati segmentati tramite la tecnica di sliding window. In particolare, sono state testate finestre di due durate differenti: 2 secondi e 3 secondi. Successivamente sono state calcolate features morfologiche, nel dominio del tempo e delle frequenze, che sono state utilizzate come input per tre metodi di machine learning: K-nearest-neighbor (KNN), Support Vector Machine (SVM), and Random Forest (RF). Sono stati considerati due problemi di classificazione: multi-classe (classificazione degli 8 gesti) e binario (bere vs altro). Nella classificazione multi-classe, i tre classificatori testati hanno permesso di ottenere risultati comparabili. L’utilizzo della finestra di segmentazione di 3 secondi ha fornito risultati migliori rispetto alla finestra di 2 secondi. In particolare sono stati ottenuti valori di f1-score pari a 76% con KNN ed SVM e pari a 75% con RF, e valori di balanced accuracy pari a 85.5%, 73% e 82.4% rispettivamente con KNN, SVM e RF. Anche il problema di classificazione binaria ha permesso di ottenere buoni risultati di classificazione (valori medi: precisione 92%, recall 91%, f1-score 92%, specificità 90% e balanced accuracy 90.9%). In particolare, relativamente al gesto del bere, sono stati ottenuti i seguenti risultati: precisione 75%, f1-score 80%, recall 91%, specificità 93% e balanced accuracy 90.9%. In conclusione, i risultati sperimentali hanno dimostrato che la classificazione dei gesti della mano tramite metodi di machine learning applicati ai segnali acquisiti tramite sensori inerziali integrati in un dispositivo da polso può essere effettuata con ragionevole accuratezza in un contesto di laboratorio, aprendo la strada a una nuova generazione di dispositivi medici per il monitoraggio della aderenza alla terapia farmacologica.
Classification of daily gestures using wrist inertial sensors and machine learning
Erfanisayyar, Maryam
2020/2021
Abstract
Improved healthcare has resulted in increased life expectancy and an aging population, as well as an increase in chronic illnesses, which are now the leading causes of mortality. People with chronic illnesses require long-term care and dedication to take their medications as recommended. Poor medication adherence is a frequent problem in patients with chronic diseases or the elderly, and it can lead to hospitalization and health complications, increased mortality and morbidity, and, as a result, increased healthcare costs. In such a scenario, mobile- and e-health applications could support the patients and their caregivers in managing the problem of medication mal-adherence. Wearable technologies, such as smart wristbands, incorporate inertial units, such as accelerometers and gyroscopes, which may be used in automatic medication adherence monitoring, offering the possibility to classify hand gestures correlated to the action of intaking a medication. Human Activity Recognition (HAR) is the process of recognizing activities by evaluating data collected by sensors integrated with wearable devices. Home behavior analysis, video surveillance, gait analysis, and gesture recognition are examples of successful HAR uses. The aim of this study was to implement a machine learning method to automatically classify eight common hand gestures (drink, eat, pour water, open a bottle, type, answer the phone, comb hair, cut) using the inertial sensors integrated into a commercial smart-wristband, posing specific attention to the drinking gesture, which is correlated to the action of intaking a pill medication. Thirteen subjects took part in the study, providing their written informed consent to perform the protocol approved by the Ethical Committee of Politecnico di Milano. Each subject was asked to wear the MetaMotion R (mbientlab Inc) wrist device, integrating a tri-axial accelerometer and a tri-axial gyroscope. The experimental protocol consisted of two phases: • Phase 1: each subject was asked to perform each gesture twice to create a training set, and then to repeat each gesture twice (test set). • Phase 2: each subject was asked to drink 15 ml of water in one sip, then 30 ml in two sips, then 45 ml in three sips, and finally 60 ml in four sips (test set). Between successive gestures, the hand was maintained still on the table. The timing was defined by the operator. Signals from the inertial sensors embedded in the wristband were acquired (fs = 50 Hz) using a smartphone application developed ad-hoc. The acquired signals were preprocessed in order to remove artifacts. Also, the hand idle portions were removed. Afterward, signals were segmented using the sliding window approach (two different window sizes: 2- and 3- second), and morphological, time, and frequency domain features were collected from each segment to create the input to the tested machine learning models: K-nearest-neighbor (KNN), Support Vector Machine (SVM), and Random Forest (RF). Specifically, two classification problems were considered: multi-class classification of the eight activities, and binary classification of the 'drinking' gesture versus 'non-drinking'. In the multi-class classification problem, the three tested classifiers allowed to obtain comparable results, with the 3-second windowing approach outperforming the 2-second windowing. In this case, KNN provided 76% f1-score and 85.5% balanced accuracy, SVM provided 76% f1-score and 73% balanced accuracy and RF provided 75% f1-score and 82.4% balanced accuracy. In the binary classification, comparable results were obtained with 2-second and 3-second windowing, reaching high average values of precision (92%), recall (91%), f1-score (92%), specificity (90%), and balanced accuracy (90.9%). Specifically, the drinking gesture was classified with acceptable precision (75%) and f1-score (80%), high values of recall (91%), specificity (93%), and balanced accuracy (90.9%). In conclusion, experimental results showed that hand gesture classification with machine learning from wrist accelerometers and gyroscopes signals can be performed with reasonable accuracy in laboratory settings, paving the way for a new generation of medical devices for monitoring medication adherence.| File | Dimensione | Formato | |
|---|---|---|---|
|
Classification of daily gestures using wrist inertial sensors and machine learning.pdf
accessibile in internet per tutti
Dimensione
3.03 MB
Formato
Adobe PDF
|
3.03 MB | Adobe PDF | Visualizza/Apri |
I documenti in POLITesi sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.
https://hdl.handle.net/10589/186913