Handwriting is an extremely complex task engaging several cognitive and motor components. Learning to write is fundamental; therefore eventual difficulties in the process must be promptly identified. On the other hand, handwriting is a promising biomarker for neurological disorders. Writing capacities can be assessed through pen-and-paper tests or through quantitative approaches, enabled by digitizing tablets. While providing quantitative information, these devices lack naturalness during the execution and can represent a technological barrier, particularly for older adults. In response to this problem, a Sensorized Ink Pen (SIP) was developed by NearLab at Politecnico di Milano to analyze handwriting in an ecological way, allowing subjects to write on paper. However, despite recording kinematic data with the embedded Inertial Measurement Unit (IMU), the SIP lacked direct recording of trace-related signals. This work presents a novel trace reconstruction algorithm, which employs IMU data fusion to reconstruct the SIP's orientation in space and discrete integration to extract the IMU translation. This information was used to estimate the position of the SIP's tip in time, which was projected on a two-dimensional plane using Principal Component Analysis, yielding the written trace. The algorithm was tested on data related to the writing of the word "uno" in cursive, performed by 5th grade elementary students. The quality of the reconstructions was validated through an Optical Character Recognition (OCR) algorithm (Google Vision), which was fed the reconstructed words along with the scans of the corresponding real written product. The Character Recognition Rate (CRR) and Character Error Rate (CER) were selected to assess the OCR performance. The results showed an acceptable readability, despite a large gap with the performances on scans. The mean CRR was 59,22 \% on reconstructed words and 77,54\% on paper scans. The mean CER were 46,02\% and 25,20\%, respectively. A non-negligible number of words was recognized in the opposite writing direction and some words were recognized as superimposed to others by the OCR, which worsened performances. An error characterization analysis was conducted to understand which features differentiate opposite reconstructed words from the rest. It was found that in oppositely reconstructed words the maximal velocity was found in the last portion of the word, likely causing distortions which were misinterpreted by the OCR algorithm. Possible improvements were identified. The traslational component of the overall displacement could be better estimated, adopting strategies to reduce accumulation errors. Force signal thresholding -- the basis for words segmentation -- and kinemtic signals filtering could be adapted to the subject's writing style. Lastly, the proposed approach should be expanded to all writing styles.
La scrittura a mano è un compito estremamente complesso che coinvolge molteplici componenti cognitive e motorie. L'apprendimento della scrittura è fondamentale; pertanto, eventuali difficoltà nel processo devono essere prontamente identificate. D'altra parte, la scrittura a mano rappresenta un promettente biomarcatore per i disturbi neurologici. Le capacità di scrittura possono essere valutate tramite test carta-e-penna o mediante approcci quantitativi, resi possibili da tavolette digitalizzate. Sebbene questi dispositivi forniscano informazioni quantitative, essi mancano di naturalezza durante l'esecuzione e possono rappresentare una barriera tecnologica, in particolare per gli anziani. In risposta a questo problema, il NearLab del Politecnico di Milano ha sviluppato una Penna ad Inchiostro Sensorizzata (SIP) per analizzare la scrittura a mano in modo ecologico, consentendo ai soggetti di scrivere su carta. Tuttavia, nonostante la registrazione dei dati cinematici tramite l'Unità di Misura Inerziale (IMU) integrata, la SIP non disponeva di una registrazione diretta dei segnali relativi alla traccia. Questo lavoro presenta un nuovo algoritmo di ricostruzione della traccia, che impiega la fusione dei dati dell’IMU per ricostruire l’orientamento della SIP nello spazio e un'integrazione discreta per estrarre la traslazione dell'IMU. Queste informazioni sono state utilizzate per stimare la posizione della punta della SIP nel tempo, proiettata poi su un piano bidimensionale tramite l'Analisi delle Componenti Principali, ottenendo la traccia scritta. L'algoritmo è stato testato su dati relativi alla scrittura della parola "uno" in corsivo, eseguita da studenti di quinta elementare. La qualità delle ricostruzioni è stata validata attraverso un algoritmo di Riconoscimento Ottico dei Caratteri (OCR) (Google Vision), al quale sono state fornite le parole ricostruite insieme alle scansioni del corrispondente prodotto scritto su carta. Il Tasso di Riconoscimento dei Caratteri (CRR) e il Tasso di Errore dei Caratteri (CER) sono stati selezionati per valutare le prestazioni dell'OCR. I risultati hanno mostrato una leggibilità accettabile, sebbene con un divario significativo rispetto alle prestazioni sulle scansioni. Il CRR medio è stato del 59,22\% sulle parole ricostruite e del 77,54\% sulle scansioni cartacee. Il CER medio è stato rispettivamente del 46,02\% e del 25,20\%. Un numero non trascurabile di parole è stato riconosciuto nella direzione opposta e alcune parole sono state identificate come sovrapposte ad altre dall’OCR, peggiorando le prestazioni. È stata condotta un'analisi di caratterizzazione dell'errore per comprendere quali caratteristiche differenziano le parole ricostruite in direzione opposta dalle altre. È emerso che, nelle parole ricostruite in direzione opposta, la velocità massima si trovava nell'ultima porzione della parola, probabilmente causando distorsioni interpretate erroneamente dall’algoritmo OCR. Sono stati identificati possibili miglioramenti. La componente traslazionale dello spostamento complessivo potrebbe essere meglio stimata adottando strategie per ridurre gli errori di accumulo. La soglia del segnale di forza, base per la segmentazione delle parole, e il filtraggio dei segnali cinematici potrebbero essere adattati allo stile di scrittura del soggetto. Infine, l'approccio proposto dovrebbe essere esteso a tutti gli stili di scrittura.
Development and validation of a trace reconstruction algorithm for a sensorized ink pen
Gentile, Chiara
2023/2024
Abstract
Handwriting is an extremely complex task engaging several cognitive and motor components. Learning to write is fundamental; therefore eventual difficulties in the process must be promptly identified. On the other hand, handwriting is a promising biomarker for neurological disorders. Writing capacities can be assessed through pen-and-paper tests or through quantitative approaches, enabled by digitizing tablets. While providing quantitative information, these devices lack naturalness during the execution and can represent a technological barrier, particularly for older adults. In response to this problem, a Sensorized Ink Pen (SIP) was developed by NearLab at Politecnico di Milano to analyze handwriting in an ecological way, allowing subjects to write on paper. However, despite recording kinematic data with the embedded Inertial Measurement Unit (IMU), the SIP lacked direct recording of trace-related signals. This work presents a novel trace reconstruction algorithm, which employs IMU data fusion to reconstruct the SIP's orientation in space and discrete integration to extract the IMU translation. This information was used to estimate the position of the SIP's tip in time, which was projected on a two-dimensional plane using Principal Component Analysis, yielding the written trace. The algorithm was tested on data related to the writing of the word "uno" in cursive, performed by 5th grade elementary students. The quality of the reconstructions was validated through an Optical Character Recognition (OCR) algorithm (Google Vision), which was fed the reconstructed words along with the scans of the corresponding real written product. The Character Recognition Rate (CRR) and Character Error Rate (CER) were selected to assess the OCR performance. The results showed an acceptable readability, despite a large gap with the performances on scans. The mean CRR was 59,22 \% on reconstructed words and 77,54\% on paper scans. The mean CER were 46,02\% and 25,20\%, respectively. A non-negligible number of words was recognized in the opposite writing direction and some words were recognized as superimposed to others by the OCR, which worsened performances. An error characterization analysis was conducted to understand which features differentiate opposite reconstructed words from the rest. It was found that in oppositely reconstructed words the maximal velocity was found in the last portion of the word, likely causing distortions which were misinterpreted by the OCR algorithm. Possible improvements were identified. The traslational component of the overall displacement could be better estimated, adopting strategies to reduce accumulation errors. Force signal thresholding -- the basis for words segmentation -- and kinemtic signals filtering could be adapted to the subject's writing style. Lastly, the proposed approach should be expanded to all writing styles.| File | Dimensione | Formato | |
|---|---|---|---|
|
2025_03_Gentile_Tesi_01.pdf
non accessibile
Dimensione
8.94 MB
Formato
Adobe PDF
|
8.94 MB | Adobe PDF | Visualizza/Apri |
|
2025_03_Gentile_Executive_Summary_02.pdf
non accessibile
Dimensione
2.07 MB
Formato
Adobe PDF
|
2.07 MB | Adobe PDF | Visualizza/Apri |
I documenti in POLITesi sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.
https://hdl.handle.net/10589/235158