Social robots have been deployed in different real application environments (In The Wild (ITW)) to provide services. A social mobile robot that is able to select targets to approach based on activities performed by people in public spaces can better optimize customer experience by providing guidelines, discount promotions, and emergency assistance. To select targets, it is important to understand what candidates are doing to approach persons who may accept the robot’s offer properly. Activity recognition ITW is a challenging task and would need machine learning algorithms to generate models from datasets representative of the situations to be faced. The state-of-the-art datasets and deep learning models for the task of Human Activity Recognition (HAR) from mobile robots do not satisfy the requirements to produce reliable HAR for social robots in service tasks that could operate in public spaces. In this thesis, we address the issue from different perspectives. First, we suggest some characteristics of a reliable ITW dataset should have to support the development of activity recognition from mobile robots and motivate how no state-of-the-art dataset fully satisfies these criteria. Given the lack of datasets to support skeleton-based activity recognition from social robots in service tasks, we collected a large-scale video dataset for activity recognition from the viewpoints of mobile robots ITW: POLIMI-ITW-S. POLIMI-ITW-S dataset is composed of 22,161 RGB video clips (approximately 46 hours) on which we have identified 37 classes of activities performed by more than 50,000 people in real shopping centers. Secondly, we discuss labeling issues and propose a hierarchically detailed labeling approach needed in many applications. We developed a new Human Activity Video Pose Tracking Annotation Tool (HAVPTAT) that could support the semi-automatic labeling of such clips. We used HAVPTAT to label the new POLIMI-ITW-S dataset thoroughly. Based on the common practices, we exploited a novel data pre-processing framework to elaborate the collected raw video dataset to obtain the skeletal data ready-to-use for training and evaluating the popular skeleton-based HAR deep learning algorithms. Then, we evaluated some advanced skeleton-based HAR models on POLIMI-ITW- S dataset. The obtained results are much worse than the results obtained on the datasets collected from constrained environments. We deeply investigate the recognition results and summarize several potential reasons that prevent obtaining good results. Furthermore, we propose a new Shift Semantic Graph Convolutional Neural Network (SS-GCN) method, enhancing Human Activity Recognition (HAR) by understanding human-object interactions semantically. We give some suggestions to improve the performance. Finally, we propose a novel semantic hierarchical model that improves the performance of state-of-the-art skeleton-based activity recognition models on this dataset by about 30%. We depict several examples to visualize the decisions made by robots showing whether to approach people based on the recognition results from the proposed model. We released the complete resources (including the annotated data composed by people tracked bounding boxes, 2-D human body skeleton, and activity classes), HAVPTAT, and the proposed semantic hierarchical model for research use.
I robot sociali sono stati dispiegati in diversi ambienti di applicazione reale (In The Wild (ITW)) per fornire servizi. Un robot mobile sociale in grado di selezionare i bersagli da avvicinare in base alle attività svolte dalle persone negli spazi pubblici può ottimizzare meglio l'esperienza del cliente fornendo linee guida, promozioni scontate e assistenza di emergenza. Per selezionare i bersagli, è importante capire cosa stanno facendo i candidati per avvicinare le persone che potrebbero accettare adeguatamente l'offerta del robot. Il riconoscimento delle attività ITW è un compito impegnativo e richiederebbe algoritmi di machine learning per generare modelli dai dataset rappresentativi delle situazioni da affrontare. I dataset e i modelli di deep learning all'avanguardia per il compito di “Human Activity Recognition (HAR)” da parte dei robot mobili non soddisfano i requisiti per produrre un HAR modello affidabile per i robot sociali nei compiti di servizio che potrebbero operare negli spazi pubblici. In questa tesi, affrontiamo la questione da diverse prospettive. In primo luogo, suggeriamo alcune caratteristiche che un affidabile dataset ITW dovrebbe avere per supportare lo sviluppo del riconoscimento delle attività da parte dei robot mobili e motiviamo come nessun dataset all'avanguardia soddisfi completamente questi criteri. Data la mancanza di dataset per supportare il riconoscimento delle attività basato su scheletri da parte dei robot sociali nei compiti di servizio, abbiamo raccolto un video dataset su larga scala per il riconoscimento delle attività dal punto di vista dei robot mobili ITW: POLIMI-ITW-S. Il dataset POLIMI-ITW-S è composto da 22.161 clip video RGB (circa 46 ore) in cui abbiamo identificato 37 classi di attività svolte da più di 50.000 persone in veri centri commerciali. In secondo luogo, discutiamo i problemi di etichettatura e proponiamo un approccio all'etichettatura dettagliato gerarchicamente necessario in molte applicazioni. Abbiamo sviluppato un nuovo strumento di annotazione “Human Activity Video Pose Tracking Annotation Tool” (HAVPTAT) che potrebbe supportare l'etichettatura semi-automatica di tali clip. Abbiamo usato HAVPTAT per etichettare accuratamente il nuovo dataset POLIMI-ITW-S. Sulla base delle pratiche comuni, abbiamo sfruttato un nuovo framework di pre-elaborazione dei dati per elaborare i video grezzi raccolti per ottenere i dati scheletrici pronti all'uso per l'addestramento e la valutazione dei popolari algoritmi di deep learning per HAR basati su scheletri. Poi, abbiamo valutato alcuni avanzati modelli di HAR basati su scheletri sul dataset POLIMI-ITW-S. I risultati ottenuti sono molto peggiori rispetto ai risultati ottenuti sui dataset raccolti da ambienti vincolati. Abbiamo approfondito i risultati del riconoscimento e riassunto diverse ragioni potenziali che impediscono di ottenere buoni risultati. Inoltre, proponiamo un nuovo metodo di “Shift Semantic Graph Convolutional Neural Network” (SS-GCN), che migliora il riconoscimento dell'attività umana comprendendo semanticamente le interazioni uomo-oggetto. Forniamo alcuni suggerimenti per migliorare le prestazioni. Infine, proponiamo un nuovo modello gerarchico semantico che migliora le prestazioni dei modelli di riconoscimento delle attività basati su scheletri all'avanguardia su questo dataset di circa il 30%. Rappresentiamo diversi esempi per visualizzare le decisioni prese dai robot mostrando se avvicinare le persone in base ai risultati del riconoscimento dal modello proposto. Abbiamo rilasciato le risorse complete (compresi i dati annotati composti da caselle di delimitazione delle persone tracciate, scheletro del corpo umano 2-D e classi di attività), HAVPTAT e il modello gerarchico semantico proposto per l'uso della ricerca.
Semantic human activity recognition from mobile robots
Quan, Hao
2022/2023
Abstract
Social robots have been deployed in different real application environments (In The Wild (ITW)) to provide services. A social mobile robot that is able to select targets to approach based on activities performed by people in public spaces can better optimize customer experience by providing guidelines, discount promotions, and emergency assistance. To select targets, it is important to understand what candidates are doing to approach persons who may accept the robot’s offer properly. Activity recognition ITW is a challenging task and would need machine learning algorithms to generate models from datasets representative of the situations to be faced. The state-of-the-art datasets and deep learning models for the task of Human Activity Recognition (HAR) from mobile robots do not satisfy the requirements to produce reliable HAR for social robots in service tasks that could operate in public spaces. In this thesis, we address the issue from different perspectives. First, we suggest some characteristics of a reliable ITW dataset should have to support the development of activity recognition from mobile robots and motivate how no state-of-the-art dataset fully satisfies these criteria. Given the lack of datasets to support skeleton-based activity recognition from social robots in service tasks, we collected a large-scale video dataset for activity recognition from the viewpoints of mobile robots ITW: POLIMI-ITW-S. POLIMI-ITW-S dataset is composed of 22,161 RGB video clips (approximately 46 hours) on which we have identified 37 classes of activities performed by more than 50,000 people in real shopping centers. Secondly, we discuss labeling issues and propose a hierarchically detailed labeling approach needed in many applications. We developed a new Human Activity Video Pose Tracking Annotation Tool (HAVPTAT) that could support the semi-automatic labeling of such clips. We used HAVPTAT to label the new POLIMI-ITW-S dataset thoroughly. Based on the common practices, we exploited a novel data pre-processing framework to elaborate the collected raw video dataset to obtain the skeletal data ready-to-use for training and evaluating the popular skeleton-based HAR deep learning algorithms. Then, we evaluated some advanced skeleton-based HAR models on POLIMI-ITW- S dataset. The obtained results are much worse than the results obtained on the datasets collected from constrained environments. We deeply investigate the recognition results and summarize several potential reasons that prevent obtaining good results. Furthermore, we propose a new Shift Semantic Graph Convolutional Neural Network (SS-GCN) method, enhancing Human Activity Recognition (HAR) by understanding human-object interactions semantically. We give some suggestions to improve the performance. Finally, we propose a novel semantic hierarchical model that improves the performance of state-of-the-art skeleton-based activity recognition models on this dataset by about 30%. We depict several examples to visualize the decisions made by robots showing whether to approach people based on the recognition results from the proposed model. We released the complete resources (including the annotated data composed by people tracked bounding boxes, 2-D human body skeleton, and activity classes), HAVPTAT, and the proposed semantic hierarchical model for research use.| File | Dimensione | Formato | |
|---|---|---|---|
|
thesis_phd_quan.pdf
non accessibile
Dimensione
15.47 MB
Formato
Adobe PDF
|
15.47 MB | Adobe PDF | Visualizza/Apri |
I documenti in POLITesi sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.
https://hdl.handle.net/10589/207618