Artificial Intelligence, a branch of Computer Science, is becoming more and more popular in the everyday life, thanks to the possibility of sharing a large amount of data and managing them. Artificial Intelligence allows the implementation of systems that can execute tasks associated to human intelligence, simulating human skills and reasoning. Recently, the increase in available data and computing power has promoted the use of deep learning techniques. Therefore, a huge amount of AI applications exist, as it can be seen in literature. In this project, the work is focused in one of the most particular ability of the human brain, the Facial Expression Recognition (FER). Face Recognition can be often found in commercial tools, applied on devices like smartphones, tablets and PC. Although already largely used in the technology sector, people emotion analysis is becoming more widespread, especially in healthcare, in marketing and business, in security, software engineering and even in robotics. For this reason, what we want to do within our project is precisely to combine the potential offered by AI and those of robotics. We want to develop an autonomous system capable of improving the empathetic qualities of a humanoid robot: Pepper. Pepper is, in fact, a robot with semi-human features and our goal is to be able to manage its communicative aspect thanks to the study of the interlocutor’s emotions. Facial expression recognition from images is particularly important because it does not require advanced equipment and allows to easily acquire data with a camera without inconveniencing the subjects. A fact that is not usually addressed, although particularly important, is that emotions cannot be divided into strict classes. Indeed, when taking about facial emotions, the same face could be recognized as expressing different emotions if seen by different people. Emotions are such a subjective aspect that the exact same face can induce different answers if seen by different people or even if it is seen by the same person in two distinct moments. With an Artificial neural network, specifically edited and efficiently trained, it could be possible to analyze and extract emotional data from an image. Some of the issues in doing this include finding the best parameters and weights to assign to the network, since they are a lot and often hardly solvable. The most arduous challenge in developing an efficient Neural Network is indeed to find the most efficient dataset suitable to properly train the Network avoiding bad fitting of the model to guarantee good predictions. Although this proves to be a really challenging project, we tried to exploit the best deep learning techniques to develop a system for Facial Emotion Recognition that could run in real time and could be integrated in an humanoid robotic technology. Even if some already created datasets were available, we decided to create the entire dataset from scratch, taking advantage of the fact that it would have been possible to choose how many and which emotions analyze. In this way is also possible to associate the best label to each photo, sorting the entire dataset to re-edit them in the most suitable manner to best feed the net. The choice is to process 7 emotion in total: the 6 primary emotion (joy, sadness, fear, anger, surprise and disgust) with the addition of a neutral state. Once the emotions were selected and the dataset completed, this was elaborated in order to get four different compositions starting from the original one. So to have the chance to compare the different results obtained by each dataset and finally decide which one could better perform and reach the final goal. A similar approach was adopted with Neural networks: four different configurations were tested. Combining all the different CNN structures with each of the possible datasets, we found the best model to be used for finalizing the project. While creating the neural network we took into consideration the fact that the model, once trained, should have been processed by an embedded device. The management of the emotion analysis is done trough a device called Google Coral Dev Board, a TPU-based board created by Google that takes advantage of the TPU technology. This TPU is used for elaborating the CNN in substitution of CPU and thus to speed up the evaluation of the emotion of the subject from a flux of frames coming from the camera of robot. To exploit this new technology it was necessary to convert the Tensorflow Network in its Lite version, using a specific interpreter installed on the device to run the whole process. Utilizing this additional device is a fundamental step since the hardware component of the robot is not designed to do this kind of operations and the transfer of all the computational operations is necessary. Meanwhile the main part of the code is developed, using the potentiality of some specific libraries of Python language. In order to obtain images suitable to be processed by the neural network, the development of a video processing program was necessary. This program, installed on the device, allows to capture frames coming from the camera, detect and encode the people faces in the video and analyse each person gender and age. This initial process is essential to extract all the characteristics (ID, age and gender) of the people the robot is looking at and which are interacting with it. This is also necessary to prepare all the frames for feeding the Network and obtain emotional data. The device is then connected to a server, specifically created, which has the role to manage all the data coming from the Coral Dev. This server is then connected on the one hand with a database where all the data are stored, while on the other is linked to the robot where its aim is to extract the necessary data when Pepper needs them to manage the conversation. The development of this work is quite complex and consists in more phases. Just to start, several attempts to train different neural networks have been made. Different configurations to be analyzed have been created and various datasets have been used for the training phase. What was found out is the best dataset and the best Neural Network’s configuration that allow the most precise emotions prediction possible. What was obtained is an acceptable prediction net, about 55% of accuracy, but not an optimal choice to manage with a real time emotion recognition. However, the fact that the dataset used for the training is only partially completed, due to the decision of creating a new one from nothing, we suppose this result can only increase completing the dataset. After that, the preprocessing phase and people recognition processes are developed in order to feed the network and collect all the people’s data we’re interested in. Then this part, together with the suitably converted model, is installed on the Google coral Dev Board to execute the whole path to extract the data. Finally, the system is linked to a server module which allows the data transfer to a database (MongoDB) and at the same time it allows to pick the stored data, according with Pepper’s requests. Pepper is implemented by the Behaviour Motor System in order to request the data regarding its interlocutor, modifying its behaviours and speeches depending on the type of data it receives. Running the code we noticed that the performances obtain on the board are not totally satisfactory. This configuration proves to be excellent for data management and analysis but not sufficiently fast to manage a real time conversation. To solve this issue some adjustment have been made in the robot BMS. At the end, what we have obtained is an integrated ”device-to-robot system” able to profile and recognize emotion just focusing on a person. This system is, certainly, not perfect in terms of performances and speed, but it could be a good compromise as regards people data analysis and outlining.
L’intelligenza artificiale, un ramo della scienza informatica, sta diventando sempre pi´u popolare nella vita di tuti i giorni, grazie alla possibilit´a di condividere e gestire facilmente una grande quantit´a di dati. Questa tecnologia permette l’implementazione di sistemi in grado di eseguire compiti legati all’intelligenza umana, simulando ragionamenti e abilit´a umane. Di recente, il forte incremento nella disponibilit´a dei dati e nello sviluppo di sistemi con una pi´u ampia potenza di calcolo ha favorito l’utilizzo di tecniche di deep learning. Di conseguenza, come si vede in letteratura, una gran quantit´a di applicazioni basate sull’AI ´e stata sviluppata. In questo progetto il lavoro ´e focalizzato su una delle pi´u particolari abilit´a del cervello umano, il riconoscimento dell’espressioni facciali. Il riconoscimneto facciale ´e una funzione che si pu´o spesso ritrovare in molti strumenti SW ormai da anni in commercio, come per esempio su dispositivi come smartphone, tablet e PC. Seppur gi´a largamente sfruttata nell’ambito della tecnologia, l’analisi delle emozioni umane sta diventando sempre pi´u diffuso, specilamente nel settore sanitario, nel settore commerciale, in ambiti di sicurezza, ingegneristico e anche nella robotica. Per questa ragione, l’obbiettivo del nostro progetto ´e proprio quello di combinare il potenziale offerto dall’AI con quello offerto dalle nuove tecnologie robotiche, sviluppando un sistema autonomo in grado di migliorare le doti empatiche di un robot umanoide denominato Pepper. Pepper ´e un robot con caratteristiche semi-umane e lo scopo principale ´e quello di gestire le sue abilit´a comunicative grazie allo studio delle emozioni del suo interlocutore. Il riconoscimento delle emozioni facciali avviene tramite l’elaborazione dalle immagini, il che ´e un dato particolarmente importante poich´e non richiede attrezzature particolarmente complesse e permette di acquisire i dati molto facilmente; ´e sufficiente infatti il solo utilizzo di una telecamera senza creare inconvenienti al soggetto. Un fatto che viene spesso trascurato, sebbene di particolare importanza, ´e che le emozioni non possono essere divise in categorie ben definite, ma quando parliamo di emozioni facciali il volto pu´o indicare emozioni diverse se visto da persone differenti. Le emozioni sono una caratteristica talmente soggettiva che la stessa identica faccia pu´o indurre a risposte diverse sia se vista da persone diverse, sia se vista dalla stessa persona in momenti diversi. Grazie all’utilizzo delle reti neurali specificamente costruite ed efficacemente addestrate ´e possibile analizzare le immagine ed estrarne dati relatvi allo stato emozionale dei soggetti inquadrati. La problematica pi´u grande nello sviluppare un’efficiente rete neurale di questo tipo ´e quella di trovare il dataset pi´u adatto per addestrare in maniera corretta il modello, evitando errori di adattamento per garantire buone predizioni. Nonostante questo sembri essere un progetto molto arduo, si ´e deciso di sfruttare le migliori tecniche di deep learning per sviluppare un sistema per il riconoscimento delle emozioni facciali che potesse essere eseguito in real-time e che potesse essere integrato con una tecnologia robotica umanoide. Anche se alcuni dataset gi´a predisposti sono disponibili online, la scelta ´e ricaduta nel creare un nostro dataset partendo da zero, sfruttando il fatto di poterlo creare a seconda delle nostre esigenze ed associando ad ogni immagine il label corretto. La nostra scelta ´e stata quella di considerare 7 emozioni complessivamente: le 6 principali (felicit´a, tristezza, paura, rabbia, sorpresa and disgusto) con l’aggiunta dell’espressione neutrale. Una volta selezionate le emozione adatte e completato il dataset, quest’ultimo ´e stato modificato per ottenere 4 diverse composizioni di dataset partendo da quello originale; in questo modo abbiamo avuto la possibilit´a di comparare i risultati ottenuti e di scegliere quale garantisse un maggior successo nel raggiungere l’obiettivo finale. L’approccio utilizzato ´e simile anche per la realizzazione delle reti neurali: 4 differenti strutture sono state create ed ognuna di queste ´e stata addestrata con ciascuno dei dataset sopra descritti. Testando tutte le diverse strutture di rete neurale con ogni configurazione di dataset abbiamo trovato la miglior combinazione che ci permettesse di completare al meglio il progetto. Durante la creazione della rete abbiamo dovuto tenere in considerazione il fatto che il modello, una volta completato l’addestramento, dovesse essere processato da un dispositivo embedded. La gestione dell’analisi emozionale, infatti, avviene grazie all’utilizzo di un dispositivo chiamato Google Coral Dev Board, una scheda basata creata da Google che sfrutta l’utilizzo di una nuova tecnologia chiamata tensor processing unit (TPU). La TPU ´e utilizzata per processare la rete neurale e quindi velocizzare la valutazione delle emozioni del soggetto grazie all’analisi di un flusso di immagini provenienti dalla videocamera del robot. Per poter sfruttare a pieno questa nuova tecnologia ´e necessario convertire la rete, strutturata grazie all’utilizzo di Tensorflow, in una versione pi´u rapida e leggera utilizzando TF lite. L’utilizzo di questo dispositivo esterno ´e assolutamete fondamentale dato che la componenti hardware incluse all’interno del robot non sono adatte ad eseguire questo tipo di operazioni ed ´e necessario il trasferimento di tutte le operazioni computazionali. Nel frattempo abbiamo sviluppato la parte principale del codice, sfruttando le potenzialit´a di alcune librerie specifiche del linguaggio Python e per ottenere immagini adatte ad essere processate dalla rete neurale ´e stato necessario lo sviluppo di un programma di elaborazione video. Questo programma, installato sul dispositivo, consente di catturare fotogrammi provenienti dalla videocamera, rilevare e codificare i volti delle persone nel video e analizzare il genere e l’et´a di ogni persona. Questo processo iniziale ´e essenziale per estrarre tutte le caratteristiche (ID, et´a e genere) delle persone che il robot sta osservando, con cui interagisce e anche per preparare tutti i frame utili per alimentare la rete e ottenere dati relativi alle emozioni del soggetto. Questo dispositivo ´e infine collegato ad un server appositamente creato che ha il compito di gestire tutti i dati provenienti dalla Coral Dev Board. Il server ´e poi connesso da un lato con un database dove tutti i dati vengono salvati, mentre dall’altro ´e in comunicazione con il robot, al quale deve fornire i dati necessari quando richiesti. Lo sviluppo del lavoro ´e stato suddiviso suddiviso in varie fasi. Per iniziare, svariati tentativi per addestrare le differenti reti neurali sono stati eseguiti utilizzando le diverse composizioni di dataset. Abbiamo quindi trovato che il migliore dataset e la miglior configurazione di rete neurale in grado di fornire la predizione emozionale pi´u accurata possibile mostrasse un valore di accuratezza molto vicino al 55%: risultato accettabile, seppure non ottimale per la gestione in tempo reale delle reazioni del robot. Con molta probabilit´a questo risultato ´e legato al fatto di avere addestrato il modello utilizzando un dataset solo parzialmente completo, mentre utilizzandone uno pi´u ampio i valori di accuratezza non potrebbero che crescere. Successivamente, la fase di preelbaorazione delle immagini e i processi di riconoscimento delle persone sono stati sviluppati per estrarre tutti i dati d’interesse e una volta che sia questa parte di elaborazione, sia il modello adeguatamente convertito sono stati installati sul dispositivo, l’intero sistema era pronto per analizzare il video e fornire tutto ci´o di cui Pepper avesse bisogno. Alla fine il sistema ´e collegato al server che permette di trasferire i dati al database (MongoDB) e al tempo stesso di estarre il parametro di cui Pepper ha bisogno. Il robot ´e implementato tramite l’utilizzo del ”Behaviour Motor System” che gli permette di richiedere al server i dati di cui ha bisogno per gestire il suo comportamento in base allo stato emozionale momentaneo del soggetto. Eseguendo l’intero programma abbiamo notato che le prestazioni ottenute sulla scheda non sono del tutto soddisfacenti, infatti questa configurazione si rivela eccellente per la gestione e l’analisi dei dati ma non sufficientemente veloce per gestire una conversazione in tempo reale. Per risolvere questo problema sono state apportate alcune modifiche al robot BMS. In conclusione ci´o che abbiamo ottenuto ´e un sistema integrato ”robot/dispositivo” in grado di profilare e riconoscere le emozioni delle persone. Questo non ´e certamente perfetto in termini di prestazioni e velocit´a, ma potrebbe essere un buon compromesso per quanto riguarda l’analisi e la definizione dei dati.
Deep convolutional networks for real-time analysis of human emotions to extend robotic empathy
CORTI, MATTEO;CERRONE, NICCOLÒ
2018/2019
Abstract
Artificial Intelligence, a branch of Computer Science, is becoming more and more popular in the everyday life, thanks to the possibility of sharing a large amount of data and managing them. Artificial Intelligence allows the implementation of systems that can execute tasks associated to human intelligence, simulating human skills and reasoning. Recently, the increase in available data and computing power has promoted the use of deep learning techniques. Therefore, a huge amount of AI applications exist, as it can be seen in literature. In this project, the work is focused in one of the most particular ability of the human brain, the Facial Expression Recognition (FER). Face Recognition can be often found in commercial tools, applied on devices like smartphones, tablets and PC. Although already largely used in the technology sector, people emotion analysis is becoming more widespread, especially in healthcare, in marketing and business, in security, software engineering and even in robotics. For this reason, what we want to do within our project is precisely to combine the potential offered by AI and those of robotics. We want to develop an autonomous system capable of improving the empathetic qualities of a humanoid robot: Pepper. Pepper is, in fact, a robot with semi-human features and our goal is to be able to manage its communicative aspect thanks to the study of the interlocutor’s emotions. Facial expression recognition from images is particularly important because it does not require advanced equipment and allows to easily acquire data with a camera without inconveniencing the subjects. A fact that is not usually addressed, although particularly important, is that emotions cannot be divided into strict classes. Indeed, when taking about facial emotions, the same face could be recognized as expressing different emotions if seen by different people. Emotions are such a subjective aspect that the exact same face can induce different answers if seen by different people or even if it is seen by the same person in two distinct moments. With an Artificial neural network, specifically edited and efficiently trained, it could be possible to analyze and extract emotional data from an image. Some of the issues in doing this include finding the best parameters and weights to assign to the network, since they are a lot and often hardly solvable. The most arduous challenge in developing an efficient Neural Network is indeed to find the most efficient dataset suitable to properly train the Network avoiding bad fitting of the model to guarantee good predictions. Although this proves to be a really challenging project, we tried to exploit the best deep learning techniques to develop a system for Facial Emotion Recognition that could run in real time and could be integrated in an humanoid robotic technology. Even if some already created datasets were available, we decided to create the entire dataset from scratch, taking advantage of the fact that it would have been possible to choose how many and which emotions analyze. In this way is also possible to associate the best label to each photo, sorting the entire dataset to re-edit them in the most suitable manner to best feed the net. The choice is to process 7 emotion in total: the 6 primary emotion (joy, sadness, fear, anger, surprise and disgust) with the addition of a neutral state. Once the emotions were selected and the dataset completed, this was elaborated in order to get four different compositions starting from the original one. So to have the chance to compare the different results obtained by each dataset and finally decide which one could better perform and reach the final goal. A similar approach was adopted with Neural networks: four different configurations were tested. Combining all the different CNN structures with each of the possible datasets, we found the best model to be used for finalizing the project. While creating the neural network we took into consideration the fact that the model, once trained, should have been processed by an embedded device. The management of the emotion analysis is done trough a device called Google Coral Dev Board, a TPU-based board created by Google that takes advantage of the TPU technology. This TPU is used for elaborating the CNN in substitution of CPU and thus to speed up the evaluation of the emotion of the subject from a flux of frames coming from the camera of robot. To exploit this new technology it was necessary to convert the Tensorflow Network in its Lite version, using a specific interpreter installed on the device to run the whole process. Utilizing this additional device is a fundamental step since the hardware component of the robot is not designed to do this kind of operations and the transfer of all the computational operations is necessary. Meanwhile the main part of the code is developed, using the potentiality of some specific libraries of Python language. In order to obtain images suitable to be processed by the neural network, the development of a video processing program was necessary. This program, installed on the device, allows to capture frames coming from the camera, detect and encode the people faces in the video and analyse each person gender and age. This initial process is essential to extract all the characteristics (ID, age and gender) of the people the robot is looking at and which are interacting with it. This is also necessary to prepare all the frames for feeding the Network and obtain emotional data. The device is then connected to a server, specifically created, which has the role to manage all the data coming from the Coral Dev. This server is then connected on the one hand with a database where all the data are stored, while on the other is linked to the robot where its aim is to extract the necessary data when Pepper needs them to manage the conversation. The development of this work is quite complex and consists in more phases. Just to start, several attempts to train different neural networks have been made. Different configurations to be analyzed have been created and various datasets have been used for the training phase. What was found out is the best dataset and the best Neural Network’s configuration that allow the most precise emotions prediction possible. What was obtained is an acceptable prediction net, about 55% of accuracy, but not an optimal choice to manage with a real time emotion recognition. However, the fact that the dataset used for the training is only partially completed, due to the decision of creating a new one from nothing, we suppose this result can only increase completing the dataset. After that, the preprocessing phase and people recognition processes are developed in order to feed the network and collect all the people’s data we’re interested in. Then this part, together with the suitably converted model, is installed on the Google coral Dev Board to execute the whole path to extract the data. Finally, the system is linked to a server module which allows the data transfer to a database (MongoDB) and at the same time it allows to pick the stored data, according with Pepper’s requests. Pepper is implemented by the Behaviour Motor System in order to request the data regarding its interlocutor, modifying its behaviours and speeches depending on the type of data it receives. Running the code we noticed that the performances obtain on the board are not totally satisfactory. This configuration proves to be excellent for data management and analysis but not sufficiently fast to manage a real time conversation. To solve this issue some adjustment have been made in the robot BMS. At the end, what we have obtained is an integrated ”device-to-robot system” able to profile and recognize emotion just focusing on a person. This system is, certainly, not perfect in terms of performances and speed, but it could be a good compromise as regards people data analysis and outlining.| File | Dimensione | Formato | |
|---|---|---|---|
|
Tesi Cerrone e Corti.pdf
accessibile in internet per tutti
Descrizione: Testo della tesi
Dimensione
9.68 MB
Formato
Adobe PDF
|
9.68 MB | Adobe PDF | Visualizza/Apri |
I documenti in POLITesi sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.
https://hdl.handle.net/10589/154244