Combining different artificial intelligence techniques for multimodal sentiment and emotion recognition

This research falls within the interdisciplinary domain of Affective Computing, which is an emerging sub-field of Artificial Intelligence (AI) that aims to develop intelligent systems capable of recognizing and interpreting human affect. Its interdisciplinary nature, which spans several different domains such as computer science, cognitive science, psychology, and social science, renders it to be an important field for researchers to study. Two of the main research topics in this field are Sentiment Analysis and Emotion Recognition. This work seeks to address three main problems within these research areas: (1) the lack of labeled datasets for training models, (2) finding effective features from multimodal data, and (3) developing high-performing machine learning algorithms for various affect prediction tasks. Accordingly, the main overarching goal of this research is to gain an understanding of the use of multimodal data and various machine learning approaches to affect prediction tasks. The influence of several key drivers for success of such tasks was investigated to ascertain how the use of real-world datasets, various feature processing methods, modalities, and classification algorithms, affect prediction performance. The fulfillment of this objective advances the academic knowledge in terms of the proposed novel approaches to collecting labeled datasets, processing multimodal features, and improving prediction performance of machine learning models. From a practitioners’ perspective, this research has demonstrated the effective use of these novel methods in real-world applications. The insights derived from this work can therefore be adopted by practitioners in the industry in the development of commercial affect recognition systems. The contributions of this research also set the ground for future work in terms of finding new ways to automatically collect labeled data, to perform data wrangling on multimodal features, and to leverage hyperbolic learning for feature embeddings of multimodal data.

N/A