Multimodal Audio-Visual Emotion Recognition for Social Robotics

Thanks to recent advances in deep learning based algorithms, humanoid social robots are increasingly exhibiting human-like behaviors. In this context, the analysis of soft biometrics, particularly emotion recognition, is crucial for enhancing communication between social robots and humans, facilitating emotion-aware dialogues. In light of these considerations, we propose a multimodal emotion recognition system tailored for social robotics applications. The system processes both video and audio data to classify the emotion in one among six different classes, employing 3D convolutional operations that eliminate the need for transformer-based architecture, effectively reducing the model’s size and making the network able to run over low power embedded devices mounted directly on board of the robot. The proposed approach was trained on the CREMA-D dataset and demonstrates impressive performance when compared to video-only and audio-only counterparts, outperforming state-of-the-art methods both unimodal and multimodal.