A Multi-Modal Emotion-Aware System for Workplace Wellness Using Emotion Recognition and an Empathetic Robot.

Abstract

Prolonged engagement with digital workstations is increasingly associated with stress, fatigue, and adverse health outcomes. To address these challenges, an intelligent application combined with an empathetic robot is proposed for real-time Facial Emotion Recognition (FER) and wellness monitoring to enhance Human- Computer Interactions (HCI) in work environments. To determine a user’s emotional and physical state, the system uses deep learning and sophisticated computer vision techniques. The primary contributions of this work are established through three major contributions. The research introduces a uniquely curated, balanced, multi-modal dataset manually validated across nine distinct emotion classes including angry, bored, disgust, fear, happy, neutral, sad, stress, and surprise to ensure demographic diversity and mitigate bias. Second, to process this data, the system utilizes the state-of-the-art You Only Look Once (YOLO) v11n-cls model, representing a significant advancement in emotion-aware computing. Comparative evaluations demonstrate that the YOLOv11n-cls model achieves 94.5% accuracy in FER, decisively outperforming optimized Convolutional Neural Network (CNN) (57.3%) and hybrid CNN-Long Short-Term Memory (78.2%) architectures. Furthermore, the YOLOv11n-cls model achieves 98.7% accuracy in posture-based emotion detection, surpassing transfer learning approaches using MobileNetV2 (85.31%). Third, the AI workflow processes these emotional states through Large Language Models, utilizing Reinforcement Learning to provide highly personalized and contextually relevant wellness interventions. In addition to the application-based HCI, the robot functions as a physical interface capable of conversational engagement, gestural interaction, and proactive interventions, such as promoting ergonomic adjustments and breaks during prolonged static periods. This dual approach, combining a software application with a physical robot, facilitates a holistic enhancement of user well-being. The proposed system offers a practical solution for sustaining productivity while promoting digital wellness, representing a significant advancement in the field of emotion-aware computing.