Abstract:
Prolonged engagement with digital workstations is increasingly associated with
stress, fatigue, and adverse health outcomes. To address these challenges, an intelligent
application combined with an empathetic robot is proposed for real-time
Facial Emotion Recognition (FER) and wellness monitoring to enhance Human-
Computer Interactions (HCI) in work environments. To determine a user’s emotional
and physical state, the system uses deep learning and sophisticated computer
vision techniques. The primary contributions of this work are established
through three major contributions. The research introduces a uniquely curated,
balanced, multi-modal dataset manually validated across nine distinct emotion
classes including angry, bored, disgust, fear, happy, neutral, sad, stress, and
surprise to ensure demographic diversity and mitigate bias. Second, to process
this data, the system utilizes the state-of-the-art You Only Look Once (YOLO)
v11n-cls model, representing a significant advancement in emotion-aware computing.
Comparative evaluations demonstrate that the YOLOv11n-cls model achieves
94.5% accuracy in FER, decisively outperforming optimized Convolutional Neural
Network (CNN) (57.3%) and hybrid CNN-Long Short-Term Memory (78.2%)
architectures. Furthermore, the YOLOv11n-cls model achieves 98.7% accuracy in
posture-based emotion detection, surpassing transfer learning approaches using
MobileNetV2 (85.31%). Third, the AI workflow processes these emotional states
through Large Language Models, utilizing Reinforcement Learning to provide
highly personalized and contextually relevant wellness interventions. In addition
to the application-based HCI, the robot functions as a physical interface capable
of conversational engagement, gestural interaction, and proactive interventions,
such as promoting ergonomic adjustments and breaks during prolonged static periods.
This dual approach, combining a software application with a physical robot,
facilitates a holistic enhancement of user well-being. The proposed system offers
a practical solution for sustaining productivity while promoting digital wellness,
representing a significant advancement in the field of emotion-aware computing.
