In clinical settings, accurately understanding patients’ emotions and responding appropriately plays a critical role in improving treatment outcomes and patient satisfaction.
In a study appearing in IEEE Access, researchers at University of Tsukuba developed a noncontact “multimodal emotion recognition” framework that combines multiple types of information, including patients’ voices, conversations with physicians, and physiological responses.
This technology acquires physiological data such as heart rate and breathing without any physical contact, integrates it with voice and speech content, and analyzes patients’ emotions accurately.
Researchers conducted consultations between experienced physicians and simulated patients trained to replicate cancer treatment scenarios.
They compared the accuracy of emotion recognition by physicians and the AI system against patients’ self-reported emotions. The AI system scored higher than physicians in terms of how accurately emotions were captured, demonstrating that it can recognize emotions more accurately.
Conventionally, physicians are regarded as having high-level empathy. However, this study shows that AI has the potential to surpass specialists in emotion recognition by integrating and processing diverse information sources.
Another significant advantage is that noncontact technology reduces patients’ physical and psychological burden while enabling emotion data collection in a natural conversational environment.
These findings highlight the potential of this technology as a new support tool for physicians, helping them accurately understand patients’ emotions and reduce the risk of “empathy fatigue” among health care professionals.
The research team plans to conduct further verification and improvements to implement the system into real-world medical settings and expand its applications to areas such as elderly care and mental health support.
More information:
Homura Kawamura et al, Framework for Emotion Recognition Using Cross-Modal Transformers With Non-Contact Multimodal Signals Aiming Clinical Service Support, IEEE Access (2025). DOI: 10.1109/ACCESS.2025.3573648
The content is provided for information purposes only.