Loneliness has a critical impact on the mental health of citizens, particularly among the elderly. Robots capable of perceiving and responding to human emotions can serve as heart-warming companions to help lift the spirits. A research team at The Hong Kong Polytechnic University (PolyU) has discovered that the combined power of music and empathetic speech in robots with artificial intelligence (AI) could foster a stronger bond between humans and machines. These findings underscore the importance of a multimodal approach in designing empathetic robots, offering significant implications for their application in health support, elder care, education and beyond.
Study design and key findings
The research project was led by Prof. Johan Hoorn, Interfaculty Full Professor of Social Robotics of the School of Design and the Department of Computing at PolyU, in collaboration with Dr. Ivy Huang at The Chinese University of Hong Kong. The study investigated how music and empathetic speech could enhance the emotional resonance of on-screen robots, revealing that music can act as a powerful adjunct to empathetic speech. The research is published in the journal ACM Transactions on Human-Robot Interaction.
As part of the study, the team examined how Cantonese-speaking participants interacted with empathetic robots across three interactive sessions. The findings showed that combining music and speech significantly increased the participants’ perceived empathy of the machines.
“Our data indicate that the presence of music continued to enhance the robot’s resemblance to humans in later sessions,” explained Prof. Hoorn. “One interpretation is that music made the interaction feel more like a real conversation with a personality, something human counselors might do by playing music to comfort their clients, which in turn made the robot seem more lifelike or socially present.”
Adapting empathy over time
However, the research pointed out that the impact of music could diminish over time when the participants became attuned to the music after repeated sessions, highlighting the importance of tailoring interaction strategies to individual users’ needs to sustain effective human-robot interaction. The study suggested that empathetic robots should be designed to adapt their responses to user feedback and context, for example, by adjusting various musical elements or gradually personalizing dialogue content to maintain sustained relevance of empathy.
Prof. Hoorn emphasized, “Our research points to the significance of multimodal communication encompassing music, speech and more through empathetic robots. It holds considerable promise for application in real-world settings, particularly in the fields of mental health support and elderly care. The integration of empathetic robots capable of delivering tailored musical experiences and engaging in sensitive conversation could provide meaningful companionship and emotional support to individuals who may experience loneliness or social isolation.”
Future directions and quantum models
Prof. Hoorn is leading another project, “Social Robots with Embedded Large Language Models Releasing Stress among the Hong Kong Population.” Concurrently serving as Associate Director of the PolyU Research Institute for Quantum Technology, Prof. Hoorn is set to explore quantum-inspired models of human affect to better capture and respond to the inherent vagueness and ambiguity of emotional experience.
Unlike traditional computational systems that struggle with the fluid and context-dependent nature of affective responses, quantum models can represent emotional states as probabilistic superpositions, reflecting the genuine uncertainty and complexity of human feelings.
“What excites me the most is the possibility of developing social robots that not only recognize the complexity of human affect but also embrace it. These robots could offer support that is adaptable, open-ended and compassionate, similar to the individuals they are designed to help,” added Prof. Hoorn.
More information:
Ivy S. Huang et al, A Talking Musical Robot over Multiple Interactions: After Bonding and Empathy Fade, Relevance and Realism Arise, ACM Transactions on Human-Robot Interaction (2025). DOI: 10.1145/3758102
The content is provided for information purposes only.