Emotionally intelligent social robot for dementia care

Dementia is a challenging health issue for healthcare systems across the globe. Communication disabilities and behavioral changes have been significantly impacting the well-being of patients with dementia. The study formulates an empathy-based conversational intervention approach for patients with dementia using multimodal sentiment analysis. The proposed system applies a cross-modal attention-based model to synthesize speech, facial expressions, and biological signals for effective emotion identification. The synthesis is further augmented with an innovative large language model-based conversational response generation module that can develop appropriate empathetic responses. Experiments conducted on public benchmark databases confirm that the trimodal fusion-based model outperforms state-of-the-art methods with an overall weighted average accuracy of approximately 87.3% for emotion identification. The proposed approach outperforms state-of-the-art methods such as MulT, MISA, and MAG-BERT. The scores on human evaluation of the generated empathetic dialogue reached 4.12 and 4.28 in terms of empathy and coherence, with improvements of 17.0% and 12.3% over baseline models. The meta-analytic synthesis of previous clinical evidence revealed significant beneficial effects of social robot interventions on depression, loneliness, and agitation of people with dementia. The comparison with commercial models such as PARO, Pepper, and NAO showed the superiority of the proposed approach over others in terms of multimodal emotion recognition and dialogue adaptability. These results show that socially interactive robots with high emotional intelligence, equipped with cutting-edge affective computing and natural language processing, have great potential for enhancing the quality of dementia care through personalized emotional support.