│Return to list│Previous page│Next page│
| ＜E-04＞：Mechanical Engineering |   |
Application of Multimodal Emotion Estimation Technology for Supporting the Diagnosis of Mental Disorders
― Achieving Emotion Estimation with Interpretability and Causal Inference ―
| Kazuyuki Matsumoto | Associate Professor |
| Information Science, Computer Science, Division of Science and Technology, Graduate School of Technology, Industrial and Social Sciences | |
| Keywords | Multimodal Emotion Estimation, Causal Inference, Interaction, Correlation, Heterogeneity |
| Laboratory URL | https://www-a2.is.tokushima-u.ac.jp/home |
| Research Overview | |
| ● Increased risk of mental disorders such as depression caused by social isolation among the elderly In an aging society, depopulation of local communities and social isolation among the elderly—exacerbated by the COVID-19 pandemic—have become major social issues, as they increase the risk of developing mental disorders such as depression and dementia. Furthermore, depopulation in rural areas limits opportunities for medical diagnosis by physicians, making early-stage disease identification difficult. To address these challenges, this research aims to develop a web-based mental disorder diagnostic support system for the early detection of mental disorders and appropriate medical consultation among elderly individuals living in socially isolated environments (Fig. 1). ● Supporting physicians’ diagnostic activities through interpretable output of state estimation results By associating the computed relationships based on co-occurrence and differences among modal features with discrete semantic concepts that are easily interpretable by humans, we achieve emotion estimation and causal inference with interpretability. As a preprocessing step, shared and independent features across modalities are extracted through operations on each modality’s features, allowing more effective relationship extraction than learning directly from raw data. The inter-modal relationships can be visualized and output in a form that is interpretable by humans through their association with semantic concepts. |
|
| Potential Applications and Commercialization Prospects | |
| ● Application to Support the Diagnosis of Mental Disorders and Stress Conditions in Counseling and Healthcare Settings Currently, we are preparing multimodal data and implementing the model. The completed multimodal emotion estimation model is expected to be applied to diagnostic systems that can detect stress accumulation or the potential for mental disorders simply by interacting with a dialogue AI, through integration with dialogue processing systems using large-scale language models. It may also be applied to diagnostic systems that predict cognitive function levels based on fluctuations in emotional states.   |
| Patents | ● WO/2025-243867 “Data Generation Method, Learning Model Generation Method, Information Processing Method, Computer Program, And Information Processing Device” |
| Publications | ● Kazuyuki Matsumoto et al., Learning a Bimodal Emotion Recognition System Based on Small Amount of Speech Data. 2022 8th International Conference on Systems and Informatics (ICSAI). ● Kazuyuki Matsumoto et al. (2023). Relationship between Client Stress and Multimodal Emotion Estimation in Counseling Settings. Proceedings of the 18th Spring Conference of the Japan Society of Kansei Engineering. (Oral presentation) March 2023 ● Kazuyuki Matsumoto et al. (2023). Toward the Development of a Standardized Interview Method for Evaluating Mental Stress in Workers. Proceedings of the 93rd Annual Meeting of the Japanese Society of Hygiene. (Poster presentation) March 2023 |
Last updated date：October 15, 2026
| Simple Inquiry | Technical consultation | Academic guidance system | Joint research/commissioned research |
