遲發性運動障礙辨識之多模態機器學習框架：基於語 音與臉部動作分析

Abstract

遲發性運動障礙(Tardive Dyskinesia, TD)是一種由藥物引起的運動障礙,估計影響20%至30%接受長期抗精神病藥物治療的患者,但在臨床實踐中仍普遍存在診斷不足的問題。TD患者的言語障礙已被臨床記錄超過三十年,但目前尚無自動化系統用於分析TD患者的言語特徵。本論文提出了一種多模態機器學習框架,透過整合來自簡短、真實場景視訊錄製中的語音和面部分析,實現TD的自動化篩檢。該框架結合了基於解耦機制的音訊模組與3D面部運動學分析模組,並透過跨模態注意力機制將兩者連接,使兩種模態能夠動態互動。我們在包含80名受試者的資料集上評估了該框架,受試者分為三類:TD、帕金森病(Parkinson's Disease)和健康對照組,並採用受試者獨立的交叉驗證方法。該多模態模型在健康對照組與TD的二分類任務中取得了0.93的AUROC值,在三分類任務中取得了0.89的AUROC值,在所有任務中均持續優於僅使用面部或僅使用音訊的基線模型。我們還採用了可解釋人工智慧(Explainable AI, XAI)方法,以提供可解釋的預測結果,將模型的注意力機制與已知TD言語障礙相關的語音片段聯繫起來。總體而言,我們的研究填補了一個持續三十餘年的診斷空白,並朝著在真實臨床環境中實現客觀TD篩檢邁出了重要一步。

Tardive Dyskinesia (TD) is a medication-induced movement disorder affecting an estimated 20-30% of patients on long-term antipsychotic treatment, yet remains widely underdiagnosed in clinical practice. Speech impairment in TD has been clinically documented for over three decades, yet no automated system has been used for analyzing speech in TD. This thesis presents a multimodal machine learning framework for automated TD screening by integrating voice and facial analysis from brief, in-the-wild video recordings. The proposed framework combines a disentanglement mechanism-based audio module with a 3D facial kinematic analysis module connected via cross-modal attention, enabling the two modalities to interact dynamically. We evaluated this framework on a dataset of 80 subjects across three classes: TD, Parkinson's Disease, and healthy controls under subject-independent cross-validation. The multimodal model achieved an AUROC of 0.93 for healthy control versus TD discrimination and 0.89 for the three-class classification task, consistently outperforming the face and audio only baseline across all tasks. We also employed Explainable AI (XAI) to provide interpretable predictions, linking model attention to speech segments consistent with known speech impairments in TD. Overall, our work addresses a diagnostic gap that has persisted for over three decades and represents a step toward objective TD screening in real-world clinical settings.