基於急診室之心電圖影像與呼吸影片預測住院風險的深度學習模型

Abstract

急診室擁擠已成為現代醫療體系中的一項重大挑戰，不僅延誤病患的及時治療，也影響資源的最佳分配。臨床醫師常依賴快速且主觀的「臨床直覺」評估來輔助分診與決定是否住院。我們假設深度學習模型同樣能利用病患的影像外觀，以更客觀且一致的方式預測住院需求。本研究提出一種創新的心肺雙模態深度學習模型，整合兩種非侵入式資料：從短程呼吸影片中擷取的呼吸波形影像，以及自心電圖（ECG）影像。為了強化模型對關鍵特徵的聚焦，我們引入注意力機制，並使用 Class-Balanced Loss 處理訓練資料中嚴重的類別不平衡。考量只有部分病患提供完整的 ECG 資料，我們先在外部資料集上以 ConvNeXt 架構預訓練 ECG 特徵擷取器，以提升心電訊號的表示能力。對於呼吸模態，我們首先在無固定攝影機角度的複雜環境中，自動定位病患的胸部感興趣區域；接著透過 3D 卷積神經網路與雙向長短期記憶（BiLSTM）網路組合，估計呼吸訊號並將其轉換為影像格式，由端到端訓練的 ConvNeXt 模型進行特徵抽取。最後，我們將兩種模態的特徵向量與一個標示 ECG 資料可用性的二元指標串聯，並透過全連接層預測住院結果。

在臺大醫院（NTUH）資料集上的實驗結果顯示，融合 ECG 與呼吸模態的模型達到 AUROC 0.863 和準確率 0.814；僅使用呼吸影片的模型也獲得 AUROC 0.810 和準確率 0.788。這些成果證明本方法具備作為急診室客觀、非侵入式臨床決策支援工具的可行性。

Emergency department overcrowding has emerged as a critical challenge in modern healthcare systems, compromising timely patient care and optimal resource allocation. Clinicians often rely on rapid, subjective “clinical gestalt” assessments to aid triage and disposition decisions. We hypothesize that a deep learning model can similarly leverage patient appearance to predict hospital admission with greater objectivity and consistency. In this study, we propose a novel Cardio-Respiratory Deep Learning Model that integrates information from two modalities: electrocardiogram (ECG) images and respiratory waveform images extracted from short patient breathing videos. To enhance the model's focus on critical features, we incorporate an attention mechanism and address severe class imbalance in our training data using Class-Balanced Loss. Recognizing that only a subset of patients provide complete ECG data, we pretrain our ECG feature extractor on an external dataset using a ConvNext model to improve cardiac signal representation. For the respiratory modality, we first deploy a deep learning method to automatically localize the patient's chest region of interest (ROI) in unconstrained environments without relying on a fixed camera position. The respiratory signal is then estimated using a combination of 3D convolutional neural networks and bidirectional long short-term memory (LSTM) networks, followed by converting the signal into an image format for feature extraction via a ConvNext model trained end-to-end. Finally, the extracted features from both modalities, along with a binary indicator denoting ECG data availability, are concatenated and processed through fully connected layers to predict hospital admission.

Experimental evaluation on the NTUH dataset demonstrates that our fusion model using both data modalities achieves AUROC of 0.863 and Accuracy of 0.814, whereas the model using only respiratory videos achieves AUROC of 0.810 and Accuracy of 0.788. These results demonstrate the potential of our method as an objective, non‑invasive clinical decision support tool for use in the emergency department.