利用元數據和衛星影像的多模態模型進行地震災後建築損壞預測

Abstract

本研究旨在建立一個多模態模型搭配建築物元數據以及災前與災後衛星影像應用於預測地震後建築物災損程度。一套完整的多模態模型的運作流程在本研究中被建立，此外還包含數據蒐集與配對、建立針對單一資料的模型、特徵融合以及最終預測模型。為了探討多模態模型的影響力，本研究亦比較了多模態模型和單一資料源模型的效能，並以兩個案例進行討論。 本研究首先調查了僅使用單一資料模型的影響力，分別是使用建築物元數據以及衛星影像。針對元數據首先進行資料前處理，模型部份採用的是傳統的多層感知器(MLP)。本研究發現元數據模型在某些類別中有其侷限性，特別是在’Destroyed’這個類別。而衛星影像的模型則採用孿生-UNet架構，透過圖像分割將建築物所在位置標記出來再進行分類，發現影像模型在’Major’這個損害類別中難以正確辨識。這些發現指出本研究需要考慮更多元的資料型態，來解決現階段預測錯誤的問題。 透過建物元數據搭配衛星影像建立成多模態模型的數據集，並分為案例一(2010年海地地震+2015年尼泊爾地震)以及案例二(2016年美濃地震+2018年花蓮地震)。案例分類的方法是依照房屋類型，在案例一中，磚土造的房屋佔多數，而在案例二中以鋼筋混凝土建築為大宗，透過不同型態的測試資料可用以驗證多模態模型的泛用性。 由案例一的結果可以發現，和使用單一資料模型相比，通過多模態模型，本研究在每個類別中都取得了改進，特別是在’Major’和’Destroyed’這兩個類別效果更顯著。這說明考慮更多元的資料類型可以幫助提升預測的準確性。而案例二在單一資料模型中於’No Damage’和’Minor’這兩個類別已達到不錯的水準，通過多模態模型後效能提升雖不顯著，但亦有修正幾個被錯誤判斷的案例。儘管多模態模型在案例一中效果顯著，但透過案例二可以說明輸入特徵的多樣性對於多模態模型修正單一資料模型的預測結果而言很重要。 總結來說，這項研究建立了一個多模態模型用以預測地震後建築物的災損程度，強調了提高數據的多樣性可以提升預測的準確度。在未來可以利用本研究建立好的完整流程，納入更多能夠幫助判別建築物損害程度的資料型態，亦可利用在其他災害，為自動化災後評估損害開創了一條新的道路。

This research establishes a multimodal model combined with building metadata and pre- and post-disaster satellite imagery to predict the extent of earthquake-induced building damage. We have developed a comprehensive workflow for the multimodal model, which includes data collection and pairing, creating models for single types of data, feature fusion, and final prediction models. To explore the impact of the multimodal model, we compared its performance with that of single-source data models and discussed the results in two case studies. Initially, we investigated the impact of using single-modal models, specifically using building metadata and satellite imagery. For the metadata, we performed data preprocessing and utilized a traditional Multilayer Perceptron (MLP) for modeling. We found that the metadata model had limitations in some categories, especially in the 'Destroyed' category. The image model used a Siamese-UNet architecture, which marks the location of buildings through image segmentation before classification. We observed that the image model struggled to accurately identify the 'Major' damage category. These findings highlight the need to consider a more diverse set of data types to address current predictive errors. By integrating building metadata with satellite imagery, we established a multimodal dataset, divided into Case Study 1 (2010 Haiti earthquake + 2015 Nepal earthquake) and Case Study 2 (2016 Meinong earthquake + 2018 Hualien earthquake). The method of categorization was based on the type of housing, with masonry structures predominating in Case Study 1 and reinforced concrete buildings in Case Study 2. Testing with different types of test data verified the versatility of the multimodal model. Results from Case Study 1 showed improvements in every category compared to single-modal models, particularly significant in the 'Major' and 'Destroyed' categories. This illustrates that considering a more diverse range of data types can help enhance prediction accuracy. In Case Study 2, the single-modal models already performed well in the 'No Damage' and 'Minor' categories. While the improvement through the multimodal model was not significant, it did correct several cases that were misjudged. Although the multimodal model was highly effective in Case Study 1, Case Study 2 demonstrated the importance of input feature diversity for correcting predictions of single-modal models. In summary, this study has established a multimodal model for predicting post-earthquake building damage, emphasizing that increasing data diversity can improve prediction accuracy. In the future, our established comprehensive process can incorporate more data types that help determine building damage levels, and can also be used in other disasters, paving a new path for automated post-disaster damage assessment.