深度學習模型融合風浪資訊於SAR船隻偵測－以臺灣周邊海域為例

Abstract

臺灣海上貿易發達，近海漁業也十分興盛，因此船隻偵測系統對於臺灣的重要性不言而喻。然而，作為船隻偵測主要手段的自動識別系統(Automatic Identification System，AIS)存在訊號範圍受限、小型船隻並無安裝及可以被刻意關閉等問題，造成船隻偵測效率不佳。合成孔徑雷達(Synthetic Aperture Radar，SAR)由於不易受到天氣影響、不受日夜限制等優勢，已被廣泛地應用於船隻偵測任務。隨著深度學習的興起，許多研究都嘗試將此技術應用於SAR影像的船隻偵測。訓練深度學習模型需要資料集，儘管自2017年以來已有五個公開的船隻資料集，這些資料集仍舊有許多不足之處。以遙測角度檢視這些資料集，它們混合不同衛星和不同極化模式的影像的行為並不合理，因為不同衛星和極化模式影像內的船隻特徵會有所差異，進而造成模型訓練困難。本研究也發現前人進行船隻偵測時並未考慮風浪等海面天氣資訊，若能將風浪資訊加入深度學習模型，不僅可以提高模型的可解釋程度，進而發現模型的真實應用場景，亦可以將風浪資訊當作船隻偵測的輔助資訊，有望進一步提升船隻偵測精度。基於上述理由，本研究將嘗試於船隻偵測加入風浪資訊，而為達成此目的，本研究提出一個全新的SAR船隻資料集，名為Taiwan SAR-based Ship and Weather Dataset (TSSWD)。TSSWD以符合遙測原則的方式，從Google Earth Engine蒐集臺灣周邊海域的Sentinel-1影像，結合浮標資訊以測試模型對實際海況的適應能力，並使用光學衛星底圖和AIS來確保船隻標註的正確性。所有深度學習模型的mAP50均超過90，其中Mask R-CNN-x152的表現最為亮眼，在預測框和預測遮罩的mAP50都超過93。本研究也對此模型於不同風浪條件下進行測試，並根據分析結果認定TSSWD的實際應用場景為浪高介於0.1至2.5公尺。此外，本研究嘗試將風浪資訊和SAR影像進行資料融合，以聯合表示的形式建立多模態模型，並進行不同風浪條件下的精度測試。儘管多模態模型的整體測試精度並未超越未加入風浪資訊的NoFusion模型，但是在穩定性上優於NoFusion，而加入風速的多模態模型於低風速的場景下，其精度表現略優於NoFusion。本研究也針對模型的誤判率進行分析，發現誤判數量隨著浪級提升而有所上升，然而中低等級風浪的誤判數量佔比為67%，是需要優先改進的部分。總結來說，TSSWD是第一個符合遙測原則並結合浮標資料的資料集，不僅提供後人將船隻與風浪資訊進行結合的機會，也有助於建立結合風浪資訊的多模態模型。本研究將風浪數值當作船隻偵測模型的輔助資訊，與SAR影像一同建立多模態深度學習模型，並證明使用風浪資訊的多模態模型具有較高的穩定性，而此模型的出現也提供後續研究者一個全新的方向。

Taiwan's developed maritime trade and thriving coastal fishing industry underscore the significance of developing ship detection systems. However, the Automatic Identification System (AIS), the primary means for ship detection, suffers from limitations such as signal range constraints, lack of installation on small vessels, and vulnerability to intentional shutdowns, leading to suboptimal ship detection efficiency. Synthetic Aperture Radar (SAR), with its advantages of weather independence and day-night operational capability, has been widely applied in ship detection tasks. With the rise of deep learning, researchers have explored applying this technique to ship detection using SAR imagery. Despite the availability of five publicly accessible ship datasets since 2017, these datasets still have several shortcomings. Examining them from a remote sensing perspective reveals an illogical mixing of imagery from different satellites and polarimetric modes, which makes model training challenging due to variations in ship features among different sources and modes. Additionally, previous ship detection studies have not considered vital oceanic information, such as wind and wave data. Integrating wind and wave information into deep learning models not only enhances interpretability to discover realistic application scenarios but also serves as auxiliary data for ship detection, potentially improving detection accuracy. In light of the above, this research considers the inclusion of wind and wave data in ship detection as a feasible attempt. To achieve this, a novel SAR-based ship dataset named Taiwan SAR-based Ship and Weather Dataset (TSSWD) is proposed. TSSWD adheres to remote sensing principles by collecting Sentinel-1 imagery from the ocean surrounding Taiwan using Google Earth Engine. It combines buoy information to evaluate model’s adaptability to real sea conditions and uses optical satellite imagery and AIS to ensure accurate ship annotation. All deep learning models achieve mAP50 values exceeding 90, with the Mask R-CNN-x152 model exhibiting exceptional performance with mAP50 values exceeding 93 in both bounding box and mask prediction tasks. The research also tests the proposed model under different wind and wave conditions and identifies the practical application scenario for TSSWD as SAR imagery with wave heights ranging from 0.1 to 2.5 meters. Moreover, the study attempts to fuse wind and wave data with SAR imagery to construct a multimodal model and undergoes accuracy testing under various wind and wave conditions. Although the overall testing accuracy of the multimodal model does not significantly surpass that of the NoFusion model without wind and wave data, the research discovers the superior stability of the multimodal model compared to NoFusion. Notably, the multimodal model with wind speed inclusion demonstrates slightly better accuracy in low wind speed scenarios. The research also analyzes the misclassification rates of the models and finds that misclassification increases with rising wave levels, with moderate to low wave conditions accounting for 67% of misclassifications, indicating an area in need of priority improvement. In summary, TSSWD is the first dataset that adheres to remote sensing principles while incorporating buoy data, providing researchers with the opportunity to integrate ship detection with wind and wave information. Moreover, it facilitates the establishment of multimodal models that combine wind and wave information, with the proposed wind-wave multimodal model being the first to use wind and wave data as auxiliary information for ship detection alongside SAR imagery. Its emergence offers subsequent researchers a new direction in ship detection research.