基於圖卷積網路及注意力機制進行系統日誌異常偵測

Abstract

異常偵測是建立安全可靠系統的關鍵步驟之一。目前，許多應用與服務都依賴於電腦系統，一旦發生故障，將對使用者和企業造成重大影響。為了避免造成巨額損失，我們可以透過監控系統日誌來了解系統的狀態，並建立自動異常偵測系統，以即時識別和解決異常情況。然而，有效分析日誌資料面臨著一些挑戰。因為日誌通常非常龐大且複雜，因此需要適當的分析工具和技術進行資料清理和預處理，以提高日誌分析的準確性和效率。過去的研究通常僅依賴於分析局部日誌事件的順序和頻率，忽略了日誌事件之間的結構關係和遠程依賴性，這可能導致潛在的誤報和性能不穩定。為此，本研究提出了一種基於圖的日誌異常偵測方法，首先將日誌進行前處理並分組成日誌序列，之後將日誌序列表示為圖結構，考慮事件之間的轉換關係，並將相關資訊作為有向邊的權重，用來捕捉了事件的發生順序和相互關係，接著通過使用圖卷積神經網絡結合注意機制，考慮到多層圖結構資訊，捕捉可能指示異常的日誌特徵並執行圖級分類。在分散式系統與超級電腦的日誌資料實驗顯示，我們提出的方法性能優於其他現有的基於日誌的異常偵測方法。

Anomaly detection is crucial for a secure and reliable system. Currently, many services rely on computer systems, and any failure can have a significant impact on users and businesses. To avoid substantial losses caused by failures, we can monitor system logs to understand the system's status and build an automated anomaly detection system to identify and resolve abnormal situations in real-time. However, effective analysis of log data faces several challenges. Due to the typically large and complex nature of logs, proper analysis tools and techniques are needed for data cleaning and preprocessing to enhance the accuracy and efficiency of log analysis. Past research often relied solely on analyzing the order and frequency of local log events, overlooking the structural relationships and long-range dependencies between log events, which could lead to potential false positives and performance instability. To address these challenges, this study proposes a graph-based approach for log anomaly detection. Firstly, the logs are preprocessed and grouped into log sequences. Then, the log sequences are represented as a graph structure, considering the transition relationships between events and using the relevant information as weights on directed edges to capture the occurrence order and interrelationships between events. Subsequently, by utilizing graph convolutional neural networks combined with attention mechanisms, the method takes into account the multi-layered graph structure information to capture log features that may indicate anomalies and perform graph-level classification. Experiments on log data from distributed systems and supercomputers demonstrate that our proposed method outperforms other existing log-based anomaly detection methods in terms of performance.