KG-LAD: 利用知識圖譜增強之高效日誌異常檢測方法

Abstract

軟體系統正迅速滲透於我們日常生活中使用的許多服務，其中系統日誌對於 服務提供商用於監控系統運行與偵測潛在故障具有極高的價值。雖然過去已經有許多研究採用機器學習與深度學習技術來分析大量系統日誌以偵測系統異常，但鮮少有研究使用知識圖譜用於檢測日誌異常的任務中。在本研究中，我們提出了一種基於知識圖譜與循環神經網路的高效日誌異常檢測方法──KG-LAD。我們將從 日誌訊息中提取的日誌模板視為知識圖譜中的實體節點，並將透過兩大通用關係 將節點間相互連接，其連接方式並不須仰賴專家編譯規則或提取特定參數，具有泛用性高的特性。節點（日誌模板）與邊（關係）所衍生出的嵌入用於計算日誌序列的距離分數，並將其輸入至基於 LSTM 模型的分類器中，以識別系統運行是否異常。為了評估我們研究方法之成效，我們在三個大型資料集上進行廣泛的實驗，不僅證明知識圖譜用於異常檢測之價值及泛化能力，還展示了其方法的穩健性及準確性。不論是在精確度、召回率、及 F1 分數，各個評估指標上都優於其他最先進的日誌異常檢測方法。

Software systems are rapidly permeating many services we use in our lives. The generated system logs are valuable for service providers to monitor service operation and to prevent potential malfunctions. While much research has adopted machine learning and deep learning approaches to analyze large volumes of logs, few studies have investigated knowledge graphs to detect anomalies. In this paper, we present KG-LAD, a novel method that leverages knowledge graphs and recurrent neural networks to detect system anomalies through log analysis. The designed knowledge graph treats log templates as nodes. Instead of relying on expert-compiled rules or domain-specific log parameters, two generic relations are designed to associate these nodes: structural connectivity and positions in log sequences. These relations form the edges between the nodes in the knowledge graph. The derived embeddings of nodes (log templates) and edges (relations) are used to measure the distance scores of a log sequence and are input to an LSTM-based classifier to determine whether the log template sequence indicates abnormal system operation. Extensive experiments conducted on three large datasets demonstrate not only the effect and the generalization of the knowledge graph, but also the robustness of KG-LAD and its superior performance over state-of-the-art log anomaly detection methods for precision, recall, and F1 score.