分散式巨量網路流量資料之交互式視覺化安全分析系統

Abstract

網路流量隨著科技的進步和普及有逐漸成長的趨勢，其勢必會帶動更複雜的網路活動，和資安攻擊事件的增長，如何儲存這巨量的網路流量並彈性且快速地取得，同時以人眼可辨識、清楚的方式呈現提供作資安分析成為一個具有挑戰性的課題。因而本論文提出一個雲端分散式處理系統，NetActy（Network Activity Visualization System），其運作在虛擬環境下並採用類似MapReduce 巨量資料平行處理架構的設計來幫助處理大量的NetFlow資料，透過雲端技術與平台動態運算資源的調配能力來幫助迅速地佈署達到運算期限所需要的資源。而快速轉換資料成視覺圖像的能力取決於資料提供的快慢，因此本論文設計幫助快速視覺化的in-memory階層式BigIP Render Tree（BRT）資料結構，針對IP活動記錄進行視覺化資料設計，提供CIDR、AS和國家範疇的網路活動查詢，並以快速且清楚、人眼可辨識的視圖方式呈現。在系統評估的結果驗證系統具有擴充性，且BRT的建立和存取表現優異，讓NetActy系統能夠針對使用者所提出的查詢迅速並且及時地提供相對應的資料供前端呈現有意義的結果給使用者。

As the network volume grows rapidly, network activities and security problems are supposed to be more complicated. For security analysis, it is challenging to store the big volume of network traffic and access the data in real time. We propose a distributed processing system of scalable cloud environment called NetActy System (Network Activity Visualization System). NetActy is built on top of a VM cluster, which could allocate computing resource flexibly and improve resource utilization. There are modules in NetActy that function like the MapReduce framework to process big data. In addition to this, NetActy is able to visualize and display hierarchical graph data in the user interface. We design an in-memory intermediate data structure called BigIP Render Tree (BRT). BRT provides IP-, CIDR-, AS- and country-level query of network activities and its in-memory design could facilitate access to graph data. In a nutshell, NetActy is a system that provides queries and show views of network communication activities easily, clearly and quickly.