巨量網路活動資料之即時運算與視覺化呈現

Abstract

網路流量隨著科技的進步和普及有逐漸成長的趨勢，其勢必會帶動更複雜的網路活動，和資安攻擊事件的增長，如何儲存這巨量的網路流量並彈性且快速地取得提供作資安分析成為一個具有挑戰性的課題。例如在分析即時的資安事件上，這需求對於過往龐大的流量資料進行快速的分析，在面對成長趨勢有著猛烈成長的流量資料來說，資安管理人員需要一套新的資料管理與計算架構與模式及方法才來達到管理者的要求。 本論文將提出一個基於雲端架構的巨量網路通訊活動資料管理系統，”搭配上行導向儲存”、”階層式檔案路徑”、”資料重複壓縮技術”，提出一個完善的架構，幫助管理者根據資料分析的巨、微觀維度，迅速地找出相關過往的NetFlow資料。此系統能夠針對使用者所提出的查詢，迅速並且及時地提供相對應的資料供前端呈現有意義的結果給使用者。 關鍵詞：雲端計算、行導向儲存、階層式檔案路徑、資料重複壓縮、巨量資料

As the network volume growing rapidly, there should have much more complicated network activities and security problems. How to store the big volume of network traffic and to access the data as soon as possible for security analysis is a challenging work. For example, in the analysis of real-time information security events, it requires past huge NetFlow data for analysis. In the face of rapidly growing trends for NetFlow data, information security managers need a new set of data management, computing architectures, models and methods to achieve the requirements. This paper will present a cloud-based architecture, network communications activity massive data management system, with “Column-based Segmentation”, “Directory path hierachy”, “Deduplication”. Proposed a comprehensive framework to help managers in accordance with giant, micro-dimensional of data analysis to rapidly retrieve relative past NetFlow data. This network communications activity massive data management system could provide queries and show views of network communication activities easily, clearly and quickly. Keywords: could computing, Column-based Segmentation, Directory path hierachy, Deduplication, big data