libyt: 一個協助模擬即時平行分析的工具和它的相關應用

Abstract

隨著要解決的物理問題越來越大，要模擬的內容越來越精細，模擬本身的大小、運算量、資料量也隨之增加。這使得模擬演化過程中的每一個時間點的資料量也跟著增加。雖然已經有相應的分析軟體可以處理這些資料，但是他們都必須要在模擬輸出整個資料結構和資料本身下才有辦法分析。假如我們想要分析一組隨時間變化的資料，那麼這組資料將會佔據大量的硬碟空間。又或者我們必須存放整筆資料以便軟體分析，但其實真正分析的只有其中的一小部分，這會非常浪費硬碟空間。我們開發libyt讓使用者可以任意用Python或yt直接分析存在記憶體裡的資料，分析完後繼續跑模擬，來解決這個問題。 我們描述如何用Python C API和NumPy API來開發libyt和連接應用程式與Python；我們把libyt實際應用在真實的物理模擬中，並說明它如何幫助分析這些大型模擬；我們把libyt和原始的方法做比較，說明它更具有延展性和優勢；最後，我們提及libyt可以如何增加更多功能以便連接到其他常用的分析軟體。

The size and scale of simulations have grown rapidly in order to solve larger physics problems and to render the data in a higher resolution. This resulted in bigger dataset in each step in simulations. Analytic tools have been developed to analyze and visualize these output datasets, but they suffer from having simulations to store the full datasets on disk first before further analyzing them. This may take up lots of disk space if we want to analyze the data in a high resolution time series. This also wastes lots of space storing unnecessary data, if we only need a small portion from them. We developed libyt to solve this problem. libyt is a C++ library for yt, that aims to make simulations able to use yt or other Python packages to analyze data arbitrarily in code runtime. We describe how we use Python C API and NumPy API to build libyt and connect simulations to Python instances. We demonstrate that it plays an important role in large production run simulations. We discuss the scalability of libyt and its advantage comparing to the original post-processing method. Finally, we show how libyt can be extended and connected to other analytic tools in the upcoming updates.