基於爬蟲的跨資料庫二元關係呈現工具與家譜數位化的應用

Abstract

中國社會學家潘光旦之著作《明清兩代嘉興的望族》中濃縮家譜議題的精華，對家譜學理論的發展有不小的貢獻。然則因於年代因素，頁面殘破與字體難以辨識，因此將其數位化有急迫性與重要的意義存在。考量數位化的四種層次，第一是將原始資料轉換成圖像，第二是圖像轉成文字，第三是為文字資料加上適當的標記，第四是為標記的資料進行視覺呈現。要達成第三種與第四種層次所花的成本遠超過前兩者，且常常面臨可擴展性、多人共同作業、資料檢核與呈現上的困難。維基與爬蟲工具搭配的形式能滿足這部份需求，在此篇論文中將基於開源的BookStack維基平台進行資料建置。 要完整呈現數位化內容，僅靠原始文本是不足的，會用到第三方資料庫進行參照。然而跨越資料庫引用有技術上的限制，因此設計爬蟲來整合跨資料庫的內容是用來突破技術限制的方法。而要利用爬蟲跨資料搜尋則需要為其設計運行規則，其中基礎的規則是使用廣度優先搜尋，這對於使用者自建的小量資料是足夠的。其中渲染資料二元關係圖形的效能與搜尋結果複雜度相關，超出執行時間導致的渲染失敗將會是重大挑戰。

Chinese sociologist Pan Guangdan's book 'The Prominent Families of Jiaxing in the Ming and Qing Dynasties' condenses the essence of genealogy issues and has made great contributions to the development of genealogy theory. However, due to age factors, the pages are broken and the fonts are difficult to identify, so it is urgent and important to digitize them. Digitization is divided into four levels. The first is to convert the original data into images. The second is to convert the images into text. The third is to add appropriate markings to the textual data. The fourth is to visually present the marked data. Achieving the third and fourth tiers costs far more than the first two, and often faces difficulties in scalability, multi-person work, and data verification and presentation. The combination of wiki and crawler tool can meet this requirement. In this paper, the data will be constructed based on the open source BookStack wiki platform. To fully present digitized content, it is not enough to rely on the original text alone, and third-party repositories are used for reference. However, cross-repository referencing has technical limitations, so designing crawlers to integrate cross-repository content is a way to overcome technical limitations. To use the crawler to search for data, it is necessary to design operation rules for it. The basic rule is to use breadth-first search, which is sufficient for a small amount of data created by users. The performance of rendering the binary relational graph of data is related to the complexity of search results, and rendering failure caused by exceeding the execution time will be a major challenge.