在注意力模型之上有效利用輔助資訊

Abstract

本論文主要探討如何讓深度學習模型更有效地利用「輔助資訊」，並解決三個關鍵問題。輔助資訊是機器學習可以利用的額外數據，例如天氣、地理位置、表格資料或文字描述，這些資訊可以幫助模型做出更精確及可解釋的預測。然而，使用注意力機制處理這些輔助資訊時，模型會面臨諸多挑戰。

注意力機制是一種常見的深度學習技術，讓模型能夠專注於輸入數據中最相關的部分，並廣泛應用於自然語言處理和其他任務中。儘管注意力模型在處理文字數據上表現優異，當面對不同形式的數據或大量輔助資訊時，還是會遇到困難。本論文針對以下三個挑戰提出了解決方案：

1. 數據模態不兼容：在像時間序列預測這樣的任務中，模型需要處理時間及表格數據，而注意力機制對於這種數據的效果不如處理文字數據那麼好。我們提出了TSMixer架構，通過多層感知機（MLP）來更有效地捕捉時間上的變化，並結合輔助資訊來提高預測精度。 2. 輔助資訊過長：在處理像表格問答這類任務時，表格數據可能非常龐大，遠超出模型能處理的上下文範圍。我們開發了 TableRAG 檢索系統，以「格」為單位在語言模型處理前先篩選出最相關的內容，幫助模型在面對大量數據時更高效地回答正確答案。 3. 泛化能力下降：當模型針對特定任務進行微調時，往往會過度專注於訓練資料，而失去對未知資料的預測能力，這在零樣本多標籤文本分類中尤其明顯。我們提出了「單向微調」框架，透過凍結標籤編碼器，保留標籤語義的豐富性，同時只微調文件編碼器，從而保持模型的零樣本預測能力。

本論文透過這些方法，展示了如何解決這些挑戰，提升注意力機制在整合輔助資訊時的效能，幫助模型在多種任務中更準確地運作。

This dissertation explores how to make deep learning models more effectively utilize auxiliary information and addresses three key challenges. Auxiliary information refers to additional data that machine learning models can leverage, such as weather, geographical location, tabular data, or text descriptions, which help improve prediction accuracy and interoperability. However, there are several challenges when using attention mechanisms to process auxiliary information.

The attention mechanism is a common deep learning technique that enables models to focus on the most relevant parts of the input data; it is widely used in natural language processing and other tasks. Although attention models perform exceptionally well with textual data, they encounter difficulties when dealing with different data types or large amounts of auxiliary information. This dissertation proposes solutions for the following three challenges:

1. Modality incompatibility: In tasks like time series forecasting, models need to process time and tabular data, but attention mechanisms are not as effective in handling these types of data as they are with text. We propose the TSMixer architecture, which uses multi-layer perceptrons (MLPs) to better capture temporal patterns and integrate auxiliary information to improve prediction accuracy. 2. Excessive length of auxiliary information: In tasks like table-based question answering, the amount of tabular data can be enormous, exceeding the model's context window. We develop the TableRAG retrieval system, which filters the most relevant information at the cell level before it is processed by the language model, allowing the model to answer questions more efficiently when dealing with large datasets. 3. Loss of generalizability: When models are fine-tuned for specific tasks, they often become too focused on the training data, losing their ability to predict unseen data, particularly in zero-shot multi-label text classification. We propose a one-sided fine-tuning framework, which freezes the label encoder to retain semantic richness while only fine-tuning the document encoder, preserving the model's zero-shot prediction capabilities.

This dissertation demonstrates how these solutions address the challenges, enhancing the performance of attention mechanisms in effectively integrating auxiliary information, and enabling models to perform more accurately across various tasks.