以AE或Beta-VAE外加特徵方式提升BERT語言模型的內嵌訊息

Abstract

以 BERT 為代表的子詞語言模型，在自然語言理解任務中表現優異，但在面對需要字符級推理的任務時（如字符計數或字串比對），往往無法精確捕捉細微的字符級特徵。相比之下，字符級語言模型能夠細緻地建構這類訊息，但訓練字符級語言模型會帶來極高的運算負擔。本論文提出一套輕量化方法，將字符級視覺特徵融合至子詞單位的向量表示中。我們將每個子詞轉換為字形序列，並透過自編碼器 (AE) 與 beta-變分自編碼器 (beta-VAE) 進行視覺特徵的壓縮與表徵學習。這些特徵經由線性映射或多層感知器 (MLP) 映射至 BERT 的嵌入空間，並與原始子詞向量結合，僅需修改輸入層即可完成整合，無須調整原始模型架構。我們將此方法應用於一項字符數預測任務，並以遮罩語言模型 (Masked Language Modeling, MLM) 的形式進行訓練與評估。實驗涵蓋 460 萬筆樣本，結果顯示加入視覺特徵後，模型表現皆優於基準模型。透過 beta-變分自編碼器所取得之特徵在僅使用線性投影的情況下即可展現明顯效果，而透過自編碼器取得之特徵則需搭配多層感知器才能達到最佳表現。實驗結果驗證了將額外的視覺特徵注入子詞向量，能有效提升模型的字符推理能力，彌補子詞建模與字符細節之間的落差。本方法使模型能以極小的計算成本，學習到字符層級的特徵表現，適用於如計數、比對等需精細字元辨識的任務。

Subword-based language models like BERT achieve strong performance in natural language understanding but often miss fine-grained character-level cues essential for symbolic reasoning tasks like counting or string matching. While character-level representations can capture these subtle patterns more effectively, incorporating them directly into large models poses significant computational challenges. This thesis presents a lightweight framework to enrich subword token embeddings with character-level visual features. We render each token as a glyph sequence and train autoencoders or beta-VAEs to learn compact visual representations. These features are projected into BERT’s embedding space via linear or MLP layers and added to the original embeddings, requiring no architectural changes beyond the input layer. We evaluate this integration on a character-counting task framed as masked language modeling. Experiments on 4.6 million examples show consistent improvements over the baseline. beta-VAE-based features are effective even with linear projections, while AE-based features benefit from non-linear mappings. Our findings indicate that augmenting subword embeddings with additional visual features significantly improves symbolic reasoning abilities, bridging the gap between subword efficiency and character-level precision. This approach enables models to capture fine-grained character patterns crucial for tasks like counting, without incurring the computational costs of full character-level architectures.