學習語言模式而非分佈：以鏈式思考在語言模型中透過歸納偏置促進泛化

Abstract

大型語言模型（LLMs）常常能夠泛化至超出訓練分布的資料，然而其歸納偏置（inductive bias）如何受到資料的影響仍未被充分理解。本研究從理論觀點出發，探討自迴歸模型如何根據特定資料集建構其歸納偏置。我們提出「最小上下文假說」（Minimal Context Hypothesis），主張語言模型在進行下一詞預測時，可透過辨識一組最小前綴詞（minimal contexts）來進行建模；這些上下文經由數學結構（part-of-math, POM）所誘導出的語言模式定義，能夠使模型在僅依賴這些最小上下文的情況下，忽略其餘前綴資訊而達成準確預測。為了形式化此概念，我們提出「自迴歸最小上下文搜尋機器」（Autoregressive Minimal Context Searching Machine, AMCSM），一種建立於 POM 結構之上的抽象框架，描述自迴歸模型如何在資料分布中隱式搜尋滿足條件獨立性的最小上下文集合。此理論框架下，最小上下文可視為根據資料分布導出的最小前綴子集，並反映語言模型對歸納偏置的建構方式。我們亦設計一系列合成乘法任務進行實驗，證實鏈式思考（chain-of-thought, CoT）監督可穩定此基於最小上下文的歸納偏置，進而促進模型對訓練分布之外資料的泛化能力。進一步的顯著性分析亦顯示，在推理過程中，模型的敏感性主要集中於這些最小上下文，呼應我們的理論預測。

Large language models (LLMs) often generalize beyond their training distributions, yet how their inductive biases are shaped by data remains poorly understood. We study this question from a theoretical perspective by formalizing how an autoregressive model may determine its inductive bias from a given dataset. We propose the Minimal Context Hypothesis, which posits that next-token prediction in LLMs can be characterized by identifying small subsets of prefix tokens—minimal contexts—identified via a formally defined language pattern induced by part-of-math (POM) structures, such that conditioning on these minimal contexts alone renders the remaining prefix conditionally irrelevant. To formalize this idea, we introduce the Autoregressive Minimal Context Searching Machine (AMCSM), a conceptual abstraction grounded in POM structures that describes how an autoregressive learner may implicitly search for minimal conditioning sets determined by the data distribution. Within this framework, minimal contexts emerge as the smallest prefix subsets satisfying a conditional independence criterion for next-token prediction, thereby yielding a distribution-dependent characterization of inductive bias. We complement our theoretical formulation with controlled experiments on synthetic multiplication tasks, demonstrating that chain-of-thought (CoT) supervision stabilizes this minimal-context-based inductive bias and enables generalization to inputs outside the support of the training distribution. Saliency analyses further corroborate our theory by showing that model sensitivity concentrates on minimal contexts during intermediate reasoning steps.