拆解上下文學習：透過函數混合式訓練探討其內在機制與分布外泛化能力

Abstract

當代基於 Transformer 的語言模型在各類真實任務中展現出卓越的表現，但其內部運作機制仍未被完全理解。近期研究逐漸聚焦於「上下文學習」（in-context learning, ICL）現象，以及模型超越訓練分布進行泛化的能力。然而，多數研究是在簡化條件下進行的，訓練與評估皆以單一、明確定義的函數生成提示（Prompt），這使得模型在結構更為多樣或模糊的情境下的表現仍不明朗。

本研究探討 ICL 透過「混合訓練」（Blended Training）後的行為，其中每個訓練提示由多個不同類型函數隨機抽樣產生，且不提供任何明確的任務標記或結構線索。我們以標準的 ICL 任務（如線性分類或二次分類... 等）為基礎，透過自行設計的實驗來驗證假說，並評估此訓練方式對模型行為、抗噪性與泛化能力的影響。

實驗結果顯示，在混合訓練情境下，模型並不是用單一函數為主軸進行函數選擇，而是展現出更具彈性的模式識別能力、對輸入雜訊的更強韌容忍度，以及更佳的異常情境泛化能力。這些發現指出，訓練中引入結構多樣性的提示，有助於提升模型在未知環境下的適應性。

Transformer-based language models have achieved remarkable success across a wide range of real-world tasks, yet the internal mechanisms that govern their behavior remain only partially understood. Recent research has increasingly focused on the phenomenon of in-context learning (ICL) and its ability to generalize beyond the training distribution. However, many of these studies are conducted under simplified conditions, where both training and evaluation use prompts derived from a single, clearly defined function. As a result, it remains unclear how models behave in more structurally diverse or ambiguous settings.

In this study, we examine ICL under a blended training paradigm, in which each training prompt contains examples sampled from multiple function classes, without any explicit task identifiers or structural signals. Using standard ICL benchmarks such as linear and quadratic classification, we assess how this training approach influences model behavior, robustness, and generalization.

Our findings indicate that under blended training, the commonly observed function selection behavior, where the model implicitly identifies and applies a single underlying function, plays a less central role. Instead, the model demonstrates more flexible pattern recognition, improved resilience to input noise, and stronger generalization to out-of-distribution tasks. These results suggest that training on structurally mixed prompts can enhance a model’s adaptability in unfamiliar scenarios.