DefectVLM：基於 Vicuna 的晶圓圖缺陷理解視覺語言模型

Abstract

晶圓圖缺陷分析對於半導體良率提升與生產可靠性是關鍵的，然而，先前 方法大多只能產生粗略的分類標籤，所以有解釋性的限制。為了克服這個侷 限，我們提出DefectVLM，一個專為工業晶圓圖理解而設計的視覺語言模型。DefectVLM 整合缺陷定位與visioninstruction prompt tuning，使模型能夠準確預測 缺陷類型，同時產生具有空間依據的自然語言解釋，提供給工程診斷。此模型採 用兩階段訓練策略：第一階段先對齊缺陷類型的視覺與文字表徵，第二階段再將 定位線索整合至vision–language instruction tuning 中。令外，我們擴充現有資料 集，加入缺陷類型描述，並建立晶圓圖VQAbenchmark，以評估更完整的分析。 根據實驗結果，DefectVLM在晶圓缺陷偵測benchmark上達到優於先前方法的表 現。此研究可當作未來晶圓缺陷分析中多模態方法發展的基礎。此外，從工業應 用角度來看，DefectVLM有助於實現智慧化良率監控與製程自動化。

Wafer map defect analysis is critical for semiconductor yield improvement and production reliability, yet existing approaches typically produce coarse labels with limited interpretability. To overcome this gap, we introduce DefectVLM, a vision–language model (VLM) tailored for industrial wafer map understanding. DefectVLM integrates defect localization with vision instruction prompt tuning, enabling accurate defect-type prediction alongside natural-language, spatially grounded explanations that better support engineer ing diagnosis. The model employs a two-stage training strategy: first aligning visual and textual representations of defect types, then incorporating localization cues into vision language instruction tuning. We further extend existing datasets with defect-type descriptions and introduce wafer map VQA benchmarks for fine-grained evaluation. Experiments show that DefectVLM achieves state-of-the-art performance on defect detection benchmarks. Our work provides a foundation for future multimodal research in wafer defect analysis. Besides, from an industrial perspective, DefectVLM facilitates intelligent yield monitoring and process automation.