在語境學習中研究各種提示以改善Text-to-SQL任務分解

Abstract

在面臨大量的查詢資料表需求時，減少人工撰寫SQL指令的工作量成為一個重要課題。因此，Text-to-SQL任務逐漸受到學術界及業界的關注。起初，解決Text-to-SQL任務的方法主要是使用seq2seq模型；然而，隨著大型語言模型的迅速發展，近年來一些研究人員開始在大型語言模型進行語境學習，以評估其性能。日後，思維鏈（chain-of-thought, COT）技巧的有效性也受到廣泛關注，促使各領域嘗試將其應用於觀念分解及推理過程，其中DIN-SQL是個典型的例子。然而，即使它在Spider資料集執行準確率（Execution Accurancy, EX）排行榜上表現出色，但在模式連結的處理上仍有不足之處。 本研究主要目的是針對DIN-SQL模型進行優化。首先進行錯誤分析，探討其預測失敗的原因，並提出改善性能的提示（prompt）方法。我們使用GPT-3.5模型進行測試，並提出了三種改善方向，分別是針對模式連結、分解與分類、與自我修正模組進行改善。實驗結果表示，這三種改善方向都成功提升模型性能。其中，針對模式連結的改善方式──將RESDSQL交叉編碼器與DINSQL自我修正模組相結合──被證明是最佳的改善方法，與基線模型相比，這種改進方法可以提高3.82%的性能，應用GPT-3.5與GPT-4模型下，分別達到74.1%與79.79%的準確率。

When faced with a large number of query data tables, reducing the workload of manually writing SQL commands has become an important topic. Therefore, the text-to-SQL task has gradually attracted the attention of academia and industry. At first, the method to solve the text-to-SQL task was mainly to use the seq2seq model; however, with the rapid development of large-scale language models (LLMs), some researchers have begun to perform in-context learning on LLMs to evaluate their performance in recent years. After that, the effectiveness of the chain-of-thought (COT) technique has also received widespread attention. It makes more researchers from various fields to try to apply it to the concept decomposition and reasoning process, of which DIN-SQL is a typical example. However, even though it performed well on the Execution Accuracy (EX) leaderboard of the Spider dataset, it still has shortcomings in the processing of schema linking. The main purpose of this study is to optimize the DIN-SQL model. Firstly, error analysis is carried out to explore the reasons for its prediction failure, and a prompt method to improve performance is proposed. I used the GPT-3.5 model for testing, and proposed three improvement directions, namely for schema linking, decomposition and classification, and self-correction modules for improvement. The experimental results show that all three improvement directions have successfully improved the model performance. Among them, the improvement method for schema linking, combining the RESDSQL cross-encoder with the DINSQL self-correction module proved to be the best improvement method. This method can improve performance by 3.82% compared with the baseline model. When using the GPT-3.5 and GPT-4 models, the EX accuracies were 74.1% and 79.79%, respectively.