人工智慧模型回答污水處理問題之比較評估

Abstract

大型語言模型越來越多地應用於教育和技術資訊領域，但其在專業工程領域的可靠性仍需評估。本研究使用45道涵蓋主要處理製程的標準化廢水處理問題，對ChatGPT、Claude和Gemini三種人工智慧模型進行了比較。十位評估員使用三分量表，從準確性、相關性和清晰度三個面向對答案進行評估。評估結果採用描述性統計和單因子重複測量變異數分析進行分析，以確定三種人工智慧模型之間是否有顯著差異。此外，也比較了高階評估員組和學生評估員組的評估結果。ChatGPT獲得了最高的總體平均分數2.8237，其次是Claude和Gemini，二者的平均分數均為2.8193，三個模型的總體平均分為2.8207。然而，重複測量變異數分析結果顯示，三種模型之間沒有統計上的顯著差異，F(2, 88) = 0.0195，p = 0 .981。模型表現因廢水處理主題而異，其中固體處理和處置的總體平均得分最高，為 2.8956，而厭氧處理的得分最低，為 2.7356。高階評估組和學生評估組之間的比較也未顯示出統計上的顯著差異，t (44) = −1.708，p = 0.150。總體而言，研究結果表明，ChatGPT、Claude 和 Gemini 在回答本研究中使用的標準化廢水處理問題方面表現出相當的性能。這些模型可以作為廢水處理教育和資訊支援的輔助工具；但是，其回答應使用權威的技術資料和專業判斷進行驗證。

Large language models are increasingly used for education and technical information, but their reliability in specialized engineering fields requires evaluation. This study compared ChatGPT, Claude, and Gemini using 45 standardized wastewater treatment questions covering major treatment processes. Responses were evaluated by ten evaluators using accuracy, relevance, and clarity on a three-point scale. The evaluation results were analyzed using descriptive statistics and a one-way repeated measures analysis of variance to determine whether significant differences existed among the three AI models. The evaluation results were also compared between advanced and student evaluator groups. ChatGPT achieved the highest overall mean score of 2.8237, followed by Claude and Gemini, both with a mean of 2.8193, giving an overall mean of 2.8207. However, the repeated measures ANOVA showed no statistically significant difference among the three models, F (2, 88) = 0.0195, p = 0.981. Model performance varied across individual wastewater treatment topics, with solids processing and disposal receiving the highest overall topic mean of 2.8956, while anaerobic treatment received the lowest at 2.7356. The comparison between advanced and student evaluator groups also showed no statistically significant difference, t (44) = −1.708, p = 0.150. Overall, the findings indicate that ChatGPT, Claude, and Gemini demonstrated comparable performance in answering the standardized wastewater treatment questions used in this study. The models may serve as supplementary tools for wastewater treatment education and information support; however, their responses should be verified using authoritative technical sources and professional judgment.