同儕互評中之評分標準品質指數

Abstract

同儕互評(Peer Assessment)在大規模開放式線上課程(Massive Open Online Course, MOOC)中，因為能夠批改大量作業並具有高可靠度等優點，逐漸成為批改大量開放式(Open-ended)作業的主要方法之一。在同儕互評的過程中，評分標準(Rubric)的品質對於互評的可靠度與成效有相當大的影響，然而過去並沒有研究致力於量化地評估評分標準的品質。為了能夠量化地評估評分標準的品質，本研究提出評分標準品質指數(Rubric Quality Index)能夠透過互評的結果計算出評分標準的品質。評分標準品質指數是應用統計模型量化地描述同儕互評中的特徵後，再將這些特徵組合而成。本研究應用Coursera工程圖學2D CAD課程中6次不同的互評資料，一共11,725筆互評分數與2,628筆助教批改的分數，測試評分標準品質指數的效果，結果顯示評分標準品質指數對評分標準的品質具有良好的解釋能力，而且評分標準品質指數的計算流程不需要助教批改全部作業就可以完成。應用評分標準品質指數，能夠在教學助教批改部分作業後監測並比較MOOC中互評的評分標準品質。

In massive open online courses (MOOCs), peer assessment is considered as one of the solutions for assessing open ended assignments at scale. In the process of peer assessment, the quality of scoring rubrics, a scoring guidance for evaluating the quality of submissions, affects the validity and reliability. While well-designed rubrics can enhance the effectiveness of peer assessment, poorly-designed rubrics lead to inaccuracies in peer assessment. However, few previous studies focused on quantitatively evaluating the quality of rubrics. In this research, we developed a rubric quality index to estimate the quality of rubrics from the results of peer grading. We applied a statistical model for quantitatively describing peer assessments' features, such as grader biases and reliabilities. Then, we combined these features into one single index called rubric quality index (RQI). Datasets from six different assignments in Coursera's Engineering Graphics 2D CAD course with 11,725 peer grades and 2,628 staff grades were used to validate the RQI. The results showed that the RQI can effectively evaluate the quality of rubrics with feasible workload for teaching staff in our validation. This RQI can be used for monitoring and comparing the quality of rubrics in MOOCs peer assessment process after staff assess partial submissions.