請用此 Handle URI 來引用此文件:
http://tdr.lib.ntu.edu.tw/jspui/handle/123456789/66950
標題: | 用於群眾外包的序列性工作者指派和批量資料標記問題 Sequential Worker Assignment and Batch Labeling for Crowdsourced Classification |
作者: | Sung-Wen Lan 藍崧文 |
指導教授: | 王奕翔(I-Hsiang Wang) |
關鍵字: | 群眾外包,主動假設檢定,線上感測器選擇,最佳停止問題, Crowdsourcing,Active Hypothesis Testing,Online Sensor Selection,Optimal Stopping, |
出版年 : | 2020 |
學位: | 碩士 |
摘要: | 已發展多年的群眾外包系統廣泛用於搜集標記資料,藉由整合多位工作者的標記而得到可靠的結果。現有研究中,多數針對在工人可靠度不同的情況下,研究如何整合標記。然而標記時間、標記預算和標記正確率之間的關係並未被關注。針對這三者之間的關係,我們探討工作者指派以及批量資料標記問題。在問題設定中,工人會接續到來,而決策者可以動態地指派工人標記資料。第一個問題裡,工人可靠度的異質性和工人動態的機率分佈能被決策者所知。在時間和標記預算上我們考慮寬鬆和嚴格兩種限制。在寬鬆預算時間限制下,我們分析理論極限的錯誤指數。在嚴格限制下,我們證明錯誤指數的上界,並提出一動態決策。程式模擬結果顯示,理論極限和我們動態決策表現差距不大。此一動態決策使用的動態資訊中只需要剩餘標記與時間比的資訊。第二個問題裡,透過迭代法我們可以最小化錯誤率代價、標記數成本和時間成本的總和。其中最佳決策會出現對已停止標記的個別資料可能重啟標記的現象。我們也比較對各別資料使用常見的逐次機率比檢定和固定標記次數的檢定,二者的表現差異。 Crowdsourcing has been developed and commonly used for collecting labels of a batch of data, where reliable labels are inferred from the aggregation of responses of multiple workers. Most studies in the literature are focused on inferring labels from response of workers with different level of reliability, while the trade-off among labeling time, labeling cost and accuracy of inference has been overlooked. To understand the fundamental trade-off, we explore worker assignment and batch labeling problems in an online setting, where workers arrive sequentially and decisions are made in a real-time fashion. In the first problem, the heterogeneous reliability and online dynamics of workers are available by the decision-making policy. We address the problem with soft constraints and hard constraints respectively. With soft constraints, the optimal error exponent is characterized. With hard constraints, an upper bound on the Bayesian error exponent is derived, which is piece-wise linear with respect to the budget-to-time ratio. Inspired by the upper bound, we propose an adaptive policy, which only needs to maintain the information of remain-budget-to-remain-time ratio instead of the whole history. Numerical results show that the gap between the upper bound and the proposed policy is small. In the second problem, we formulate an optimization problem to jointly consider the penalty of error, the cost of labels, and the penalty of time of the batch of tasks. It is shown that the optimal policy can be derived by the method of value iteration. Interestingly, in most cases, the optimal policy requires some tasks restart labeling even when they stopped to collect labels at some point. We then compare the performance of fixed-length (fixed-label) tests with a sub-optimal policy consisting of independent sequential probability ratio test for each labeling task. This sub-optimal policy outperforms fixed-length ones when the order of temporal penalty small enough with respect to the order of the number of tasks. |
URI: | http://tdr.lib.ntu.edu.tw/jspui/handle/123456789/66950 |
DOI: | 10.6342/NTU202003625 |
全文授權: | 有償授權 |
顯示於系所單位: | 電信工程學研究所 |
文件中的檔案:
檔案 | 大小 | 格式 | |
---|---|---|---|
U0001-1608202022391300.pdf 目前未授權公開取用 | 1.73 MB | Adobe PDF |
系統中的文件,除了特別指名其著作權條款之外,均受到著作權保護,並且保留所有的權利。