畢業後一般醫學訓練住院醫師「客觀結構式臨床技能測驗」的可行性評估

Abstract

背景 過去客觀結構式臨床技能測驗在醫學生跟幾個畢業後專科的評估都被發現具有信度及效度。但是有學者對於它的可行性以及是否適合評估核心能力導向醫學教育有疑慮。 過去對於跨專科客觀結構式臨床技能測驗在畢業後醫學教育的資料有限。 台灣有獨特的畢業後一般醫學訓練及(Postgraduate year)PGY住院醫師制度。 過去針對這個族群使用客觀結構式臨床技能測驗的信度及效度證據也有限。 目標跟研究問題 1. 評估跨專科客觀結構式臨床技能測驗在畢業後一般醫學訓練住院醫師的可行性 2. 檢視客觀結構式臨床技能測驗在這個族群及情境下的信度及效度 3. 探討可能影響這個測驗效度的因素 方法 在2016年6月以及2017年6月，總共有83位PGY學員參加亞東紀念醫院的期末客觀結構式臨床技能測驗。 我們報告了測驗的設計、藍圖、考官訓練、標主化病患訓練、成績、問卷調等為測業技術面可行性探討。在經濟可行性方面我們估計了一次測驗的費用。 信度及效度評估包含了站與站之間的內部一致性、單一站項目的內部一致性、評估者間信度檢驗、分數跟及格率、項目分數跟整體評估的關聯性等。 我們評估站數、考官訓練、多位考官對於信度的影響。 結果 即使客觀結構式臨床技能測驗需要相當多的資源，這是一個可行的測驗，也得到教職跟學員正向的反應。 站與站之間的信度(Cronbach’s α 0.104~0.464)跟項目與項目之間的信度(Cronbach’s α -0.217 to 0.483)偏低。 大致上評估者間的信度為中等到好的信度，項目分數跟整體評估的關聯性也大致都是高的。 使用多位評估者讓站與站之間的信度增加。 三位評估者的評估者間信度比兩位評估者來的高。 討論 我們提供了單一家教學醫院的跨專科客觀結構式臨床技能測驗在畢業後一般醫學訓練住院醫師的可行性證據。 但對於不同規模的訓練醫院，同樣的評估方式不一定有同樣的可行性。在亞東紀念醫院，透過醫院的教學部以及教職的支持，這樣的評估技術上以及經濟上都是可行的。 同時也需要不同專科密切的配合。 站與站之間跟項目與項目之間的信度偏低。 一個可能的解釋是我們用的是因為在跨專科的考試中評估了不同面向以及長站的設計在單一站中評估了不一樣的核心能力跟臨床技能。 我們發現使用多位評估者的分數讓站與站間的信度改站。在文獻回顧中，其他可能影響站與站之間跟項目與項目之間的信度的因素還包含站數，評估項目數，跟評核的臨床技能。 在我們的測驗中，評估者間信度整題而言是高的。 三位評估者的評估者間信度高於兩位評估者。 整題而言，項目分數跟整體評估的關聯性、高的評估者間信度以及內容、回應、內部架構等提供這個評估的信度以及效度證據。 結論 在台灣的畢業後一般醫學訓練，以跨專科客觀結構式臨床技能測驗作為評估模式是技術上以及經濟上可行的。 這個測驗可以評估多項核心能力跟重要的臨床技能。 信度可能受不同因素的影響，使用兩位評估者是一個有效率增加信度的方式。

Background The objective structured clinical examination (OSCE) is known as a reliable and valid assessment of both undergraduate medical students and in several post-graduate medical specialties. However, there are concerns regarding its feasibility and whether it is suitable as an assessment of competence based medical education (CBME). There is limited data on the use of multi-specialty OSCE for assessment in postgraduate medicine. Taiwan also has an unique postgraduate year (PGY) general medicine training program and there is limited reliability or validity evidence of OSCE in this population. Aim and Research Questions 1. Evaluate the feasibility of multi-specialty OSCE in the post-graduate year (PGY) general medicine training program 2. Examine the reliability and validity of multi-specialty OSCE in this population and setting 3. Identify potential factors that affect reliability of this assessment Concise Methods During June 2016 and June 2017, 83 PGY residents participated in four seperate end-of-year OSCE assessment at Far Eastern Memorial Hospital (FEMH). The design, blueprint, faculty and standardized patient training, outcomes, as well as questionnaire responses were reported as evidence of technical feasibility. Economically feasibility was evaluated through estimating the cost of applying an OSCE exam. Reliability and validity evidence was gathered through analysis of across-station, across-item, inter-rater reliability, as well as scores and correlations parameters. The effects of rater training and different station lengths and having multiple raters were examined. Results OSCE was a feasible but resource demanding method of assessment with positive response and satisfaction from faculty and trainees. The across-station station reliability (Cronbach’s α 0.104~0.464) and across-item reliability (Cronbach’s α -0.217 to 0.483) were low. Overall good correlation between checklist items with global rating (Coefficient of determination of R2 0.32~0.907) and moderate to good inter-rater reliability was found. Using the scores of multiple raters improved across-station reliability and inter-rater of three raters was higher than two raters. Discussion Feasibility of multi-specialty OSCE in PGY residents of the general medicine training program in one hospital was provided. However, feasibility using the same format may not apply to all hospitals, depending on the size of each program. It was technically and economically feasible at FEMH due to the strong support of the hospital’s medical education department and faculty and also through the close collaboration between the different specialties involved. Across-station and across-item reliability were in general low, and a potential explanation is evaluation of distinct constructs due to the multi-specialty design and measure of multiple clinical skills and competence in long stations. We found that having multiple raters improved across-station reliability. Based on previous literature, other potential factors that may affect across-station and across-item reliability include number of stations, length of checklist and clinical skill tested. Overall inter-rater reliability was good and three raters compared to two raters in general improved inter-rater reliability. Overall correlation between checklist and global rating, inter-rater reliability, validity in the form of content, response process, internal structure and criterion validity provided evidence towards overall fair reliability and validity of this assessment. Conclusion A multi-specialty OSCE as an end-of-year summative and formative assessment in Taiwanese PGY residents general medicine training program is technically and also economically feasible. It can be used to assess multiple core competencies and important clinical skills. Reliability may be affected by various factors, and the use of double raters is an effective way to increase reliability.