預測模型效能之評估:預測曲線及其幾何摘要

Abstract

預測模型是臨床醫學與公共衛生中不可或缺的重要工具，有助於疾病風險的估計與決策的支持。傳統的評估方法，例如接收者操作特徵曲線及其下面積，主要評估模型的區分能力，但在評估族群層級的風險分層方面提供的資訊有限。最初為了評估生物標記而提出的預測曲線，能以視覺方式呈現預測風險在整體族群中的分布情形，但在多變量預測模型的應用中仍較少被探討。 本研究將預測曲線的方法延伸至多變量風險預測模型的評估中，系統性地探討其幾何特性，並導出三個互補的效能指標：Pietra指標、Gini指標，以及標準化的 Brier 分數。這三個指標分別量化模型在解析中間風險個案、區分預測風險層級，以及提高預測確定性方面的能力。為了確保預測風險的校正與不偏性，本研究提出了一個三步驟流程，包括交叉驗證、保序迴歸校正，以及自助法平均。 透過說明性範例及一項涉及台灣 23,839 名肺癌患者的真實應用案例，研究展示並驗證了此方法的可行性。Pietra指標、Gini指標，以及標準化的 Brier 分數分別捕捉到風險分層表現的不同面向。即使某些模型在其中一項指標上表現相同，其他指標仍可能呈現顯著差異，顯示這些指標具有互補性。在肺癌個案研究中，最終調整後的預測模型能有效區分死亡風險，將 25.1% 的患者歸類為極低風險（<10% 死亡率），50.1% 為高風險（>75% 死亡率），僅有 5.1% 的患者落在接近平均風險的「灰色區域」。該調整後模型的Pietra指標為 0.6719，Gini指標為 0.7850，標準化的 Brier 分數為 0.5186。不同細胞類型的肺癌 (肺腺癌、鱗狀細胞癌、小細胞癌和大細胞癌) 在預測表現上差異顯著，反映出不同細胞類型的肺癌在風險分層能力上的差異。 預測曲線及其幾何效能指標提供一種強大、透明且以族群為導向的框架，用以超越傳統指標來評估多變量風險預測模型的表現。這些方法能清楚地展現模型如何進行風險分層及其影響到的族群比例，進而提升模型可解釋性，協助臨床醫師與研究人員更有效地優化與應用預測模型於臨床與公共衛生實務中。

Prediction models are essential tools in clinical medicine and public health, facilitating disease risk estimation and supporting decision-making. Traditional evaluation methods such as the receiver operating characteristic (ROC) curve and its area under the curve (AUC) primarily assess discrimination but provide limited insight into population-level risk stratification. The predictiveness curve, originally proposed for biomarker evaluation, visually illustrates how predicted risks distribute across a population but has been underexplored in the context of multivariable prediction models. This study extends the predictiveness curve methodology to evaluate multivariable risk prediction models, systematically exploring its geometric properties and deriving three complementary performance indices: the Pietra index, Gini index, and scaled Brier score. These indices respectively quantify a model’s ability to resolve intermediate-risk cases, achieve separation among predicted risks, and enhance prediction certainty. A three-step procedure—cross-validation, isotonic regression calibration, and bootstrap averaging—was proposed to ensure calibration and unbiasedness of predicted risks. Illustrative examples and a real-world application involving 23,839 lung cancer patients from Taiwan were used to demonstrate and validate the methodology. The Pietra index, Gini index, and scaled Brier score captured distinct dimensions of risk stratification performance. Models with identical values of one index could differ markedly in the other indices, underscoring their complementary nature. In the lung cancer case study, the final adjusted prediction model effectively stratified patients across the fatality risk spectrum, identifying 25.1% of patients as very low-risk (<10% fatality) and 50.1% as high-risk (>75% fatality), with only 5.1% of patients falling within a “gray zone” near the average fatality risk. The adjusted model achieved a Pietra index of 0.6719, a Gini index of 0.7850, and a scaled Brier score of 0.5186. Substantial variation in predictive performance was observed among adenocarcinoma, squamous cell carcinoma, small cell carcinoma, and large cell carcinoma subtypes, reflecting differential risk stratification capabilities by cell type. The predictiveness curve and its geometric summaries—the Pietra index, Gini index, and scaled Brier score—provide a powerful, transparent, and population-oriented framework for evaluating the performance of multivariable risk prediction models beyond traditional metrics. By clearly illustrating how a model stratifies risk and for which proportion of the population, these methods enhance interpretability, supporting clinicians and researchers in refining and applying predictive models effectively in clinical and public health practice.