event-icon

Workshop - Instructional

W21: Caught between a ROC and a Hard Bayes

8:00 AM–11:30 AM Nov 17, 2019 (US - Eastern)

Kalorama

Description

In an ideal word, all diagnostic tests would make perfect predictions, all therapies would be completely effective and harmless, and resources would be limitless. However, even with ongoing innovation in machine learning algorithms that underlie the current era of precision medicine, outcomes prediction remains imperfect, therapies are not completely effective and resources remain limited. Therefore, the output of these algorithms must be placed in a realistic clinical context to support truly effective clinical decision-making. Through incorporation of prior evidence-based knowledge of clinical valuation of outcomes and decision points, Bayesian analysis plays a key role in translating results of machine learning algorithms into actionable information. Bayesian analysis is essential for understanding the degree of sensitivity and specificity that a new diagnostic test or predictive algorithm needs to have in the context of the effectiveness and the expense of existing therapies to impact clinical decision-making.

Related to Bayesian analysis, Receiver Operating Characteristic (ROC) analysis is used to illustrate the tradeoff between a test’s sensitivity and specificity, where a better test is often considered to have a greater area under the curve (AUC), equivalent to the “c-statistic.” While most analytic software can calculate the c-statistic, the translation of ROC findings into clinical decision-making rules is not always straightforward given different impact of false negative and false positive results. A calculation of a partial area under the ROC curve informed by expert clinical judgments may provide a better real-world assessment of the value of a predictive algorithm. Furthermore, many machine learning algorithms calculate a single threshold, above which a predicted outcome is likely to occur, and below which, the predicted outcome is not likely to occur. Formal ROC analysis enables a more nuanced view of the predicted outcome by supporting the calculation of stratum-specific likelihood ratios where different ranges of results of predictive algorithms have different strengths of associations with the outcome.

This workshop will help attendees understand and apply Bayesian and decision-analytic fundamentals, as well as derive and interpret ROC curves and stratum specific likelihood ratios. Real-world application of these techniques will be discussed, focusing on use cases derived from predictive analytics literature.

Learning Objective: To articulate the role of Bayesian and ROC analysis in supporting clinical decision-making in conjunction with the outcome of machine learning algorithms
To apply Bayesian principles in modeling expected outcomes of large scale clinical interventions
To interpret reports from Bayesian and ROC analyses in support of clinical decision-making
To answer successfully questions on the Clinical Informatics Boards

Authors:

Mark Weiner (Presenter)
Lewis Katz School of Medicine at Temple University

Harold Lehmann (Presenter)
Johns Hopkins School of Medicine

Presentation Materials:

Tags