Tutorial on rigorous evaluation of clinical prediction models (BMJ)

Discussions
, ,
1

Very thorough primer just published in the BMJ for proper evaluation of clinical prediction models (part 1 of 3):

With the data at our disposal in Aotearoa I suggest focusing particularly on understanding internal/internal-external validation and on how performance measures can be misleading for individual subgroups of interest (e.g. indigenous populations).

3 Likes
2

Very cool - @robyn.whittaker @CkJin we should have a look at this.

Jon

1 Like
3

Thanks Jon.
Feel free to reach out if you want a head start on how to run an evaluation project. Because it’s the backbone of my thesis, I already have most of this set up in R. Happy to collaborate.

4

Thanks for sharing. Nice introduction to model evaluation and introduces an important topic, although there is a lot of detail under the hood, which is modelling technique-specific. Certainly, going forward, when NAIAEAG or equivalent evaluate any predictive or explanatory model, we must obtain an overt statement of the detailed methods used in the model development and the performance metrics obtained on test data and independent data sets. This is the 101 stuff of model building and is inherent to building a model. This should not surprise anyone, and the results should be freely available.

This is exactly analogous to how we would think about a new drug. Pharmac or MedSafe would not proceed with an application for a new drug without the vendor supplying the safety and effectiveness data.

One has to take an adverse inference if the vendor claims the data doesn’t exist or they don’t want to share.

Small rant over.

2 Likes
5

Agreed. If a prediction model is to be used to decide whether to prescribe medication, it should be at the very least held to a standard commensurate to the risk of the drug.

1 Like
6

Part 2/3: how to undertake an external validation study

1 Like
7

And finally, 3/3: sample size calculation

1 Like