Missing Data in Prediction Research: A Five-Step Approach for Multiple Imputation,  Illustrated in the CENTER-TBI Study.

Biomedical Research/methods/statistics & numerical data; Brain Injuries, Traumatic/diagnosis/epidemiology; Cohort Studies; Data Interpretation, Statistical; Databases, Factual/statistics & numerical data; Europe/epidemiology; Humans; Prognosis; Prospective Studies; imputation; missing data; prediction; traumatic brain injury; tutorial

Abstract :

[en] In medical research, missing data is common. In acute diseases, such as traumatic brain injury (TBI), even well-conducted prospective studies may suffer from missing data in baseline characteristics and outcomes. Statistical models may simply drop patients with any missing values, potentially leaving a selected subset of the original cohort. Imputation is widely accepted by methodologists as an appropriate way to deal with missing data. We aim to provide practical guidance on handling missing data for prediction modeling. We hereto propose a five-step approach, centered around single and multiple imputation: 1) explore the missing data patterns; 2) choose a method of imputation; 3) perform imputation; 4) assess diagnostics of the imputation; and 5) analyze the imputed data sets. We illustrate these five steps with the estimation and validation of the IMPACT (International Mission on Prognosis and Analysis of Clinical Trials in Traumatic Brain Injury) prognostic model in 1375 patients from the CENTER-TBI database, included in 53 centers across 17 countries, with moderate or severe TBI in the prospective European CENTER-TBI study. Future prediction modeling studies in acute diseases may benefit from following the suggested five steps for optimal statistical analysis and interpretation, after maximal effort has been made to minimize missing data.

Disciplines :

Human health sciences: Multidisciplinary, general & others

Author, co-author :

Gravesteijn, Benjamin Yaël; Department of Public Health, Erasmus Medical Center, Rotterdam, The Netherlands.

Sewalt, Charlie Aletta; Department of Public Health, Erasmus Medical Center, Rotterdam, The Netherlands.

Venema, Esmee; Department of Public Health, Erasmus Medical Center, Rotterdam, The Netherlands.

Nieboer, Daan; Department of Public Health, Erasmus Medical Center, Rotterdam, The Netherlands.

Steyerberg, Ewout W; Department of Public Health, Erasmus Medical Center, Rotterdam, The Netherlands. ; Department of Biomedical Data Sciences, Leiden University Medical Center, Leiden,

CENTER-TBI, Collaborators

Other collaborator :

Åkerlund, Cecilia

Amrein, Krisztina

Andelic, Nada

Andreassen, Lasse

Anke, Audny

Antoni, Anna

Audibert, Gérard

Azouvi, Philippe

Azzolini, Maria Luisa

Bartels, Ronald

More authors (230 more)

Language :

English

Title :

Missing Data in Prediction Research: A Five-Step Approach for Multiple Imputation, Illustrated in the CENTER-TBI Study.

Publication date :

01 June 2021

Journal title :

Journal of Neurotrauma

ISSN :

0897-7151

eISSN :

1557-9042

Volume :

Issue :

Pages :

1842-1857

Peer reviewed :

Peer Reviewed verified by ORBi

Available on ORBi :

since 27 March 2022

Statistics

Number of views

437 (3 by ULiège)

Number of downloads

1133 (3 by ULiège)

More statistics

Scopus citations^®

Scopus citations^®
without self-citations

OpenCitations

OpenAlex citations

Bibliography

van Buuren, S. (2018). Flexible Imputation of Missing Data. CRC: Boca Raton, FL.
Steyerberg, E.W. (2019). Clinical Prediction Models. Springer International: Cham, Switzerland.
Steyerberg, E., Moons, K.G.M., van der Windt, D., Hayden, J., Perel, P., Schroter, S., Riley, R., Hemingway, H., and Altman, R.B. (2013). Prognosis Research Strategy (PROGRESS) series 3: prognostic model research. PLoS Med. 10, e1001381.
Riley, R.D., Hayden, J.A., Steyerberg, E.W., Moons, K.G.M., Abrams, K., Kyzas, P.A., Malats, N., Briggs, A., Schroter, S., Altman, D.G., and Hemingway, H. (2013). Prognosis Research Strategy (PROGRESS) 2: prognostic factor research. PLoS Med. 10, e1001380.
Molenberghs, G., Kenward, M., and Ebrahim, G.J. (2007). Missing data in clinical studies. J. Trop. Pediatr. 53, 294-294.
Mislevy, R.J. (1991). Book reviews: statistical analysis with missing data. J. Educ. Stat. 16, 150.
Nielson, J.L., Cooper, S.R., Seabury, S.A., Luciani, D., Fabio, A., Temkin, N.R., and Ferguson, A.R.; TRACK-TBI Investigators. (2020). Statistical guidelines for handling missing data in traumatic brain injury clinical research. J. Neurotrauma. doi: 10.1089/neu.2019.6702.
Leisman, D.E., Harhay, M.O., Lederer, D.J., Abramson, M., Adjei, A.A., Bakker, J., Ballas, Z.K., Barreiro, E., Bell, S.C., Bellomo, R., Bernstein, J.A., Branson, R.D., Brusasco, V., Chalmers, J.D., Chokroverty, S., Citerio, G., Collop, N.A., Cooke, C.R., Crapo, J.D., Donaldson, G., Fitzgerald, D.A., Grainger, E., Hale, L., Herth, F.J., Kochanek, P.M., Marks, G., Moorman, J.R., Ost, D.E., Schatz, M., Sheikh, A., Smyth, A.R., Stewart, I., Stewart, P.W., Swenson, E.R., Szymusiak, R., Teboul, J.-L., Vincent, J.-L., Wedzicha, J.A., and Maslove, D.M. (2020). Development and reporting of prediction models: guidance for authors from editors of respiratory, sleep, and critical care journals. Crit. Care Med. 48, 623-633.
Richter, S., Stevenson, S., Newman, T., Wilson, L., Menon, D., Maas, A., Nieboer, D., Lingsma, H., Steyerberg, E.W., and Newcombe, V. (2019). Handling of missing outcome data in traumatic brain injury research: a systematic review. J. Neurotrauma 36, 2743-2752.
Maas, A.I.R., Menon, D.K., Steyerberg, E.W., Citerio, G., Lecky, F., Manley, G.T., Hill, S., Legrand, V., and Sorgner, A.; CENTER-TBI Participants and Investigators. (2015). Collaborative European NeuroTrauma Effectiveness Research in Traumatic Brain Injury (CENTER-TBI): a prospective longitudinal observational study. Neurosurgery 76, 67-80.
Steyerberg, E.W., Wiegers, E., Sewalt, C., Buki, A., Citerio, G., Keyser, V. De, Ercole, A., and Kunzmann, K. (2019). Case-mix , care pathways , and outcomes in patients with traumatic brain injury in CENTER-TBI: a European prospective, multicentre, longitudinal, cohort study. Lancet Neurol. 18, 923-934.
Shmueli, G. (2010). To explain or to predict? Stat. Sci. 25, 289-310.
Steyerberg, E.W., Mushkudiani, N., Perel, P., Butcher, I., Lu, J., McHugh, G.S., Murray, G.D., Marmarou, A., Roberts, I., Habbema, J.D.F., and Maas, A.I.R. (2008). Predicting outcome after traumatic brain injury: development and international validation of prognostic scores based on admission characteristics. PLoS Med. 5, e165; discussion, e165.
Perel, P., Arango, M., Clayton, T., Edwards, P., Komolafe, E., Poccock, S., Roberts, I., Shakur, H., Steyerberg, E., and Yutthakasemsunt, S. (2008). Predicting outcome after traumatic brain injury: practical prognostic models based on large cohort of international patients. BMJ 336, 425-429.
Dijkland, S.A., Foks, K.A., Polinder, S., Dippel, D.W.J., Maas, A.I.R., Lingsma, H.F., and Steyerberg, E.W. (2020). Prognosis in moderate and severe traumatic brain injury: a systematic review of contemporary models and validation studies. J. Neurotrauma 37, 1-13.
MRC CRASH Trial Collaborators. (2008). Predicting outcome after traumatic brain injury: practical prognostic models based on large cohort of international patients. BMJ 336, 425-429.
Cox, D. (1958). Two further applications of a model for binary regression. Biometrika 45, 562-565.
White, I.R., and Carlin, J.B. (2010). Bias and efficiency of multiple imputation compared with complete-case analysis for missing covariate values. Stat. Med. 29, 2920-2931.
Collins, G.S., Reitsma, J.B., Altman, D.G., and Moons, K.G.M. (2015). Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): the TRIPOD statement. Ann. Intern. Med. 162, 55-63.
Hughes, R.A., Heron, J., Sterne, J.A.C., and Tilling, K. (2019). Accounting for missing data in statistical analyses: Multiple imputation is not always the answer. Int. J. Epidemiol. 48, 1294-1304.
Van Houwelingen, H.C., and Sauerbrei, W. (2013). Cross-validation, shrinkage and variable selection in linear regression revisited. Open J. Stat. 3, 79-102.
Rubin, D.B. (2004). Multiple Imputation for Nonresponse in Surveys. Wiley-Interscience: Hoboken, NJ.
Lingsma, H.F., Roozenbeek, B., Steyerberg, E.W., Murray, G.D., and Maas, A.I. (2010). Early prognosis in traumatic brain injury: from prophecies to predictions. Lancet Neurol. 9, 543-554.
Roozenbeek, B., Lingsma, H.F., Lecky, F.E., Lu, J., Weir, J., Butcher, I., McHugh, G.S., Murray, G.D., Perel, P., Maas, A.I., and Steyerberg, E.W.; International Mission on Prognosis Analysis of Clinical Trials in Traumatic Brain Injury (IMPACT) Study Group, Corticosteroid Randomisation After Significant Head Injury (CRASH) Trial Collaborators, and Trauma Audit and Research Network (TARN). (2012). Prediction of outcome after moderate and severe traumatic brain injury. Crit. Care Med. 40, 1609-1617.
Majdan, M., Lingsma, H.F., Nieboer, D., Mauritz, W., Rusnak, M., and Steyerberg, E.W. (2014). Performance of IMPACT, CRASH and Nijmegen models in predicting six month outcome of patients with severe or moderate TBI: an external validation study. Scand. J. Trauma. Resusc. Emerg. Med. 22, 68.
Lingsma, H., Andriessen, T.M.J.C., Haitsema, I., Horn, J., van der Naalt, J., Franschman, G., Maas, A.I.R., Vos, P.E., and Steyerberg, E.W. (2013). Prognosis in moderate and severe traumatic brain injury. J. Trauma Acute Care Surg. 74, 639-646.
Teasdale, G., and Jennett, B. (1974). Assessment of coma and impaired consciousness. A practical scale. Lancet 2, 81-84.
Steyerberg, E.W., Nieboer, D., Debray, T.P.A., and Houwelingen, H.C. (2019). Assessment of heterogeneity in an individual participant data meta-analysis of prediction models: an overview and illustration. Stat. Med. 38, 4290-4309.
Lee, K.J., and Carlin, J.B. (2012). Recovery of information from multiple imputation: a simulation study. Emerg. Themes Epidemiol. 9, 3.
Madley-Dowd, P., Hughes, R., Tilling, K., and Heron, J. (2019). The proportion of missing data should not be used to guide decisions on multiple imputation. J. Clin. Epidemiol. 110, 63-73.
Von Hippel, P., and Lynch, J. (2013). Efficiency gains from using auxiliary variables in imputation. arxiv.org/ftp/arxiv/papers/1311/ 1311.5249.pdf (Last accessed February 3, 2021).
Mustillo, S. (2012). The effects of auxiliary variables on coefficient bias and efficiency in multiple imputation. Sociol. Methods Res. 41, 335-361.
Moons, K.G.M., Donders, R.A.R.T., Stijnen, T., and Harrell, F.E. (2006). Using the outcome for imputation of missing predictor values was preferred. J. Clin. Epidemiol. 59, 1092-1101.
Foks, K.A., Van Den Brand, C.L., Lingsma, H.F., Van Der Naalt, J., Jacobs, B., De Jong, E., Den Boogert, H.F., Sir, O., Patka, P., Polinder, S., Gaakeer, M.I., Schutte, C.E., Jie, K.E., Visee, H.F., Hunink, M.G.M., Reijners, E., Braaksma, M., Schoonman, G.G., Steyerberg, E.W., Jellema, K., and Dippel, D.W.J. (2018). External validation of computed tomography decision rules for minor head injury: prospective, multicentre cohort study in the Netherlands. BMJ 362, k3527.
Kunzmann, K., Wernisch, L., Richardson, S., Steyerberg, E., Lingsma, H., Ercole, A., Maas, A., Menon, D., and Wilson, L. (2021). Imputation of ordinal outcomes: a comparison of approaches in traumatic brain injury. J. Neurotrauma. 38, 455-463.
vonHippel, P.T. (2007).Regressionwithmissing Ys: an improved strategy for analyzing multiply imputed data. Sociol. Methodol. 37, 83-117.
Kontopantelis, E., White, I.R., Sperrin, M., and Buchan, I. (2017). Outcome-sensitive multiple imputation: a simulation study. BMC Med. Res. Methodol. 17, 2.
Sullivan, T.R., Salter, A.B., Ryan, P., and Lee, K.J. (2015). Bias and precision of the ''multiple imputation, then deletion'' method for dealing with missing outcome data. Am. J. Epidemiol. 182, 528-534.
Wahlby, U., Jonsson, E.N., and Karlsson, M.O. (2002). Comparison of stepwise covariate model building strategies in population pharmacokinetic-pharmacodynamic analysis. AAPS PharmSci. 4, E27.
Derksen, S., and Keselman, H.J. (1992). Backward, forward and stepwise automated subset selection algorithms: frequency of obtaining authentic and noise variables. Br. J. Math. Stat. Psychol. 45, 265-282.
Wood, A.M., White, I.R., and Royston, P. (2008). How should variable selection be performed with multiply imputed data? Stat. Med. 27, 3227-3246.
Steyerberg, E.W., and Harrell, F.E. (2016). Prediction models need appropriate internal, internal-external, and external validation HHS Public Access. J. Clin. Epidemiol. 69, 245-247.
Steyerberg, E.W., Harrell, F.E., Borsboom, G.J.J.M., Eijkemans, M.J.C., Vergouwe, Y., and Habbema, J.D.F. (2001). Internal validation of predictive models: efficiency of some procedures for logistic regression analysis. J. Clin. Epidemiol. 54, 774-781.
Efron, B., and Tibshirani, R.J. (1993). An Introduction to the Bootstrap. CRC: Boca Raton, FL.
Schomaker, M., and Heumann, C. (2018). Bootstrap inference when using multiple imputation. Stat. Med. 37, 2252-2266.
Bouwmeester, W., Zuithoff, N.P.A., Mallett, S., Geerlings, M.I., Vergouwe, Y., Steyerberg, E.W., Altman, D.G., and Moons, K.G.M. (2012). Reporting and methods in clinical prediction research: a systematic review. PLoS Med. 9, e1001221.
Mercaldo, S.F., and Blume, J.D. (2020). Missing data and prediction: the pattern submodel. Biostatistics 21, 236-252.
Seaman, S.R., and White, I.R. (2013). Review of inverse probability weighting for dealing with missing data. Stat. Methods Med. Res. 22, 278-295.
Perkins, N.J., Cole, S.R., Harel, O., Tchetgen Tchetgen, E.J., Sun, B., Mitchell, E.M., and Schisterman, E.F. (2018). Principled approaches to missing data in epidemiologic studies. Am. J. Epidemiol. 187, 568-575.