Machine learning algorithms performed no better than regression models for prognostication in traumatic brain injury

Gravesteijn, Benjamin Y.; Nieboer, Daan; Ercole, Ari; Lingsma, Hester F.; Nelson, David W.; van Calster, Ben; Steyerberg, E. W.; Åkerlund, Cecilia; Amrein, Krisztina; Andelic, Nada; Andreassen, Lasse; Anke, Audny; Antoni, Anna; Audibert, Gérard; Azouvi, Philippe; Azzolini, Maria Luisa; Bartels, Ronald; Barzó, Pál; Beauvais, Romuald; Beer, Ronny; Bellander, Bo Michael; Belli, Antonio; Benali, Habib; Berardino, Maurizio; Beretta, Luigi; Blaabjerg, M.; Bragge, P.; Brazinova, A.; Brinck, V.; Brooker, J.; Brorsson, C.; Buki, A.; Bullinger, M.; Cabeleira, M.; Caccioppola, A.; Calappi, E.; Calvi, M. R.; Cameron, P.; Lozano, G. C.; Carbonara, M.; Chevallard, G.; Chieregato, A.; Citerio, G.; Cnossen, M.; Coburn, M.; Coles, J.; Cooper, D. J.; Correia, M.; Čović, A.; Curry, N.; Czeiter, E.; Czosnyka, M.; Dahyot-Fizelier, C.; Dawes, H.; De Keyser, Véronique; Degos, V.; Della Corte, F.; Boogert, H. D.; Depreitere, B.; Đilvesi, Đ.; Dixit, A.; Donoghue, E.; Guy-Loup Dulière, J. D.; Esser, P.; Martin Fabricius, E. E.; Feigin, Kelly Foks; Frisvold, S.; Furmanov, A.; Gagliardo, P.; Galanaud, D.; Gantner, D.; Gao, G.; George, Pradeep; Ghuysen, Alexandre; Giga, L.; Glocker, B.; Golubovic, J.; Gomez, P. A.; Gratz, J.; Gravesteijn, B.; Grossi, F.; Gruen, R. L.; Gupta, D.; Haagsma, J. A.; Haitsma, I.; Helbok, R.; Helseth, E.; Horton, L.; Huijben, J.; Hutchinson, P. J.; Jacobs, B.; Jankowski, S.; Ji-yao Jiang, M. J.; Jones, K.; Karan, M.; Kolias, A. G.; Kompanje, E.; Kondziella, D.; Koraropoulos, E.; Koskinen, L.-O.; Kovács, N.; Lagares, A.; Lanyon, L.; Laureys, Steven; Lecky, F.; Lefering, R.; Legrand, Victor; Lejeune, Aurélie; Levi, L.; Lightfoot, R.; Lingsma, H.; Maas, A. I. R.; Castaño-León, A. M.; Maegele, M.; Majdan, M.; Manara, A.; Manley, G.; Martino, C.; Maréchal, H.; Mattern, J.; McMahon, C.; Melegh, B.; Menon, D.; Menovsky, T.; Mulazzi, D.; Muraleedharan, V.; Murray, L.; Nair, N.; Negru, A.; Newcombe, V.; Noirhomme, Quentin; Nyirádi, J.; Olubukola, O.; Oresic, M.; Ortolano, F.; Palotie, A.; Parizel, P. M.; Payen, J.-F.; Perera, N.; Perlbarg, V.; Persona, P.; Peul, W.; Piippo-Karjalainen, A.; Pirinen, M.; Ples, H.; Polinder, S.; Pomposo, I.; Posti, J. P.; Puybasset, L.; Radoi, A.; Ragauskas, A.; Raj, R.; Rambadagalla, M.; Real, R.; Rhodes, J.; Richardson, S.; Richter, S.; Ripatti, S.; Rocka, S.; Roe, C.; Roise, O.; Rosand, J.; Rosenfeld, J. V.; Rosenlund, C.; Rosenthal, G.; Rossaint, R.; Rossi, S.; Rueckert, D.; Rusnák, M.; Sahuquillo, J.; Sakowitz, O.; Sanchez-Porras, R.; Sandor, J.; Schäfer, N.; Schmidt, S.; Schoechl, H.; Schoonman, G.; Schou, R. F.; Schwendenwein, E.; Sewalt, C.; Skandsen, T.; Smielewski, P.; Sorinola, A.; Stamatakis, E.; Stanworth, S.; Kowark, A.; Stevens, R.; Stewart, W.; Stocchetti, N.; Sundström, N.; Synnot, A.; Takala, R.; Tamás, V.; Tamosuitis, T.; Taylor, M. S.; Ao, B. T.; Tenovuo, O.; Theadom, A.; Thomas, M.; Tibboel, D.; Timmers, M.; Tolias, C.; Trapani, T.; Tudora, C. M.; Vajkoczy, P.; Vallance, S.; Valeinis, E.; Vámos, Z.; Van der Steen, G.; van der Naalt, J.; van Dijck, J. T. J. M.; van Essen, T. A.; Van Hecke, W.; van Heugten, C.; Van Praag, D.; Vyvere, T. V.; VANHAUDENHUYSE, Audrey; van Wijk, R. P. J.; Vargiolu, A.; Vega, E.; Velt, K.; Verheyden, J.; Vespa, P. M.; Vik, A.; Vilcinis, R.; Volovici, V.; von Steinbüchel, N.; Voormolen, D.; Vulekovic, P.; Wang, K. K. W.; Wiegers, E.; Williams, G.; Wilson, L.; Winzeck, S.; Wolf, S.; Yang, Z.; Ylén, P.; Younsi, A.; Zeiler, F. A.; Zelinkova, V.; Ziverte, A.; Zoerle, T.; CENTER-TBI collaborators

doi:10.1016/j.jclinepi.2020.03.005

Download

Article (Scientific journals)

Machine learning algorithms performed no better than regression models for prognostication in traumatic brain injury

Gravesteijn, Benjamin Y.; Nieboer, Daan; Ercole, Ari et al.

2020 • In Journal of Clinical Epidemiology, 122, p. 95-107

Peer Reviewed verified by ORBi

Permalink
https://hdl.handle.net/2268/249323

DOI
10.1016/j.jclinepi.2020.03.005

PubMed
32201256

Files (1)Send to Details Statistics Bibliography Similar publications

Files

Full Text

Gravesteij_Machine_2020Journal-of-Clinical-Epidemiology.pdf

Publisher postprint (469.81 kB)

Download

This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/ 4.0/).

All documents in ORBi are protected by a user license.

Send to

RIS BibTex APA Chicago Permalink X Linkedin

Details

Keywords :

Cohort study; Data science; Machine learning; Prediction; Prognosis; Traumatic brain injury; Article; Glasgow coma scale

Abstract :

[en] Objective: We aimed to explore the added value of common machine learning (ML) algorithms for prediction of outcome for moderate and severe traumatic brain injury. Study Design and Setting: We performed logistic regression (LR), lasso regression, and ridge regression with key baseline predictors in the IMPACT-II database (15 studies, n = 11,022). ML algorithms included support vector machines, random forests, gradient boosting machines, and artificial neural networks and were trained using the same predictors. To assess generalizability of predictions, we performed internal, internal-external, and external validation on the recent CENTER-TBI study (patients with Glasgow Coma Scale <13, n = 1,554). Both calibration (calibration slope/intercept) and discrimination (area under the curve) was quantified. Results: In the IMPACT-II database, 3,332/11,022 (30%) died and 5,233(48%) had unfavorable outcome (Glasgow Outcome Scale less than 4). In the CENTER-TBI study, 348/1,554(29%) died and 651(54%) had unfavorable outcome. Discrimination and calibration varied widely between the studies and less so between the studied algorithms. The mean area under the curve was 0.82 for mortality and 0.77 for unfavorable outcomes in the CENTER-TBI study. Conclusion: ML algorithms may not outperform traditional regression approaches in a low-dimensional setting for outcome prediction after moderate or severe traumatic brain injury. Similar to regression-based prediction models, ML algorithms should be rigorously validated to ensure applicability to new populations. © 2020 The Authors

Research Center/Unit :

CHU de Liège-Centre du Cerveau² - ULiège

Disciplines :

Neurosciences & behavior

Author, co-author :

Gravesteijn, Benjamin Y.; Departments of Public Health, Erasmus MC – University Medical Centre Rotterdam, Postbus 2040, Rotterdam, CA 3000, Netherlands

Nieboer, Daan; Departments of Public Health, Erasmus MC – University Medical Centre Rotterdam, Rotterdam, Netherlands

Ercole, Ari; Division of Anaesthesia, University of Cambridge, Cambridge, United Kingdom

Lingsma, Hester F.; Departments of Public Health, Erasmus MC – University Medical Centre Rotterdam, Rotterdam, Netherlands

Nelson, David W.; Department of Physiology and Pharmacology, Section of Perioperative Medicine and Intensive Care, Karolinska Institutet, Stockholm, Sweden

van Calster, Ben; Department of Development and Regeneration, KU Leuven, Belgium, Department of Biomedical Data Sciences, Leiden University Medical Centre, Leiden, Netherlands

Steyerberg, E. W.; Departments of Public Health, Erasmus MC – University Medical Centre Rotterdam, Rotterdam, Netherlands, Department of Biomedical Data Sciences, Leiden University Medical Centre, Leiden, Netherlands

Åkerlund, Cecilia

Amrein, Krisztina

Andelic, Nada

More authors (233 more)

Language :

English

Title :

Machine learning algorithms performed no better than regression models for prognostication in traumatic brain injury

Publication date :

2020

Journal title :

Journal of Clinical Epidemiology

ISSN :

0895-4356

eISSN :

1878-5921

Publisher :

Elsevier USA

Volume :

122

Pages :

95-107

Peer reviewed :

Peer Reviewed verified by ORBi

Available on ORBi :

since 07 July 2020

Statistics

Number of views

180 (6 by ULiège)

Number of downloads

71 (3 by ULiège)

More statistics

Scopus citations^®

124

Scopus citations^®
without self-citations

106

OpenCitations

OpenAlex citations

150

Bibliography

Maas, A.I.R., Menon, D.K., Adelson, P.D., Andelic, N., Bell, M.J., Belli, A., et al. Traumatic brain injury: integrated approaches to improve prevention, clinical care, and research. Lancet Neurol, 16, 2017, 987.
Majdan, M., Plancikova, D., Brazinova, A., Rusnak, M., Nieboer, D., Feigin, V., et al. Epidemiology of traumatic brain injuries in Europe: a cross-sectional analysis. Lancet Public Heal 1 (2016), e76–e83.
Saatman, K.E., Duhaime, A., Bullock, R., Maas, A.I., Valadka, A., Manley, G.T., et al. Classification of traumatic brain injury for targeted therapies. J Neurotrauma 25 (2008), 719–738.
Lingsma, H.F., Roozenbeek, B., Steyerberg, E.W., Murray, G.D., Maas, A.I., Early prognosis in traumatic brain injury: from prophecies to predictions. Lancet Neurol 9 (2010), 543–554.
Liu, N.T., Salinas, J., Machine learning for predicting outcomes in trauma. Shock 48 (2017), 504–510.
Burges, C.J.C., A tutorial on support vector machines for pattern recognition. Data Mining Knowledge Discovery 2 (1998), 121–167.
Jain, A.K., Mao, J., Mohiuddin, K.M., Artificial neural networks: a tutorial. Computer (Long Beach Calif) 29 (1996), 31–44.
Afanador, N.L., Smolinska, A., Tran, T.N., Blanchet, L., Unsupervised random forest: a tutorial with case studies. J Chemom 30 (2016), 232–241.
Natekin, A., Knoll, A., Gradient boosting machines, a tutorial. Front Neurorobot, 7, 2013, 21.
Rau, C.-S., Kuo, P.-J., Chien, P.-C., Huang, C.-Y., Hsieh, H.-Y., Hsieh, C.-H., Mortality prediction in patients with isolated moderate and severe traumatic brain injury using machine learning models. PLoS One, 13, 2018, e0207192.
Matsuo, K., Aihara, H., Nakai, T., Morishita, A., Tohma, Y., Kohmura, E., Machine learning to predict in-hospital Morbidity and mortality after traumatic brain injury. J Neurotrauma 37 (2018), 202–210.
Feng, J., Wang, Y., Peng, J., Sun, M., Zeng, J., Jiang, H., Comparison between logistic regression and machine learning algorithms on survival prediction of traumatic brain injuries. J Crit Care 54 (2019), 110–116.
van der Ploeg, T., Austin, P.C., Steyerberg, E.W., Modern modelling techniques are data hungry: a simulation study for predicting dichotomous endpoints. BMC Med Res Methodol, 14, 2014, 137.
Christodoulou, E., Jie, M.A., Collins, G.S., Steyerberg, E.W., Verbakel, J.Y., van Calster, B., A systematic review shows no performance benefit of machine learning over logistic regression for clinical prediction models. J Clin Epidemiol 110 (2019), 12–22.
van Os HJA, Ramos, L.A., Hilbert, A., van Leeuwen, M., van Walderveen, M.A.A., Kruyt, N.D., et al. Predicting outcome of endovascular treatment for acute ischemic stroke: potential value of machine learning algorithms. Front Neurol, 9, 2018, 784.
Churpek, M.M., Yuen, T.C., Winslow, C., Meltzer, D.O., Kattan, M.W., Edelson, D.P., Multicenter comparison of machine learning methods and conventional regression for predicting clinical deterioration on the wards. Crit Care Med 44 (2016), 368–374.
Lee, H.-C., Yoon, S., Yang, S.-M., Kim, W., Ryu, H.-G., Jung, C.-W., et al. Prediction of acute kidney injury after liver transplantation: machine learning approaches vs. logistic regression model. J Clin Med, 7, 2018, 428.
Bisaso, K.R., Karungi, S.A., Kiragga, A., Mukonzo, J.K., Castelnuovo, B., A comparative study of logistic regression based machine learning techniques for prediction of early virological suppression in antiretroviral initiating HIV patients. BMC Med Inform Decis Mak, 18, 2018, 77.
Decruyenaere, A., Decruyenaere, P., Peeters, P., Vermassen, F., Dhaene, T., Couckuyt, I., Prediction of delayed graft function after kidney transplantation: comparison between logistic regression and machine learning methods. BMC Med Inform Decis Mak, 15, 2015, 83.
Harrell, F.E., Regression Modeling Strategies. 2001, Springer New York, New York, NY.
Steyerberg, E.W., Clinical Prediction Models. 2009, Springer New York, New York, NY.
Van Calster, B., Nieboer, D., Vergouwe, Y., De Cock, B., Pencina, M.J., Steyerberg, E.W., A calibration hierarchy for risk models was defined: from utopia to empirical data. J Clin Epidemiol 74 (2016), 167–176.
Collins, G.S., Reitsma, J.B., Altman, D.G., Moons, K.G.M., Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD Statement. Ann Intern Med, 162, 2015, 55.
Moons, K.G.M., Kengne, A.P., Grobbee, D.E., Royston, P., Vergouwe, Y., Altman, D.G., et al. Risk prediction models: II. External validation, model updating, and impact assessment. Heart 98 (2012), 691–698.
Steyerberg, E.W., Mushkudiani, N., Perel, P., Butcher, I., Lu, J., McHugh, G.S., et al. Predicting outcome after traumatic brain injury: development and international validation of prognostic scores based on admission characteristics. PLoS Med, 5, 2008, e165.
Marmarou, A., Lu, J., Butcher, I., McHugh, G.S., Mushkudiani, N.A., Murray, G.D., et al. IMPACT database of traumatic brain injury: design and description. J Neurotrauma 24 (2007), 239–250.
Maas, A.I.R., Menon, D.K., Steyerberg, E.W., Citerio, G., Lecky, F., Manley, G.T., et al. Collaborative European neurotrauma effectiveness research in traumatic brain injury (CENTER-TBI): a prospective longitudinal observational study. Neurosurgery 76 (2015), 67–80.
Steyerberg, E.W., Clinical prediction models: a practical approach to development, validation and updating. 2009, Springer, New York.
Tibshirani, R., Regression shrinkage and selection via the lasso. J R Stat Soc Ser B 58 (1996), 267–288.
FIRTH, D., Bias reduction of maximum likelihood estimates. Biometrika 80 (1993), 27–38.
Rubin, D.B., Multiple imputation for nonresponse in surveys. 2004, John Wiley & Sons, Hoboken, New Jersey.
Buuren, S van, Flexible imputation of missing data. 2018, CRC Press, Cleveland, Ohio.
Royston, P., Parmar, M.K.B., Sylvester, R., Construction and validation of a prognostic model across several studies, with an application in superficial bladder cancer. Stat Med 23 (2004), 907–926.
Steyerberg, E.W., Harrell, F.E., Prediction models need appropriate internal, internal-external, and external validation HHS Public Access. J Clin Epidemiol 69 (2016), 245–247.
Cox, D., Two further applications of a model for binary regression. Biometrika 45 (1958), 562–565.
DeLong, E.R., DeLong, D.M., Clarke-Pearson, D.L., Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics 44 (1988), 837–845.
Steyerberg, E.W., Vickers, A.J., Cook, N.R., Gerds, T., Gonen, M., Obuchowski, N., et al. Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology 21 (2010), 128–138.
DerSimonian, R., Laird, N., Meta-analysis in clinical trials. Control Clin Trials 7 (1986), 177–188.
Steyerberg, E.W., Vergouwe, Y., Towards better clinical prediction models: Seven steps for development and an ABCD for validation. Eur Heart J 35 (2014), 1925–1931.
Breiman, L., Random Forests. Mach Learn 45 (2001), 5–32.
Probst, P., Boulesteix, A.-L., Bischl, B., Tunability: importance of hyperparameters of machine learning algorithms. J Mach Learn Res 20 (2019), 1–32.
Sakr, S., Elshawi, R., Ahmed, A.M., Qureshi, W.T., Brawner, C.A., Keteyian, S.J., et al. Comparison of machine learning techniques to predict all-cause mortality using fitness data: the Henry ford exercIse testing (FIT) project. BMC Med Inform Decis Mak, 17, 2017, 174.
König, I.R., Malley, J.D., Weimar, C., Diener, H.-C., Ziegler, A., German Stroke Study Collaboration. Practical experiences on the necessity of external validation. Stat Med 26 (2007), 5499–5511.
Thelin, E.P., Nelson, D.W., Vehvilä Inen, J., Nyströ, H., Kivisaari, R., Siironen, J., et al. Evaluation of novel computerized tomography scoring systems in human traumatic brain injury: an observational, multicenter study. PLoS Med, 14, 2017, e1002368.
Ngiam, K.Y., Khor, I.W., Big data and machine learning algorithms for health-care delivery. Lancet Oncol 20 (2019), e262–e273.
van der Ploeg, T., Nieboer, D., Steyerberg, E.W., Modern modeling techniques had limited external validity in predicting mortality from traumatic brain injury. J Clin Epidemiol 78 (2016), 83–89.
Poldrack, R.A., Gorgolewski, K.J., Making big data open: data sharing in neuroimaging. Nat Neurosci 17 (2014), 1510–1517.
Neurology, T.L., The changing landscape of traumatic brain injury research. Lancet Neurol, 11, 2012, 651.
Charles, D., Gabriel, M., Searcy, T., Adoption of electronic health record systems among U.S. non-federal acute care hospitals: 2008-2014. ONC Data Brief 35 (2016), 1–9.
Rajkomar, A., Oren, E., Chen, K., Dai, A.M., Hajaj, N., Hardt, M., et al. Scalable and accurate deep learning with electronic health records. NPJ Digit Med, 1, 2018, 18.
Beam, A.L., Kohane, I.S., Big data and machine learning in health care. JAMA, 319, 2018, 1317.
Delahanty, R.J., Kaufman, D., Jones, S.S., Development and evaluation of an automated machine learning algorithm for in-hospital mortality risk adjustment among critical care patients. Crit Care Med 46 (2018), e481–e488.
Desautels, T., Das, R., Calvert, J., Trivedi, M., Summers, C., Wales, D.J., et al. Prediction of early unplanned intensive care unit readmission in a UK tertiary care hospital: a cross-sectional machine learning approach. BMJ Open, 7, 2017, e017199.
Klau, S., Jurinovic, V., Hornung, R., Herold, T., Boulesteix, A.-L., Priority-Lasso: a simple hierarchical approach to the prediction of clinical outcome using multi-omics data. BMC Bioinformatics, 19, 2018, 322.
Reps, J.M., Schuemie, M.J., Suchard, M.A., Ryan, P.B., Rijnbeek, P.R., Design and implementation of a standardized framework to generate and evaluate patient-level prediction models using observational healthcare data. J Am Med Inform Assoc 25 (2018), 969–975.