classification; cross-validation; binomial; permutation test
Abstract :
[en] Multivariate classification is used in neuroimaging studies to infer brain activation or in medical applications to infer diagnosis. Their results are often assessed through either a binomial or a permutation test. Here, we simulated classification results of generated random data to assess the influence of the cross-validation scheme on the significance of results. Distributions built from classification of random data with crossvalidation did not follow the binomial distribution. The binomial test is therefore not adapted. On the contrary, the permutation test was unaffected by the cross-validation scheme. The influence of the crossvalidation was further illustrated on real-data from a brain–computer interface experiment in patients with
disorders of consciousness and from an fMRI study on patients with Parkinson disease. Three out of 16 patients with disorders of consciousness had significant accuracy on binomial testing, but only one showed significant accuracy using permutation testing. In the fMRI experiment, the mental imagery of gait could discriminate significantly between idiopathic Parkinson’s disease patients and healthy subjects according to the permutation test but not according to the binomial test. Hence, binomial testing could lead to biased estimation of significance and false positive or negative results. In our view, permutation testing is thus recommended for clinical application of classification with cross-validation.
Research center :
GIGA CRC (Cyclotron Research Center) In vivo Imaging-Aging & Memory - ULiège
Disciplines :
Neurology Engineering, computing & technology: Multidisciplinary, general & others
Author, co-author :
Noirhomme, Quentin ; Université de Liège - ULiège > Centre de recherches du cyclotron
Lesenfants, Damien ; Université de Liège - ULiège > Centre de recherches du cyclotron
Gomez, Francisco; Universidad Central de Colombia > Computer Science Department > Complexus Group
Soddu, Andrea; University of Western Ontario > Department of Physics & Astronomy > Brain and Mind Institute
Schrouff, Jessica ; Université de Liège - ULiège > Dép. d'électric., électron. et informat. (Inst.Montefiore) > Systèmes et modélisation
Garraux, Gaëtan ; Université de Liège - ULiège > Département des sciences cliniques > Neurologie
Luxen, André ; Université de Liège - ULiège > Département de chimie (sciences) > Chimie organique de synthèse
Phillips, Christophe ✱; Université de Liège - ULiège > Centre de recherches du cyclotron
Laureys, Steven ✱; Université de Liège - ULiège > Centre de recherches du cyclotron
✱ These authors have contributed equally to this work.
Language :
English
Title :
Biased binomial assessment of cross-validated estimation of classification accuracies illustrated in diagnosis predictions
Benjamini Y., Hochberg Y. Controlling the false discovery rate: a practical and powerful approach to multiple testing Journal of the Royal Statistical Society B 57 1995 289 300
Berrar D., Bradbury I. Avoiding model selection bias in small-sample genomic datasets Bioinformatics (Oxford, England) 22 10 2006 1245 1250 10.1093/bioinformatics/btl066 16500931
Billinger M., Daly I. Is it significant? B.Z. Allison S. Dunne R. Leeb R. Millan Jdel A. Nijholt Guidelines for Reporting BCI Performance. Towards Practical Brain-Computer Interfaces 2013 Springer Berlin, Heidelberg 333 354
Burges C. A tutorial on support vector machines for pattern recognition Data Mining and Knowledge Discovery 2 1998 121 167 10.1023/A:1009715923555
Chang C.-C., Lin C.-J. LIBSVM: a library for support vector machines ACM Transactions on Intelligent Systems and Technology 2 3 2011 27:21 27:27
Cremers J., D'Ostilio K. Brain activation pattern related to gait disturbances in Parkinson's disease Movement Disorders: Official Journal of the Movement Disorder Society 27 12 2012 1498 1505 10.1002/mds.25139 23008169
Cruse D., Chennu S. Bedside detection of awareness in the vegetative state: a cohort study Lancet 378 9809 2011 2088 2094 10.1016/S0140-6736(11) 61224-5 22078855
Donchin E., Spencer K.M. The mental prosthesis: assessing the speed of a P300-based brain-computer interface IEEE Transactions on Rehabilitation Engineering: A Publication of the IEEE Engineering in Medicine and Biology Society 8 2 2000 174 179 10.1109/86.847808 10896179
Efron B., Tibshirani R. Improvements on cross-validation: the.632+ bootstrap method Journal of the American Statistical Association 92 438 1997 548 560 10.1080/01621459.1997.1047400710.2307/2965703
Etzel J.A., Gazzola V. An introduction to anatomical ROI-based fMRI classification analysis Brain Research 1282 2009 114 125 10.1016/j.brainres. 2009.05.090 19505449
Farwell L.A., Donchin E. Talking off the top of your head: toward a mental prosthesis utilizing event-related brain potentials Electroencephalography and Clinical Neurophysiology 70 6 1988 510 523 10.1016/0013-4694(88)90149-6 2461285
Focke N.K., Helms G. Individual voxel-based subtype prediction can differentiate progressive supranuclear palsy from idiopathic Parkinson syndrome and healthy controls Human Brain Mapping 32 11 2011 1905 1915 10.1002/hbm.21161 21246668
Furdea A., Halder S. An auditory oddball (P300) spelling system for brain-computer interfaces Psychophysiology 46 3 2009 617 625 10.1111/j.1469-8986.2008.00783.x 19170946
Galanaud D., Perlbarg V. Assessment of white matter injury and outcome in severe brain trauma: a prospective multicenter cohort Anesthesiology 117 6 2012 1300 1310 10.1097/ALN.0b013e3182755558 23135261
Garraux G., Phillips C. Multiclass classification of FDG PET scans for the distinction between Parkinson's disease and atypical parkinsonian syndromes NeuroImage: Clinical 2 2013 883 893 10.1016/j.nicl.2013.06.004 24179839
Goldfine A.M., Bardin J.C. Reanalysis of "bedside detection of awareness in the vegetative state: a cohort study" Lancet 381 9863 2013 289 291 10.1016/S0140-6736(13)60125-7 23351802
Golub T.R., Slonim D.K. Molecular classification of cancer: class discovery and class prediction by gene expression monitoring Science (New York, N.Y.) 286 5439 1999 531 537 10.1126/science.286.5439.531 10521349
Good P. Permutation, Parametric and Bootstrap Tests of Hypotheses 2005 Springer United States of America
Howell D.C. Statistical Methods for Psychology 2012 Wadsworth
Hughes A.J., Daniel S.E. Accuracy of clinical diagnosis of idiopathic Parkinson's disease: a clinico-pathological study of 100 cases Journal of Neurology, Neurosurgery, and Psychiatry 55 3 1992 181 184 10.1136/jnnp.55.3.181 1564476
Kohavi R. A study of cross-validation and bootstrap for accuracy estimation and model selection International Joint Conference on Artificial Intelligence 1995
Krusienski D.J., Sellers E.W. A comparison of classification techniques for the P300 Speller Journal of Neural Engineering 3 4 2006 299 305 10.1088/1741-2560/3/4/007 17124334
Kubler A., Birbaumer N. Brain-computer interfaces and communication in paralysis: extinction of goal directed thinking in completely paralysed patients? Clinical Neurophysiology: Official Journal of the International Federation of Clinical Neurophysiology 119 11 2008 2658 2666 10.1016/j.clinph.2008.06.019 18824406
Laureys S., Schiff N.D. Coma and consciousness: paradigms (re)framed by neuroimaging Neuroimage 61 2 2012 478 491 10.1016/j.neuroimage.2011.12.041 22227888
Lemm S., Blankertz B. Introduction to machine learning for brain imaging Neuroimage 56 2 2011 387 399 10.1016/j.neuroimage.2010.11.004 21172442
Lule D., Noirhomme Q. Probing command following in patients with disorders of consciousness using a brain-computer interface Clinical Neurophysiology: Official Journal of the International Federation of Clinical Neurophysiology 124 1 2013 101 106 10.1016/j.clinph.2012.04.030 22920562
Luyt C.E., Galanaud D. Diffusion tensor imaging to predict long-term outcome after cardiac arrest: a bicentric pilot study Anesthesiology 117 6 2012 1311 1321 10.1097/ALN.0b013e318275148c 23135257
Maillet A., Pollak P. Imaging gait disorders in parkinsonism: a review Journal of Neurology, Neurosurgery, and Psychiatry 83 10 2012 986 993 10.1136/jnnp-2012-302461 22773859
Maris E., Oostenveld R. Nonparametric statistical testing of EEG- and MEG-data Journal of Neuroscience Methods 164 1 2007 177 190 10.1016/j.jneumeth. 2007.03.024 17517438
Martin J.K., Hirschberg D.S. Small Sample Statistics for Classification Error Rates II: Confidence Intervals and Significance Tests 1996 Department of Information and Computer Science, University of California, Irvine CA.
Mukherjee S., Golland P. Permutation Tests for Classification 2003 Massachusetts Institute of Technology Cambridge, MA 22
Müller-Putz G.R., Scherer R. Better than random? A closer look on BCI results International. Journal of Bioelectromagnetism. 10 1 2008 52 55
Nichols T.E., Holmes A.P. Nonparametric permutation tests for functional neuroimaging: a primer with examples Human Brain Mapping 15 1 2002 1 25 10.1002/hbm.1058 11747097
Ojala M., Garriga G.C. Permutation tests for studying classifier performance Journal of Machine Learning Research 11 2010 1833 1863
Orru G., Pettersson-Yeo W. Using support vector machine to identify imaging biomarkers of neurological and psychiatric disease: a critical review Neuroscience and Biobehavioral Reviews 36 4 2012 1140 1152 10.1016/j.neubiorev. 2012.01.004 22305994
Pereira F., Botvinick M. Information mapping with pattern classifiers: a comparative study Neuroimage 56 2 2011 476 496 10.1016/j.neuroimage.2010.05.026 20488249
Pereira F., Detre G. Generating text from functional brain images Frontiers in Human Neuroscience 5 2011 72 21927602
Pereira F., Mitchell T. Machine learning classifiers and fMRI: a tutorial overview NeuroImage 45 1 Suppl. 2009 S199 S209 10.1016/j.neuroimage.2008.11.007 19070668
Phillips C.L., Bruno M.A. "Relevance vector machine" consciousness classifier applied to cerebral metabolism of vegetative and locked-in patients Neuroimage 56 2 2011 797 808 10.1016/j.neuroimage.2010.05.083 20570741
Picard R.R., Cook R.D. Cross-validation of regression models Journal of the American Statistical Association 79 1984 575 583 10.1080/01621459.1984. 10478083
Schalk G., McFarland D.J. BCI2000: A general-purpose brain-computer interface (BCI) system IEEE Transactions on Biomedical Engineering 51 6 2004 1034 1043 10.1109/TBME.2004.827072 15188875
Schrouff J., Cremers J. Discriminant BOLD activation patterns during mental imagery in Parkinson's disease Machine Learning and Interpretation in NeuroImaging workshop at NIPS 2012 NIPS South Lake Tahoe, United States of America 8
Schrouff J., Cremers J. Localizing and comparing weight maps generated from linear kernel machine learning models 3rd Workshop on Pattern Recognition in NeuroImaging (PRNI 2013) 2013 IEEE Computer Society Conference Publishing Services Philadelphia, USA 4
Schrouff J., Rosa M.J. PRoNTo: pattern recognition for neuroimaging toolbox Neuroinformatics 11 3 2013 319 337 10.1007/s12021-013-9178-1 23417655
Sellers E.W., Donchin E. A P300-based brain-computer interface: initial tests by ALS patients Clinical Neurophysiology: Official Journal of the International Federation of Clinical Neurophysiology 117 3 2006 538 548 10.1016/j.clinph.2005.06.027 16461003
Sharbrough F., Chatrian G.-E. American Electroencephalographic Society Guidelines for Standard Electrode Position Nomenclature Journal of Clinical Neurophysiology: Official Publication of the American Electroencephalographic Society 8 1991 200 202 10.1097/00004691-199104000-00007 2050819
Simon R., Radmacher M.D. Pitfalls in the use of DNA microarray data for diagnostic and prognostic classification Journal of the National Cancer Institute 95 1 2003 14 18 10.1093/jnci/95.1.14 12509396
Cruse D., Gantner I., Soddu A., Owen A.M. Lies, damned lies, and diagnoses: Estimating the clinical utility of assessments of covert awareness in the Vegetative State. Brain Inj: in press.