Article (Scientific journals)
Biased binomial assessment of cross-validated estimation of classification accuracies illustrated in diagnosis predictions
Noirhomme, Quentin; Lesenfants, Damien; Gomez, Francisco et al.
2014In NeuroImage: Clinical, 4, p. 687-694
Peer Reviewed verified by ORBi
 

Files


Full Text
noirhomme_NICL_2014.pdf
Publisher postprint (925.82 kB)
Download

All documents in ORBi are protected by a user license.

Send to



Details



Keywords :
classification; cross-validation; binomial; permutation test
Abstract :
[en] Multivariate classification is used in neuroimaging studies to infer brain activation or in medical applications to infer diagnosis. Their results are often assessed through either a binomial or a permutation test. Here, we simulated classification results of generated random data to assess the influence of the cross-validation scheme on the significance of results. Distributions built from classification of random data with crossvalidation did not follow the binomial distribution. The binomial test is therefore not adapted. On the contrary, the permutation test was unaffected by the cross-validation scheme. The influence of the crossvalidation was further illustrated on real-data from a brain–computer interface experiment in patients with disorders of consciousness and from an fMRI study on patients with Parkinson disease. Three out of 16 patients with disorders of consciousness had significant accuracy on binomial testing, but only one showed significant accuracy using permutation testing. In the fMRI experiment, the mental imagery of gait could discriminate significantly between idiopathic Parkinson’s disease patients and healthy subjects according to the permutation test but not according to the binomial test. Hence, binomial testing could lead to biased estimation of significance and false positive or negative results. In our view, permutation testing is thus recommended for clinical application of classification with cross-validation.
Research center :
GIGA CRC (Cyclotron Research Center) In vivo Imaging-Aging & Memory - ULiège
Disciplines :
Neurology
Engineering, computing & technology: Multidisciplinary, general & others
Author, co-author :
Noirhomme, Quentin ;  Université de Liège - ULiège > Centre de recherches du cyclotron
Lesenfants, Damien ;  Université de Liège - ULiège > Centre de recherches du cyclotron
Gomez, Francisco;  Universidad Central de Colombia > Computer Science Department > Complexus Group
Soddu, Andrea;  University of Western Ontario > Department of Physics & Astronomy > Brain and Mind Institute
Schrouff, Jessica ;  Université de Liège - ULiège > Dép. d'électric., électron. et informat. (Inst.Montefiore) > Systèmes et modélisation
Garraux, Gaëtan  ;  Université de Liège - ULiège > Département des sciences cliniques > Neurologie
Luxen, André ;  Université de Liège - ULiège > Département de chimie (sciences) > Chimie organique de synthèse
Phillips, Christophe   ;  Université de Liège - ULiège > Centre de recherches du cyclotron
Laureys, Steven   ;  Université de Liège - ULiège > Centre de recherches du cyclotron
 These authors have contributed equally to this work.
Language :
English
Title :
Biased binomial assessment of cross-validated estimation of classification accuracies illustrated in diagnosis predictions
Publication date :
April 2014
Journal title :
NeuroImage: Clinical
eISSN :
2213-1582
Publisher :
Elsevier
Volume :
4
Pages :
687-694
Peer reviewed :
Peer Reviewed verified by ORBi
Available on ORBi :
since 25 April 2014

Statistics


Number of views
136 (14 by ULiège)
Number of downloads
167 (4 by ULiège)

Scopus citations®
 
93
Scopus citations®
without self-citations
81
OpenCitations
 
100

Bibliography


Similar publications



Contact ORBi