Available on ORBi since
04 June 2020
Article (Scientific journals)
Variability in the analysis of a single neuroimaging dataset by many teams.
Botvinik-Nezer, Rotem; Holzmeister, Felix; Camerer, Colin F. et al.
2020 • In Nature, 582 (7810), p. 84-88
Peer Reviewed verified by ORBi
 

Files


Full Text
VariabilityInTheAnalysisOfASingleNeuroimagingDatasetByManyTeams_2020.pdf
Author preprint (1.3 MB)

All documents in ORBi are protected by a user license.

Send to



Details



Abstract :
[en] Data analysis workflows in many scientific domains have become increasingly complex and flexible. Here we assess the effect of this flexibility on the results of functional magnetic resonance imaging by asking 70 independent teams to analyse the same dataset, testing the same 9 ex-ante hypotheses(1). The flexibility of analytical approaches is exemplified by the fact that no two teams chose identical workflows to analyse the data. This flexibility resulted in sizeable variation in the results of hypothesis tests, even for teams whose statistical maps were highly correlated at intermediate stages of the analysis pipeline. Variation in reported results was related to several aspects of analysis methodology. Notably, a meta-analytical approach that aggregated information across teams yielded a significant consensus in activated regions. Furthermore, prediction markets of researchers in the field revealed an overestimation of the likelihood of significant findings, even by researchers with direct knowledge of the dataset(2-5). Our findings show that analytical flexibility can have substantial effects on scientific conclusions, and identify factors that may be related to variability in the analysis of functional magnetic resonance imaging. The results emphasize the importance of validating and sharing complex analysis workflows, and demonstrate the need for performing and reporting multiple analyses of the same data. Potential approaches that could be used to mitigate issues related to analytical variability are discussed.
Disciplines :
Neurosciences & behavior
Author, co-author :
Botvinik-Nezer, Rotem
Holzmeister, Felix
Camerer, Colin F.
Dreber, Anna
Huber, Juergen
Johannesson, Magnus
Kirchler, Michael
Iwanir, Roni
Mumford, Jeanette A.
Adcock, R. Alison
Avesani, Paolo
Baczkowski, Blazej M.
Bajracharya, Aahana
Bakst, Leah
Ball, Sheryl
Barilari, Marco
Bault, Nadège
Beaton, Derek
Beitner, Julia
Benoit, Roland G.
Berkers, Ruud M. W. J.
Bhanji, Jamil P.
Biswal, Bharat B.
Bobadilla-Suarez, Sebastian
Bortolini, Tiago
Bottenhorn, Katherine L.
Bowring, Alexander
Braem, Senne
Brooks, Hayley R.
Brudner, Emily G.
Calderon, Cristian B.
Camilleri, Julia A.
Castrellon, Jaime J.
Cecchetti, Luca
Cieslik, Edna C.
Cole, Zachary J.
Collignon, Olivier ;  Université de Liège - ULiège > Département des sciences cliniques > Département des sciences cliniques
Cox, Robert W.
Cunningham, William A.
Czoschke, Stefan
Dadi, Kamalaker
Davis, Charles P.
Luca, Alberto De
Delgado, Mauricio R.
Demetriou, Lysia
Dennison, Jeffrey B.
Di, Xin
Dickie, Erin W.
Dobryakova, Ekaterina
Donnat, Claire L.
Dukart, Juergen
Duncan, Niall W.
Durnez, Joke
Eed, Amr
Eickhoff, Simon B.
Erhart, Andrew
Fontanesi, Laura
Fricke, G. Matthew
Fu, Shiguang
Galván, Adriana
Gau, Remi
Genon, Sarah ;  Université de Liège - ULiège > CRC In vivo Imaging-Aging & Memory
Glatard, Tristan
Glerean, Enrico
Goeman, Jelle J.
Golowin, Sergej A. E.
González-García, Carlos
Gorgolewski, Krzysztof J.
Grady, Cheryl L.
Green, Mikella A.
Guassi Moreira, João F.
Guest, Olivia
Hakimi, Shabnam
Hamilton, J. Paul
Hancock, Roeland
Handjaras, Giacomo
Harry, Bronson B.
Hawco, Colin
Herholz, Peer
Herman, Gabrielle
Heunis, Stephan
Hoffstaedter, Felix
Hogeveen, Jeremy
Holmes, Susan
Hu, Chuan-Peng
Huettel, Scott A.
Hughes, Matthew E.
Iacovella, Vittorio
Iordan, Alexandru D.
Isager, Peder M.
Isik, Ayse I.
Jahn, Andrew
Johnson, Matthew R.
Johnstone, Tom
Joseph, Michael J. E.
Juliano, Anthony C.
Kable, Joseph W.
Kassinopoulos, Michalis
Koba, Cemal
Kong, Xiang-Zhen
Koscik, Timothy R.
Kucukboyaci, Nuri Erkut
Kuhl, Brice A.
Kupek, Sebastian
Laird, Angela R.
Lamm, Claus
Langner, Robert
Lauharatanahirun, Nina
Lee, Hongmi
Lee, Sangil
Leemans, Alexander
Leo, Andrea
Lesage, Elise
Li, Flora
Li, Monica Y. C.
Lim, Phui Cheng
Lintz, Evan N.
Liphardt, Schuyler W.
Losecaat Vermeer, Annabel B.
Love, Bradley C.
Mack, Michael L.
Malpica, Norberto
Marins, Theo
Maumet, Camille
McDonald, Kelsey
McGuire, Joseph T.
Melero, Helena
Méndez Leal, Adriana S.
Meyer, Benjamin
Meyer, Kristin N.
Mihai, Glad
Mitsis, Georgios D.
Moll, Jorge
Nielson, Dylan M.
Nilsonne, Gustav
Notter, Michael P.
Olivetti, Emanuele
Onicas, Adrian I.
Papale, Paolo
Patil, Kaustubh R.
Peelle, Jonathan E.
Pérez, Alexandre
Pischedda, Doris
Poline, Jean-Baptiste
Prystauka, Yanina
Ray, Shruti
Reuter-Lorenz, Patricia A.
Reynolds, Richard C.
Ricciardi, Emiliano
Rieck, Jenny R.
Rodriguez-Thompson, Anais M.
Romyn, Anthony
Salo, Taylor
Samanez-Larkin, Gregory R.
Sanz-Morales, Emilio
Schlichting, Margaret L.
Schultz, Douglas H.
Shen, Qiang
Sheridan, Margaret A.
Silvers, Jennifer A.
Skagerlund, Kenny
Smith, Alec
Smith, David V.
Sokol-Hessner, Peter
Steinkamp, Simon R.
Tashjian, Sarah M.
Thirion, Bertrand
Thorp, John N.
Tinghög, Gustav
Tisdall, Loreen
Tompson, Steven H.
Toro-Serey, Claudio
Torre Tresols, Juan Jesus
Tozzi, Leonardo
Truong, Vuong
Turella, Luca
van 't Veer, Anna E.
Verguts, Tom
Vettel, Jean M.
Vijayarajah, Sagana
Vo, Khoi
Wall, Matthew B.
Weeda, Wouter D.
Weis, Susanne
White, David J.
Wisniewski, David
Xifra-Porxas, Alba
Yearling, Emily A.
Yoon, Sangsuk
Yuan, Rui
Yuen, Kenneth S. L.
Zhang, Lei ;  Université de Liège - ULiège > Terra
Zhang, Xu
Zosky, Joshua E.
Nichols, Thomas E.
Poldrack, Russell A.
Schonberg, Tom
More authors (187 more) Less
Language :
English
Title :
Variability in the analysis of a single neuroimaging dataset by many teams.
Publication date :
2020
Journal title :
Nature
ISSN :
0028-0836
eISSN :
1476-4687
Publisher :
Nature Publishing Group, London, United Kingdom
Volume :
582
Issue :
7810
Pages :
84-88
Peer reviewed :
Peer Reviewed verified by ORBi

Statistics


Number of views
397 (6 by ULiège)
Number of downloads
147 (2 by ULiège)

Scopus citations®
 
314
Scopus citations®
without self-citations
257
OpenCitations
 
370

Bibliography


Similar publications



Contact ORBi