Attenzione: i dati modificati non sono ancora stati salvati. Per confermare inserimenti o cancellazioni di voci è necessario confermare con il tasto SALVA/INSERISCI in fondo alla pagina
IRIS
Data analysis workflows in many scientific domains have become increasingly
complex and flexible. Here we assess the effect of this flexibility on the results of
functional magnetic resonance imaging by asking 70 independent teams to analyse
the same dataset, testing the same 9 ex-ante hypotheses1. The flexibility of analytical
approaches is exemplified by the fact that no two teams chose identical workflows to
analyse the data. This flexibility resulted in sizeable variation in the results of
hypothesis tests, even for teams whose statistical maps were highly correlated at
intermediate stages of the analysis pipeline. Variation in reported results was related
to several aspects of analysis methodology. Notably, a meta-analytical approach that
aggregated information across teams yielded a significant consensus in activated
regions. Furthermore, prediction markets of researchers in the field revealed an
overestimation of the likelihood of significant findings, even by researchers with
direct knowledge of the dataset2–5. Our findings show that analytical flexibility can
have substantial effects on scientific conclusions, and identify factors that may be
related to variability in the analysis of functional magnetic resonance imaging. The
results emphasize the importance of validating and sharing complex analysis
workflows, and demonstrate the need for performing and reporting multiple analyses
of the same data. Potential approaches that could be used to mitigate issues related to
analytical variability are discussed.
Variability in the analysis of a single neuroimaging dataset by many teams / Botvinik-Nezer, Rotem; Holzmeister, Felix; Camerer, Colin F.; Dreber, Anna; Huber, Juergen; Johannesson, Magnus; Kirchler, Michael; Iwanir, Roni; Mumford, Jeanette A.; Adcock, R. Alison; Avesani, Paolo; Baczkowski, Blazej M.; Bajracharya, Aahana; Bakst, Leah; Ball, Sheryl; Barilari, Marco; Bault, Nadège; Beaton, Derek; Beitner, Julia; Benoit, Roland G.; Berkers, Ruud M. W. J.; Bhanji, Jamil P.; Biswal, Bharat B.; Bobadilla-Suarez, Sebastian; Bortolini, Tiago; Bottenhorn, Katherine L.; Bowring, Alexander; Braem, Senne; Brooks, Hayley R.; Brudner, Emily G.; Calderon, Cristian B.; Camilleri, Julia A.; Castrellon, Jaime J.; Cecchetti, Luca; Cieslik, Edna C.; Cole, Zachary J.; Collignon, Olivier; Cox, Robert W.; Cunningham, William A.; Czoschke, Stefan; Dadi, Kamalaker; Davis, Charles P.; Luca, Alberto De; Delgado, Mauricio R.; Demetriou, Lysia; Dennison, Jeffrey B.; Di, Xin; Dickie, Erin W.; Dobryakova, Ekaterina; Donnat, Claire L.; Dukart, Juergen; Duncan, Niall W.; Durnez, Joke; Eed, Amr; Eickhoff, Simon B.; Erhart, Andrew; Fontanesi, Laura; Fricke, G. Matthew; Fu, Shiguang; Galván, Adriana; Gau, Remi; Genon, Sarah; Glatard, Tristan; Glerean, Enrico; Goeman, Jelle J.; Golowin, Sergej A. E.; González-García, Carlos; Gorgolewski, Krzysztof J.; Grady, Cheryl L.; Green, Mikella A.; Guassi Moreira, João F.; Guest, Olivia; Hakimi, Shabnam; Hamilton, J. Paul; Hancock, Roeland; Handjaras, Giacomo; Harry, Bronson B.; Hawco, Colin; Herholz, Peer; Herman, Gabrielle; Heunis, Stephan; Hoffstaedter, Felix; Hogeveen, Jeremy; Holmes, Susan; Hu, Chuan-Peng; Huettel, Scott A.; Hughes, Matthew E.; Iacovella, Vittorio; Iordan, Alexandru D.; Isager, Peder M.; Isik, Ayse I.; Jahn, Andrew; Johnson, Matthew R.; Johnstone, Tom; Joseph, Michael J. E.; Juliano, Anthony C.; Kable, Joseph W.; Kassinopoulos, Michalis; Koba, Cemal; Kong, Xiang-Zhen; Koscik, Timothy R.; Kucukboyaci, Nuri Erkut; Kuhl, Brice A.; Kupek, Sebastian; Laird, Angela R.; Lamm, Claus; Langner, Robert; Lauharatanahirun, Nina; Lee, Hongmi; Lee, Sangil; Leemans, Alexander; Leo, Andrea; Lesage, Elise; Li, Flora; Li, Monica Y. C.; Lim, Phui Cheng; Lintz, Evan N.; Liphardt, Schuyler W.; Losecaat Vermeer, Annabel B.; Love, Bradley C.; Mack, Michael L.; Malpica, Norberto; Marins, Theo; Maumet, Camille; Mcdonald, Kelsey; Mcguire, Joseph T.; Melero, Helena; Méndez Leal, Adriana S.; Meyer, Benjamin; Meyer, Kristin N.; Mihai, Glad; Mitsis, Georgios D.; Moll, Jorge; Nielson, Dylan M.; Nilsonne, Gustav; Notter, Michael P.; Olivetti, Emanuele; Onicas, Adrian I.; Papale, Paolo; Patil, Kaustubh R.; Peelle, Jonathan E.; Pérez, Alexandre; Pischedda, Doris; Poline, Jean-Baptiste; Prystauka, Yanina; Ray, Shruti; Reuter-Lorenz, Patricia A.; Reynolds, Richard C.; Ricciardi, Emiliano; Rieck, Jenny R.; Rodriguez-Thompson, Anais M.; Romyn, Anthony; Salo, Taylor; Samanez-Larkin, Gregory R.; Sanz-Morales, Emilio; Schlichting, Margaret L.; Schultz, Douglas H.; Shen, Qiang; Sheridan, Margaret A.; Silvers, Jennifer A.; Skagerlund, Kenny; Smith, Alec; Smith, David V.; Sokol-Hessner, Peter; Steinkamp, Simon R.; Tashjian, Sarah M.; Thirion, Bertrand; Thorp, John N.; Tinghög, Gustav; Tisdall, Loreen; Tompson, Steven H.; Toro-Serey, Claudio; Torre Tresols, Juan Jesus; Tozzi, Leonardo; Truong, Vuong; Turella, Luca; van ‘t Veer, Anna E.; Verguts, Tom; Vettel, Jean M.; Vijayarajah, Sagana; Vo, Khoi; Wall, Matthew B.; Weeda, Wouter D.; Weis, Susanne; White, David J.; Wisniewski, David; Xifra-Porxas, Alba; Yearling, Emily A.; Yoon, Sangsuk; Yuan, Rui; Yuen, Kenneth S. L.; Zhang, Lei; Zhang, Xu; Zosky, Joshua E.; Nichols, Thomas E.; Poldrack, Russell A.; Schonberg, Tom. - In: NATURE. - ISSN 0028-0836. - STAMPA. - 2020:582(2020), pp. 84-88. [10.1038/s41586-020-2314-9]
Variability in the analysis of a single neuroimaging dataset by many teams
Botvinik-Nezer, Rotem;Holzmeister, Felix;Camerer, Colin F.;Dreber, Anna;Huber, Juergen;Johannesson, Magnus;Kirchler, Michael;Iwanir, Roni;Mumford, Jeanette A.;Adcock, R. Alison;Avesani, Paolo;Baczkowski, Blazej M.;Bajracharya, Aahana;Bakst, Leah;Ball, Sheryl;Barilari, Marco;Bault, Nadège;Beaton, Derek;Beitner, Julia;Benoit, Roland G.;Berkers, Ruud M. W. J.;Bhanji, Jamil P.;Biswal, Bharat B.;Bobadilla-Suarez, Sebastian;Bortolini, Tiago;Bottenhorn, Katherine L.;Bowring, Alexander;Braem, Senne;Brooks, Hayley R.;Brudner, Emily G.;Calderon, Cristian B.;Camilleri, Julia A.;Castrellon, Jaime J.;Cecchetti, Luca;Cieslik, Edna C.;Cole, Zachary J.;Collignon, Olivier;Cox, Robert W.;Cunningham, William A.;Czoschke, Stefan;Dadi, Kamalaker;Davis, Charles P.;Luca, Alberto De;Delgado, Mauricio R.;Demetriou, Lysia;Dennison, Jeffrey B.;Di, Xin;Dickie, Erin W.;Dobryakova, Ekaterina;Donnat, Claire L.;Dukart, Juergen;Duncan, Niall W.;Durnez, Joke;Eed, Amr;Eickhoff, Simon B.;Erhart, Andrew;Fontanesi, Laura;Fricke, G. Matthew;Fu, Shiguang;Galván, Adriana;Gau, Remi;Genon, Sarah;Glatard, Tristan;Glerean, Enrico;Goeman, Jelle J.;Golowin, Sergej A. E.;González-García, Carlos;Gorgolewski, Krzysztof J.;Grady, Cheryl L.;Green, Mikella A.;Guassi Moreira, João F.;Guest, Olivia;Hakimi, Shabnam;Hamilton, J. Paul;Hancock, Roeland;Handjaras, Giacomo;Harry, Bronson B.;Hawco, Colin;Herholz, Peer;Herman, Gabrielle;Heunis, Stephan;Hoffstaedter, Felix;Hogeveen, Jeremy;Holmes, Susan;Hu, Chuan-Peng;Huettel, Scott A.;Hughes, Matthew E.;Iacovella, Vittorio;Iordan, Alexandru D.;Isager, Peder M.;Isik, Ayse I.;Jahn, Andrew;Johnson, Matthew R.;Johnstone, Tom;Joseph, Michael J. E.;Juliano, Anthony C.;Kable, Joseph W.;Kassinopoulos, Michalis;Koba, Cemal;Kong, Xiang-Zhen;Koscik, Timothy R.;Kucukboyaci, Nuri Erkut;Kuhl, Brice A.;Kupek, Sebastian;Laird, Angela R.;Lamm, Claus;Langner, Robert;Lauharatanahirun, Nina;Lee, Hongmi;Lee, Sangil;Leemans, Alexander;Leo, Andrea;Lesage, Elise;Li, Flora;Li, Monica Y. C.;Lim, Phui Cheng;Lintz, Evan N.;Liphardt, Schuyler W.;Losecaat Vermeer, Annabel B.;Love, Bradley C.;Mack, Michael L.;Malpica, Norberto;Marins, Theo;Maumet, Camille;McDonald, Kelsey;McGuire, Joseph T.;Melero, Helena;Méndez Leal, Adriana S.;Meyer, Benjamin;Meyer, Kristin N.;Mihai, Glad;Mitsis, Georgios D.;Moll, Jorge;Nielson, Dylan M.;Nilsonne, Gustav;Notter, Michael P.;Olivetti, Emanuele;Onicas, Adrian I.;Papale, Paolo;Patil, Kaustubh R.;Peelle, Jonathan E.;Pérez, Alexandre;Pischedda, Doris;Poline, Jean-Baptiste;Prystauka, Yanina;Ray, Shruti;Reuter-Lorenz, Patricia A.;Reynolds, Richard C.;Ricciardi, Emiliano;Rieck, Jenny R.;Rodriguez-Thompson, Anais M.;Romyn, Anthony;Salo, Taylor;Samanez-Larkin, Gregory R.;Sanz-Morales, Emilio;Schlichting, Margaret L.;Schultz, Douglas H.;Shen, Qiang;Sheridan, Margaret A.;Silvers, Jennifer A.;Skagerlund, Kenny;Smith, Alec;Smith, David V.;Sokol-Hessner, Peter;Steinkamp, Simon R.;Tashjian, Sarah M.;Thirion, Bertrand;Thorp, John N.;Tinghög, Gustav;Tisdall, Loreen;Tompson, Steven H.;Toro-Serey, Claudio;Torre Tresols, Juan Jesus;Tozzi, Leonardo;Truong, Vuong;Turella, Luca;van ‘t Veer, Anna E.;Verguts, Tom;Vettel, Jean M.;Vijayarajah, Sagana;Vo, Khoi;Wall, Matthew B.;Weeda, Wouter D.;Weis, Susanne;White, David J.;Wisniewski, David;Xifra-Porxas, Alba;Yearling, Emily A.;Yoon, Sangsuk;Yuan, Rui;Yuen, Kenneth S. L.;Zhang, Lei;Zhang, Xu;Zosky, Joshua E.;Nichols, Thomas E.;Poldrack, Russell A.;Schonberg, Tom
2020-01-01
Abstract
Data analysis workflows in many scientific domains have become increasingly
complex and flexible. Here we assess the effect of this flexibility on the results of
functional magnetic resonance imaging by asking 70 independent teams to analyse
the same dataset, testing the same 9 ex-ante hypotheses1. The flexibility of analytical
approaches is exemplified by the fact that no two teams chose identical workflows to
analyse the data. This flexibility resulted in sizeable variation in the results of
hypothesis tests, even for teams whose statistical maps were highly correlated at
intermediate stages of the analysis pipeline. Variation in reported results was related
to several aspects of analysis methodology. Notably, a meta-analytical approach that
aggregated information across teams yielded a significant consensus in activated
regions. Furthermore, prediction markets of researchers in the field revealed an
overestimation of the likelihood of significant findings, even by researchers with
direct knowledge of the dataset2–5. Our findings show that analytical flexibility can
have substantial effects on scientific conclusions, and identify factors that may be
related to variability in the analysis of functional magnetic resonance imaging. The
results emphasize the importance of validating and sharing complex analysis
workflows, and demonstrate the need for performing and reporting multiple analyses
of the same data. Potential approaches that could be used to mitigate issues related to
analytical variability are discussed.
I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione
Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11572/266254
Citazioni
309
487
458
social impact
Conferma cancellazione
Sei sicuro che questo prodotto debba essere cancellato?
simulazione ASN
Il report seguente simula gli indicatori relativi alla propria produzione scientifica in relazione alle soglie ASN 2023-2025 del proprio SC/SSD. Si ricorda che il superamento dei valori soglia (almeno 2 su 3) è requisito necessario ma non sufficiente al conseguimento dell'abilitazione. La simulazione si basa sui dati IRIS e sugli indicatori bibliometrici alla data indicata e non tiene conto di eventuali periodi di congedo obbligatorio, che in sede di domanda ASN danno diritto a incrementi percentuali dei valori. La simulazione può differire dall'esito di un’eventuale domanda ASN sia per errori di catalogazione e/o dati mancanti in IRIS, sia per la variabilità dei dati bibliometrici nel tempo. Si consideri che Anvur calcola i valori degli indicatori all'ultima data utile per la presentazione delle domande.
La presente simulazione è stata realizzata sulla base delle specifiche raccolte sul tavolo ER del Focus Group IRIS coordinato dall’Università di Modena e Reggio Emilia e delle regole riportate nel DM 589/2018 e allegata Tabella A. Cineca, l’Università di Modena e Reggio Emilia e il Focus Group IRIS non si assumono alcuna responsabilità in merito all’uso che il diretto interessato o terzi faranno della simulazione. Si specifica inoltre che la simulazione contiene calcoli effettuati con dati e algoritmi di pubblico dominio e deve quindi essere considerata come un mero ausilio al calcolo svolgibile manualmente o con strumenti equivalenti.