[en] An increasing number of studies have investigated the relationships between inter-individual variability in brain regions' connectivity and behavioral phenotypes, making use of large population neuroimaging datasets. However, the replicability of brain-behavior associations identified by these approaches remains an open question. In this study, we examined the cross-dataset replicability of brain-behavior association patterns for fluid cognition and openness predictions using a previously developed region-wise approach, as well as using a standard whole-brain approach. Overall, we found moderate similarity in patterns for fluid cognition predictions across cohorts, especially in the Human Connectome Project Young Adult, Human Connectome Project Aging, and Enhanced Nathan Kline Institute Rockland Sample cohorts, but low similarity in patterns for openness predictions. In addition, we assessed the generalizability of prediction models in cross-dataset predictions, by training the model in one dataset and testing in another. Making use of the region-wise prediction approach, we showed that first, a moderate extent of generalizability could be achieved with fluid cognition prediction, and that, second, a set of common brain regions related to fluid cognition across cohorts could be identified. Nevertheless, the moderate replicability and generalizability could only be achieved in specific contexts. Thus, we argue that replicability and generalizability in connectivity-based prediction remain limited and deserve greater attention in future studies.
Disciplines :
Neurosciences & behavior
Author, co-author :
Wu, Jianxiao; Institute for Systems Neuroscience, Medical Faculty, Heinrich-Heine University
Li, Jingwei; Institute for Systems Neuroscience, Medical Faculty, Heinrich-Heine University
Eickhoff, Simon B; Institute for Systems Neuroscience, Medical Faculty, Heinrich-Heine University
Hoffstaedter, Felix; Institute for Systems Neuroscience, Medical Faculty, Heinrich-Heine University
Hanke, Michael; Institute for Systems Neuroscience, Medical Faculty, Heinrich-Heine University
Yeo, B T Thomas; Department of Electrical and Computer Engineering, National University of
Genon, Sarah ; Université de Liège - ULiège > Département des sciences cliniques > Neuroimagerie des troubles de la mémoire et revalidation cognitive ; Institute for Systems Neuroscience, Medical Faculty, Heinrich-Heine University
Language :
English
Title :
Cross-cohort replicability and generalizability of connectivity-based psychometric prediction patterns.
Arslan, S., Ktena, S.I., Makropoulos, A., Robinson, E.C., Rueckert, D., Parisot, S., Human brain mapping: a systematic comparison of parcellation methods for the human cerebral cortex. Neuroimage 170 (2018), 5–30, 10.1016/j.neuroimage.2017.04.014.
Avery, E.W., Yoo, K., Rosenberg, M.D., Greene, A.S., Gao, S., Na, D.L., Scheinost, D., Constable, T.R., Chun, M.M., Distributed patterns of functional connectivity predict working memory performance in novel healthy and memory-impaired individuals. J. Cogn. Neurosci. 32 (2020), 241–255, 10.1162/jocn_a_01487.
Barbey, A.K., Colom, R., Paul, E.J., Grafman, J., Architecture of fluid intelligence and working memory revealed by lesion mapping. Brain Struct. Funct. 219 (2013), 485–494, 10.1007/s00429-013-0512-z.
Beaty, R.E., Kenett, Y.N., Christensen, A.P., Rosenberg, M.D., Benedek, M., Chen, Q., Fink, A., Qui, J., Kwapil, T.R., Kane, M.J., Silvia, P.J., Robust prediction of individual creative ability from brain functional connectivity. Proc. Natl. Acad. Sci. U. S. A. 115 (2018), 1087–1092, 10.1073/pnas.1713532115.
Benjamini, Y., Hochberg, Y., Controlling the false discovery rate: a practical and powerful approach to multiple testing. J. R. Stat. Soc. 59 (1995), 289–300 http://doi.wiley.com/10.1111/j.2517-6161.1995.tb02031.x.
Bilker, W.B., Hansen, J.A., Brensinger, C.M., Richard, J., Gur, R.E., Gur, R.C., Development of abbreviated nine-item forms of the Raven's standard progressive matrices test. Assessment 19 (2012), 354–369, 10.1177/1073191112446655.
Bookheimer, S.Y., Salat, D.H., Terpstra, M., Ances, B.M., Barch, D.M., Buckner, R.L., Burgess, G.C., Curtiss, S.W., Diaz-Santos, M., Elam, J.S., et al. The lifespan Human Connectome Project in aging: an overview. Neuroimage 185 (2019), 335–348, 10.1016/j.neuroimage.2018.10.009.
Boser, E.B., Guyon, I.M., Vapnik, V.N., A training algorithm for optimal margin classifiers. Proceedings of the 5th Annual Workshop on Computational Learning Theory, 1992, 144–152, 10.1145/130385.130401.
Caruso, J.C., Reliability generalization of the NEO personality scales. Educ. Psychol. Meas. 60 (2000), 236–254, 10.1177/00131640021970484.
Caspers, S., Moebus, S., Lux, S., Pundt, N., Schütz, H., Mühleisen, T.W., Gras, V., Eickhoff, S.B., Romanzetti, S., Stöcker, T., et al. Studying variability in human brain aging in a population-based German cohort—rationale and design of 1000BRAINS. Front. Aging Neurosci., 6, 2014, 149, 10.3389/fnagi.2014.00149.
Chen, J., Mueller, V.I., Dukart, J., Hoffstaedter, F., Baker, J.T., Holmes, A.J., Vatansever, D., Nickl-Jockschat, T., Liu, X., Derntl, B., et al. Intrinsic connectivity patterns of task-defined brain networks allow individual prediction of cognitive symptom dimension of schizophrenia and are linked to molecular architecture. Biol. Psychiatry 89 (2020), 308–319, 10.1016/j.biopsych.2020.09.024.
Chen, J., Tam, A., Kebets, V., Orban, C., Ooi, L.Q.R., Asplund, C.L., Marek, S., Dosenbach, N.U.F., Eickhoff, S.B., Bzdok, D., Holmes, A.J., Yeo, B.T.T., Shared and unique brain network features predict cognitive, personality, and mental health scores in the ABCD study. Nat. Commun., 13, 2022, 2217, 10.1038/s41467-022-29766-8.
Christov-Moore, L., Reggente, N., Douglas, P.K., Feusner, J.D., Iacoboni, M., Predicting empathy from resting state brain connectivity: a multivariate approach. Front. Integr. Neurosci., 14, 2020, 3, 10.3389/fnint.2020.00003.
Deary, I.J., Weiss, A., Batty, D., Intelligence and personality as predictors of illness and death: how researchers in differential psychology and chronic disease epidemiology are collaborating to understand and address health inequalities. Psychol. Sci. Public Interest 11 (2011), 53–79, 10.1177/1529100610387081.
Dubois, J., Galdi, P., Han, Y., Paul, L.K., Adolphs, R., Resting-state functional brain connectivity best predicts personality dimension of openness to experience. Personal. Neurosci., 1, 2018, E6, 10.1017/pen.2018.8.
Dubois, J., Galdi, P., Paul, L.K., Adolphs, R., A distributed brain network predicts general intelligence from resting-state human neuroimaging data. Philos. Trans. R. Soc. B Biol. Sci., 373, 2018, 20170284, 10.1098/rstb.2017.0284.
Ebisch, S.J., Perrucci, M.G., Mercuri, P., Romanelli, R., Mantini, D., Romani, G.L., Colom, R., Saggino, A., Common and unique neuro-functional basis of induction, visualization, and spatial relationships as cognitive components of fluid intelligence. Neuroimage 62 (2012), 331–342, 10.1016/j.neuroimage.2012.04.053.
Egan, V., Deary, I., Austin, E., The NEO-FFI: emerging British norms and an item-level analysis suggest N, A and C are more reliable than O and E. Personal. Individ. Differ. 29 (2000), 907–920, 10.1016/S0191-8869(99)00242-1.
Esteban, O., Markiewicz, C.J., Blair, R.W., Moodie, C.A., Isik, A.I., Erramuzpe, A., Kent, J.D., Goncalves, M., DuPre, E., Snyder, M., et al. fMRIPrep—a robust preprocessing pipeline for functional MRI. Nat. Methods 16 (2019), 111–116, 10.1038/s41592-018-0235-4.
Glasser, M.F., Sotiropoulos, S.N., Wilson, J.A., Coalson, T.S., Fischl, B., Andersson, J.L., Xu, J., Jbabdi, S., Webster, M., Polimeni, J.R., et al. The minimal preprocessing pipeline for the Human Connectome Project. Neuroimage 80 (2013), 105–124, 10.1016/j.neuroimage.2013.04.127.
Gray, J.R., Chabris, C.F., Braver, T.S., Neural mechanisms of general fluid intelligence. Nat. Neurosci. 6 (2003), 316–322, 10.1038/nn1014.
Griffanti, L., Salimi-Khorshidi, G., Beckmann, C.F., Auerbach, E.J., Douaud, G., Sexton, C.E., Zsoldos, E., Ebmeier, K.P., Filippini, N., Mackay, C.E., et al. ICA-based artefact removal and accelerated fMRI acquisition for improved resting state network imaging. Neuroimage 95 (2014), 232–247, 10.1016/j.neuroimage.2014.03.034.
Halchenko, Y.O., Meyer, K., Poldrack, B., Solanky, D.S., Wagner, A.S., Gors, J., MacFarlane, D., Pustina, D., Sochat, V., Ghosh, S.S., et al. DataLad: distributed system for joint management of code, data, and their relationship. J. Open Source Softw., 6, 2021, 3262 https://joss.theoj.org/papers/10.21105/joss.03262.
Haufe, S., Meinecke, F., Görgen, K., Dähne, S., Haynes, J., Blankertz, B., Bießmann, F., On the interpretation of weight vectors of linear models in multivariate neuroimaging. Neuroimage 87 (2014), 96–110, 10.1016/j.neuroimage.2013.10.067.
Harms, M.P., Somerville, L.H., Ances, B.M., Andersson, J., Barch, D.M., Bastiani, M., Bookheimer, S.Y., Brown, T.B., Buckner, R.L., Burgess, G.C., Extending the Human Connectome Project across ages: imaging protocols for the lifespan development and aging projects. Neuroimage 183 (2018), 972–984, 10.1016/j.neuroimage.2018.09.060.
He, T., Kong, R., Holmes, A.J., Nguyen, M., Sabuncu, M.R., Eickhoff, S.B., Bzdok, D., Feng, J., Yeo, B.T.T., Deep neural networks and kernel regression achieve comparable accuracies for functional connectivity prediction of behavior and demographics. Neuroimage, 206, 2020, 116276, 10.1016/j.neuroimage.2019.116276.
Horien, C., Greene, A.S., Constable, T., Scheinost, D., Regions and connections: complementary approaches to characterize brain organization and function. Neuroscientist 26 (2019), 117–133, 10.1177/1073858419860115.
Humphreys, M.S., Revelle, W., Personality, motivation, and performance: a theory for the relationship between individual differences and information processing. Psychol. Rev. 91 (1984), 153–184, 10.1037/0033-295X.91.2.153.
Jain, A.K., Waller, W.G., On the optimal number of features in the classification of multivariate Gaussian data. Pattern Recognit. 10 (1978), 365–374, 10.1016/0031-3203(78)90008-0.
Jiang, R., Calhoun, V.D., Zuo, N., Lin, D., Li, J., Fan, L., Qi, S., sun, H., Fu, Z., Song, M., et al. Connectome-based individualized prediction of temperament trait scores. Neuroimage 183 (2018), 366–374, 10.1016/j.neuroimage.2018.08.038.
Jiang, R., Calhoun, V., Fan, L., Zuo, N., Jung, R., Qi, S., Lin, D., Li, J., Zhuo, C., Song, M., Gender differences in connectome-based predictions of individualized intelligence quotient and sub-domain scores. Cereb. Cortex 30 (2020), 888–900, 10.1093/cercor/bhz134.
Joliot, M., Jobard, G., Naveau, M., Delcroix, N., Petit, L., Zago, L., Crivello, F., Mellet, E., Mayzoyer, B., Tzourio-Mazoyer, N., AICHA: an atlas of intrinsic connectivity of homotopic areas. J. Neurosci. Methods 254 (2015), 46–59, 10.1016/j.jneumeth.2015.07.013.
Kane, M.J., Engle, R.W., The role of prefrontal cortex in working-memory capacity, executive attention, and general fluid intelligence: an individual-differences perspective. Psychon. Bull. Rev. 9 (2002), 637–671, 10.3758/BF03196323.
Kong, R., Yang, Q., Gordon, E., Xue, A., Yan, X., Orban, C., Zuo, X., Spreng, N., Ge, T., Holmes, A., et al. Individual-specific areal-level parcellations improve functional connectivity prediction of behavior. Cereb. Cortex 31 (2021), 4477–4500, 10.1093/cercor/bhab101.
Kwon, Y.H., Yoo, K., Nguyen, H., Jeong, Y., Chung, M.M., Predicting multilingual effects on executive function and individual connectomes in children: an ABCD study. Proc. Natl. Acad. Sci. U. S. A., 118, 2021, e2110811118, 10.1073/pnas.2110811118.
Li, J., Kong, R., Liegeois, R., Orban, C., Tan, Y., Sun, N., Holmes, A., Sabuncu, M.R., Ge, T., Yeo, B.T.T., Global signal regression strengthens association between resting-state functional connectivity and behavior. Neuroimage 196 (2019), 126–141, 10.1016/j.neuroimage.2019.04.016.
Maglanoc, L.A., Kaufmann, T., van der Meer, D., Marquand, A.F., Wolfers, T., Jonassen, R., Hilland, E., Andreassen, O.A., Landrø, N.I., Westlye, L.T., Brain connectome mapping of complex human traits and their polygenic architecture using machine learning. Biol. Psychol. 87 (2019), 717–726, 10.1016/j.biopsych.2019.10.011.
McCrae, R.R., Costa, P.T., A contemplated revision of the NEO Five-Factor Inventory. Personal. Individ. Differ. 36 (2004), 587–596, 10.1016/S0191-8869(03)00118-1.
Mueller, S., Wang, D., Fox, M.D., Yeo, B.T.T., Sepulcre, J., Sabuncu, M.R., Shafee, R., Lu, J., Liu, H., Individual variability in functional connectivity architecture of the human brain. Neuron 77 (2012), 586–595, 10.1016/j.neuron.2012.12.028.
Mueller, S., Wang, D., Fox, M.D., Pan, R., Lu, J., Li, K., Sun, W., Buckner, R.L., Liu, H., Reliability correction for functional connectivity: theory and implementation. Hum. Brain Mapp. 36 (2015), 4664–4680, 10.1002/hbm.22947.
Murray, G., Rawlings, D., Allen, N.B., Trinder, J., NEO Five-Factor Inventory scores: psychometric properties in a community sample. Meas. Eval. Couns. Dev. 36 (2003), 140–149, 10.1080/07481756.2003.11909738.
Noble, S., Spann, M.N., Tokoglu, F., Shen, X., Constable, R.T., Scheinost, D., Influences on the test-retest reliability of functional connectivity MRI and its relationship with behavioral utility. Cereb. Cortex 27 (2017), 5415–5429, 10.1093/cercor/bhx230.
Noble, S., Scheinost, D., Constable, R.T., A decade of test-retest reliability of functional connectivity: a systematic review and meta-analysis. Neuroimage, 203, 2019, 116157, 10.1016/j.neuroimage.2019.116157.
Nooner, K.B., Colcombe, S.J., Tobe, R.H., Mennes, M., Benedict, M.M., Moreno, A.L., Panek, L.J., Brown, S., Zavitz, S.T., Li, Q., et al. The NKI-Rockland Sample: a model for accelerating the pace of discovery science in psychiatry. Front. Neurosci., 6, 2012, 152, 10.3389/fnins.2012.00152.
Nostro, A.D., Mueller, V., Varikuti, D., Plaeschke, R., Hoffstaedter, F., Langner, R., Patil, K., Eickhoff, S.B., Predicting personality from network-based resting-state functional connectivity. Brain Struct. Funct. 223 (2018), 2699–2719, 10.1007/s00429-018-1651-z.
O'Connor, D., Lake, E.M.R., Scheinost, D., Constable, R.T., Resample aggregating improves the generalizability of Connectome Predictive Modelling. Neuroimage, 2021, 118044, 10.1016/j.neuroimage.2021.118044.
Pervaiz, U., Vidaurre, D., Woolrich, M.W., Smith, S.M., Optimising network modelling methods for fMRI. Neuroimage, 221, 2020, 116604, 10.1016/j.neuroimage.2020.116604.
Plaeschke, R.N., Patil, K.R., Cieslik, E.C., Nostro, A.D., Varikuti, D.P., Plachti, A., Losche, P., Hoffstaedter, F., Langner, R., Eickhoff, S.B., Age differences in predicting working memory performance from network-based functional connectivity. Cortex 132 (2020), 441–459, 10.1016/j.cortex.2020.08.012.
Preusse, F., van der Meer, E., Deshpande, G., Krueger, F., Wartenburger, I., Fluid intelligence allows flexible recruitment of the parieto-frontal network in analogical reasoning. Front. Hum. Neurosci., 5, 2011, 22, 10.3389/fnhum.2011.00022.
Pruim, R.H.R., Mennes, M., van Rooij, Daan, Llera, A., Buitelaar, J.K., Beckmann, C.F., ICA-AROMA: a robust ICA-based strategy for removing motion artifacts from fMRI data. Neuroimage 112 (2015), 267–277, 10.1016/j.neuroimage.2015.02.064.
Pruim, R.H.R., Mennes, M., Buitelaar, J.K., Beckmann, C.F., Evaluation of ICA-AROMA and alternative strategies for motion artifact removal in resting state fMRI. Neuroimage 112 (2015), 278–287, 10.1016/j.neuroimage.2015.02.063.
Qian, J., Hastie, T., Friedman, J., Tibshirani, R., Simon, N., 2013. Glmnet for Matlab. http://www.stanford.edu/∼hastie/glmnet_matlab (last accessed 15 March 2019).
Rosa, M., Parr, A., Thompson, R., Woolgar, A., Torralva, T., Antoun, N., Manes, F., Duncan, J., Executive function and fluid intelligence after frontal lobe lesions. Brain 133 (2010), 234–247, 10.1093/brain/awp269.
Rosenberg, M.D., Finn, E.S., Scheinost, D., Papademetris, X., Shen, X., Constable, R.T., Chun, M.M., A neuromarker of sustained attention from whole-brain functional connectivity. Nat. Neurosci. 19 (2016), 165–171, 10.1038/nn.4179.
Santarnecchi, E., Momi, D., Mencarelli, L., Plessow, F., Saxena, S., Rossi, S., Rossi, A., Mathan, S., Pascual-Leone, A., Overlapping and dissociable brain activations for fluid intelligence and executive functions. Cogn. Affect. Behav. Neurosci. 21 (2021), 327–346, 10.3758/s13415-021-00870-4.
Schaefer, A., Kong, R., Gordon, E.M., Laumann, T.O., Zuo, X., Holmes, A.J., Eickhoff, S.B., Yeo, B.T.T., Local-global parcellation of the human cerebral cortex from intrinsic functional connectivity MRI. Cereb. Cortex 28 (2018), 3095–3114, 10.1093/cercor/bhx179.
Shah, L.M., Cramer, J.A., Ferguson, M.A., Birn, R.M., Anderson, J.S., Reliability and reproducibility of individual differences in functional connectivity acquired during task and resting state. Brain Behav., 6, 2016, e00456, 10.1002/brb3.456.
Smith, S.M., Beckmann, C.F., Andersson, J., Auerbach, E., Bijsterbosch, J., Dounaud, G., Duff, E., Feinberg, D.A., Griffanti, L., Harms, M.P., et al. Resting-state fMRI in the Human Connectome Project. Neuroimage 80 (2013), 144–168, 10.1016/j.neuroimage.2013.05.039.
Smith, S.M., Vidaurre, D., Glasser, M., Winkler, A., McCarthy, P., Robinson, E., Chen, X., Horton, W., Jenkinson, M., Duff, E., et al., 2016. Second beta-release oft he HCP functional connectivity MegaTrawl. Available at: http://db.humanconnectome.org/megatraw (Accessed: 15 Mar 2019).
Speer, S.P.H., Smidts, A., Boksem, M.A.S., Individual differences in (dis)honesty are represented in the brain's functional connectivity at rest. Neuroimage, 246, 2022, 118761, 10.1016/j.neuroimage.2021.118761.
Sui, J., Jiang, R., Bustillo, J., Calhoun, V., Neuroimaging-based individualized prediction of cognition and behavior for mental disorders and health: methods and premises. Biol. Psychol. 88 (2020), 818–828, 10.1016/j.biopsych.2020.02.016.
Tian, Y., Margulies, D.S., Breakspear, M., Zalesky, A., Topographic organization of the human subcortex unveiled with functional connectivity gradients. Nat. Neurosci. 23 (2020), 1421–1432, 10.1038/s41593-020-00711-6.
Tian, Y., Zalesky, A., Machine learning prediction of cognition from functional connectivity: are feature weights reliable?. Neuroimage, 245, 2021, 118648, 10.1016/j.neuroimage.2021.118648.
Turner, B.O., Paul, E.J., Miller, M.B., Barbey, A.K., Small sample sizes reduce the replicability of task-based fMRI studies. Commun. Biol., 1, 2018, 62, 10.1038/s42003-018-0073-z.
van den Heuvel, M.P., Sporns, O., Network hubs in the human brain. Trends Cogn. Sci. 17 (2013), 683–696, 10.1016/j.tics.2013.09.012.
Van Essen, D.C., Smith, S.M., Barch, D.M., Behrens, T.E., Yacoub, E., Ugurbil, K., Consortium, for the WU-Minn HCP, The WU-Minn Human Connectome Project: an overview. Neuroimage 80 (2013), 62–79, 10.1016/j.neuroimage.2013.05.041.
Varikuti, D.P., Genon, S., Sotiras, A., Schwender, H., Hoffstaedter, F., Patil, K.R., Jockwitz, C., Caspers, S., Moebus, S., Amunts, K., Evaluation of non-negative matrix factorization of grey matter in age prediction. Neuroimage 173 (2018), 394–410, 10.1016/j.neuroimage.2018.03.007.
Yeung, A.W.K., More, S., Wu, J., Eickhoff, S.B., Reporting details of neuroimaging studies on individual traits prediction: a literature survey. Neuroimage, 119275, 2022, 10.1016/j.neuroimage.2022.119275.
Zou, H., Hastie, T., Regularization and variable selection via the elastic net. J. R. Stat. Soc. 67 (2005), 301–320 http://doi.wiley.com/10.1111/j.1467-9868.2005.00503.x.
