[en] While a replicability crisis has shaken psychological sciences, the replicability of multivariate approaches for psychometric data factorization has received little attention. In particular, Exploratory Factor Analysis (EFA) is frequently promoted as the gold standard in psychological sciences. However, the application of EFA to executive functioning, a core concept in psychology and cognitive neuroscience, has led to divergent conceptual models. This heterogeneity severely limits the generalizability and replicability of findings. To tackle this issue, in this study, we propose to capitalize on a machine learning approach, OPNMF (Orthonormal Projective Non-Negative Factorization), and leverage internal cross-validation to promote generalizability to an independent dataset. We examined its application on the scores of 334 adults at the Delis-Kaplan Executive Function System (D-KEFS), while comparing to standard EFA and Principal Component Analysis (PCA). We further evaluated the replicability of the derived factorization across specific gender and age subsamples. Overall, OPNMF and PCA both converge towards a two-factor model as the best data-fit model. The derived factorization suggests a division between low-level and high-level executive functioning measures, a model further supported in subsamples. In contrast, EFA, highlighted a five-factor model which reflects the segregation of the D-KEFS battery into its main tasks while still clustering higher-level tasks together. However, this model was poorly supported in the subsamples. Thus, the parsimonious two-factors model revealed by OPNMF encompasses the more complex factorization yielded by EFA while enjoying higher generalizability. Hence, OPNMF provides a conceptually meaningful, technically robust, and generalizable factorization for psychometric tools.
Disciplines :
Neurosciences & behavior
Author, co-author :
Camilleri, J. A.
Eickhoff, S. B.
Weis, Suzanne
Chen, J.
Amunts, J.
Sotiras, A.
Genon, Sarah ; Université de Liège - ULiège > Département des sciences cliniques > Neuroimagerie des troubles de la mémoire et revalid. cogn.
Language :
English
Title :
A machine learning approach for the factorization of psychometric data with application to the Delis Kaplan Executive Function System.
scite shows how a scientific paper has been cited by providing the context of the citation, a classification describing whether it supports, mentions, or contrasts the cited claim, and a label indicating in which section the citation was made.
Bibliography
Ioannidis, J. P. Why most published research findings are false. PLoS Med. 2(8), e124 (2005). DOI: 10.1371/journal.pmed.0020124
Masouleh, S. K., Eickhoff, S. B., Hoffstaedter, F., Genon, S. & Alzheimer’s Disease Neuroimaging Initiative. Empirical examination of the replicability of associations between brain structure and psychological variables. Elife 8, e43464 (2019). DOI: 10.7554/eLife.43464
Lindsay, D. S. Replication in psychological science. Psychol. Sci. 26, 1827–1832 (2015). DOI: 10.1177/0956797615616374
Pashler, H. & Wagenmakers, E. Editors’ introduction to the special section on replicability in psychological science: A crisis of confidence?. Perspect. Psychol. Sci. 7(6), 528–530 (2012). DOI: 10.1177/1745691612465253
Avinun, R., Israel, S., Knodt, A. R. & Hariri, A. R. Little evidence for associations between the big five personality traits and variability in brain gray or white matter. Neuroimage 220, 117092 (2020). DOI: 10.1016/j.neuroimage.2020.117092
Boekel, W. et al. A purely confirmatory replication study of structural brain-behavior correlations. Cortex 66, 115–133 (2015). DOI: 10.1016/j.cortex.2014.11.019
Genon, S. et al. Searching for behavior relating to grey matter volume in a-priori defined right dorsal premotor regions: Lessons learned. Neuroimage 157, 144–156 (2017). DOI: 10.1016/j.neuroimage.2017.05.053
Shrout, P. E. & Rodgers, J. L. Psychology, science, and knowledge construction: Broadening perspectives from the replication crisis. Annu. Rev. Psychol. 69, 487–510 (2018). DOI: 10.1146/annurev-psych-122216-011845
Botvinik-Nezer, R. et al. Variability in the analysis of a single neuroimaging dataset by many teams. Nature 582, 1–7 (2020). DOI: 10.1038/s41586-020-2314-9
Simmons, J. P., Nelson, L. D. & Simonsohn, U. False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychol. Sci. 22(11), 1359–1366 (2011). DOI: 10.1177/0956797611417632
Carp, J. On the plurality of (methodological) worlds: Estimating the analytic flexibility of FMRI experiments. Front. Neurosci. 6, 149 (2012). DOI: 10.3389/fnins.2012.00149
Martínez, K. et al. Reproducibility of brain-cognition relationships using three cortical surface-based protocols: An exhaustive analysis based on cortical thickness. Hum. Brain Mapp. 36(8), 3227–3245 (2015). DOI: 10.1002/hbm.22843
Wagenmakers, E., Wetzels, R., Borsboom, D., van der Maas, H. L. J. & Kievit, R. A. An agenda for purely confirmatory research. Perspect. Psychol. Sci. 7(6), 632–638 (2012). DOI: 10.1177/1745691612463078
Treiblmaier, H. & Filzmoser, P. Exploratory factor analysis revisited: How robust methods support the detection of hidden multivariate data structures in IS research. Inform. Manag. 47(4), 197–207 (2010). DOI: 10.1016/j.im.2010.02.002
Watkins, M. W. Exploratory factor analysis: A guide to best practice. J. Black Psychol. 44(3), 219–246 (2018). DOI: 10.1177/0095798418771807
Spearman, C. “General intelligence,” objectively determined and measured. Am. J. Psychol. 15, 201–292 (1904). DOI: 10.2307/1412107
Latzman, R. D. & Markon, K. E. The factor structure and age-related factorial invariance of the Delis–Kaplan Executive Function System (D-KEFS). Assessment 17(2), 172–184 (2010). DOI: 10.1177/1073191109356254
Friedman, N. P. et al. Individual differences in executive functions are almost entirely genetic in origin. J. Exp. Psychol. Gen. 137(2), 201 (2008). DOI: 10.1037/0096-3445.137.2.201
Collette, F., Hogge, M., Salmon, E. & Van der Linden, M. Exploration of the neural substrates of executive functioning by functional neuroimaging. Neuroscience 139(1), 209–221 (2006). DOI: 10.1016/j.neuroscience.2005.05.035
Armstrong, J. S. & Soelberg, P. On the interpretation of factor analysis. Psychol. Bull. 70(5), 361 (1968). DOI: 10.1037/h0026434
Comrey, A. L. Common methodological problems in factor analytic studies. J. Consult. Clin. Psychol. 46(4), 648 (1978). DOI: 10.1037/0022-006X.46.4.648
MacCallum, R. A comparison of factor analysis programs in SPSS, BMDP, and SAS. Psychometrika 48(2), 223–231 (1983). DOI: 10.1007/BF02294017
Weiss, D. J., Rand McNally and Co & United States of America. Multivariate procedures. In Handbook of Industrial and Organizational Psychology (ed. Dunnette, M. D.) see ncj-52907 (1976).
Ford, J. K., MacCallum, R. C. & Tait, M. The application of exploratory factor analysis in applied psychology: A critical review and analysis. Pers. Psychol. 39(2), 291–314 (1986). DOI: 10.1111/j.1744-6570.1986.tb00583.x
Chen, J. et al. Neurobiological divergence of the positive and negative schizophrenia subtypes identified on a new factor structure of psychopathology using non-negative factorization: An international machine learning study. Biol. Psychiatr. 87(3), 282–293 (2020). DOI: 10.1016/j.biopsych.2019.08.031
Blackburn, R., Renwick, S. J., Donnelly, J. P. & Logan, C. Big five or big two? superordinate factors in the NEO five factor inventory and the antisocial personality questionnaire. Personal. Individ. Differ. 37(5), 957–970 (2004). DOI: 10.1016/j.paid.2003.10.017
Amieva, H., Phillips, L. & Della Sala, S. Behavioral dysexecutive symptoms in normal aging. Brain Cogn. 53(2), 129–132 (2003). DOI: 10.1016/S0278-2626(03)00094-0
Bennett, P. C., Ong, B. & Ponsford, J. Assessment of executive dysfunction following traumatic brain injury: Comparison of the BADS with other clinical neuropsychological measures. J. Int. Neuropsychol. Soc.: JINS 11(5), 606 (2005). DOI: 10.1017/S1355617705050721
Burgess, P. W. Theory and methodology in executive function research. In Methodology of Frontal and Executive Function 87–121 (Routledge, 2004).
Chan, R. C. Dysexecutive symptoms among a non-clinical sample: A study with the use of the dysexecutive questionnaire. Br. J. Psychol. 92(3), 551–565 (2001). DOI: 10.1348/000712601162338
Robbins, T. W. et al. A study of performance on tests from the CANTAB battery sensitive to frontal lobe dysfunction in a large sample of normal volunteers: Implications for theories of executive functioning and cognitive aging. J. Int. Neuropsychol. Soc. 4(5), 474–490 (1998). DOI: 10.1017/S1355617798455073
Bentler, P. M. & Kano, Y. On the equivalence of factors and components. Multivar. Behav. Res. 25(1), 67–74 (1990). DOI: 10.1207/s15327906mbr2501_8
Floyd, F. J. & Widaman, K. F. Factor analysis in the development and refinement of clinical assessment instruments. Psychol. Assess. 7(3), 286 (1995). DOI: 10.1037/1040-3590.7.3.286
Gorsuch, R. L. Common factor analysis versus component analysis: Some well and little known facts. Multivar. Behav. Res. 25(1), 33–39 (1990). DOI: 10.1207/s15327906mbr2501_3
Costello, A. B. & Osborne, J. Best practices in exploratory factor analysis: Four recommendations for getting the most from your analysis. Pract. Assess. Res. Eval. 10(1), 7 (2005).
Arrindell, W. A. & Van der Ende, J. An empirical test of the utility of the observations-to-variables ratio in factor and components analysis. Appl. Psychol. Meas. 9(2), 165–178 (1985). DOI: 10.1177/014662168500900205
Guadagnoli, E. & Velicer, W. F. Relation of sample size to the stability of component patterns. Psychol. Bull. 103(2), 265 (1988). DOI: 10.1037/0033-2909.103.2.265
Conway, J. M. & Huffcutt, A. I. A review and evaluation of exploratory factor analysis practices in organizational research. Organ. Res. Methods 6(2), 147–168 (2003). DOI: 10.1177/1094428103251541
Zelazo, P. D. & Müller, U. Executive function in typical and atypical development. In Handbook of Childhood Cognitive Development 445–469 (2002).
Zelazo, P. D., Carter, A., Reznick, J. S. & Frye, D. Early development of executive function: A problem-solving framework. Rev. Gen. Psychol. 1(2), 198–226 (1997). DOI: 10.1037/1089-2680.1.2.198
Lezak, M. D. The problem of assessing executive functions. Int. J. Psychol. 17(1–4), 281–297 (1982). DOI: 10.1080/00207598208247445
Miyake, A. et al. The unity and diversity of executive functions and their contributions to complex “frontal lobe” tasks: A latent variable analysis. Cogn. Psychol. 41(1), 49–100 (2000). DOI: 10.1006/cogp.1999.0734
Jurado, M. B. & Rosselli, M. The elusive nature of executive functions: A review of our current understanding. Neuropsychol. Rev. 17(3), 213–233 (2007). DOI: 10.1007/s11065-007-9040-z
Miyake, A. & Friedman, N. P. The nature and organization of individual differences in executive functions: Four general conclusions. Curr. Dir. Psychol. Sci. 21(1), 8–14 (2012). DOI: 10.1177/0963721411429458
Karr, J. E. et al. The unity and diversity of executive functions: A systematic review and re-analysis of latent variable studies. Psychol. Bull. 144(11), 1147 (2018). DOI: 10.1037/bul0000160
Delis, D. C., Kaplan, E. &; Kramer, J. H. Delis–Kaplan Executive Function System (2001).
Floyd, R. G., Bergeron, R., Hamilton, G. & Parra, G. R. How do executive functions fit with the Cattell–Horn–Carroll model? Some evidence from a joint factor analysis of the Delis–Kaplan executive function system and the Woodcock–Johnson III tests of cognitive abilities. Psychol. Sch. 47(7), 721–738 (2010).
McFarland, D. J. Factor-analytic evidence for the complexity of the Delis–Kaplan Executive Function System (D-KEFS). Assessment 27(7), 1645–1656 (2020). DOI: 10.1177/1073191119843584
Chen, J. et al. Intrinsic connectivity patterns of task-defined brain networks allow individual prediction of cognitive symptom dimension of schizophrenia and are linked to molecular architecture. Biol. Psychiatr. 89, 308–319 (2020). DOI: 10.1016/j.biopsych.2020.09.024
Love, J. et al. Software to sharpen your stats. APS Obs. 28(3), 27–29 (2015).
Nooner, K. B. et al. The NKI-rockland sample: A model for accelerating the pace of discovery science in psychiatry. Front. Neurosci. 6, 152 (2012). DOI: 10.3389/fnins.2012.00152
Kim, J. et al. Somatic ERCC2 mutations are associated with a distinct genomic signature in urothelial tumors. Nat. Genet. 48(6), 600–606 (2016). DOI: 10.1038/ng.3557
Hofree, M., Shen, J. P., Carter, H., Gross, A. & Ideker, T. Network-based stratification of tumor mutations. Nat. Methods 10(11), 1108–1115 (2013). DOI: 10.1038/nmeth.2651
Sadanandam, A. et al. A colorectal cancer classification system that associates cellular phenotype and responses to therapy. Nat. Med. 19(5), 619–625 (2013). DOI: 10.1038/nm.3175
Sotiras, A., Resnick, S. M. & Davatzikos, C. Finding imaging patterns of structural covariance via non-negative matrix factorization. Neuroimage 108, 1–16 (2015). DOI: 10.1016/j.neuroimage.2014.11.045
Yang, Z. & Oja, E. Linear and nonlinear projective nonnegative matrix factorization. IEEE Trans. Neural Netw. 21(5), 734–749 (2010). DOI: 10.1109/TNN.2010.2041361
Hubert, L. & Arabie, P. Comparing partitions. J. Classif. 2(1), 193–218 (1985). DOI: 10.1007/BF01908075
Meilă, M. Comparing clusterings—An information based distance. J. Multivar. Anal. 98(5), 873–895 (2007). DOI: 10.1016/j.jmva.2006.11.013
Raguideau, S., Plancade, S., Pons, N., Leclerc, M. & Laroche, B. Inferring aggregated functional traits from metagenomic data using constrained non-negative matrix factorization: Application to fiber degradation in the human gut microbiota. PLoS Comput. Biol. 12(12), e1005252 (2016). DOI: 10.1371/journal.pcbi.1005252
Fisk, J. E. & Sharp, C. A. Age-related impairment in executive functioning: Updating, inhibition, shifting, and access. J. Clin. Exp. Neuropsychol. 26(7), 874–890 (2004). DOI: 10.1080/13803390490510680
Lehto, J. E., Juujärvi, P., Kooistra, L. & Pulkkinen, L. Dimensions of executive functioning: Evidence from children. Br. J. Dev. Psychol. 21(1), 59–80 (2003). DOI: 10.1348/026151003321164627
Hull, R., Martin, R. C., Beier, M. E., Lane, D. & Hamilton, A. C. Executive function in older adults: A structural equation modeling approach. Neuropsychology 22(4), 508 (2008). DOI: 10.1037/0894-4105.22.4.508
Cattell, R. B. The scree test for the number of factors. Multivar. Behav. Res. 1(2), 245–276 (1966). DOI: 10.1207/s15327906mbr0102_10
Horn, J. L. A rationale and test for the number of factors in factor analysis. Psychometrika 30(2), 179–185 (1965). DOI: 10.1007/BF02289447
D'agostino, R. B. & Russell, H. K. Scree test. In Encyclopedia of Biostatistics, Vol. 7 (2005).
Wood, N. D., Akloubou Gnonhosou, D. C. & Bowling, J. W. Combining parallel and exploratory factor analysis in identifying relationship scales in secondary data. Marriage Fam. Rev. 51(5), 385–395 (2015). DOI: 10.1080/01494929.2015.1059785
Franklin, S. B., Gibson, D. J., Robertson, P. A., Pohlmann, J. T. & Fralish, J. S. Parallel analysis: A method for determining significant principal components. J. Veg. Sci. 6(1), 99–106 (1995). DOI: 10.2307/3236261
McDonald, R. P. & Marsh, H. W. Choosing a multivariate model: Noncentrality and goodness of fit. Psychol. Bull. 107(2), 247 (1990). DOI: 10.1037/0033-2909.107.2.247
The jamovi Project. jamovi (Version 1.2) [Computer Software] (2021). Retrieved from https://www.jamovi.org.
Deckel, A. W. & Hesselbrock, V. Behavioral and cognitive measurements predict scores on the MAST: A 3-year prospective study. Alcohol.: Clin. Exp. Res. 20(7), 1173–1178 (1996). DOI: 10.1111/j.1530-0277.1996.tb01107.x
Testa, R., Bennett, P. & Ponsford, J. Factor analysis of nineteen executive function tests in a healthy adult population. Arch. Clin. Neuropsychol. 27(2), 213–224 (2012). DOI: 10.1093/arclin/acr112
Vaughan, L. & Giovanello, K. Executive function in daily life: Age-related influences of executive processes on instrumental activities of daily living. Psychol. Aging 25(2), 343 (2010). DOI: 10.1037/a0017729
Baron, S. I. Delis–Kaplan executive function system. Child Neuropsychol. 10(2), 147–152 (2004). DOI: 10.1080/09297040490911140
Karr, J. E., Hofer, S. M., Iverson, G. L. & Garcia-Barrera, M. A. Examining the latent structure of the Delis–Kaplan executive function system. Arch. Clin. Neuropsychol. 34(3), 381–394 (2019). DOI: 10.1093/arclin/acy043
Brydges, C. R., Reid, C. L., Fox, A. M. & Anderson, M. A unitary executive function predicts intelligence in children. Intelligence 40(5), 458–469 (2012). DOI: 10.1016/j.intell.2012.05.006
Hughes, C. & Ensor, R. Individual differences in growth in executive function across the transition to school predict externalizing and internalizing behaviors and self-perceived academic success at 6 years of age. J. Exp. Child Psychol. 108(3), 663–676 (2011). DOI: 10.1016/j.jecp.2010.06.005
Fournier-Vicente, S., Larigauderie, P. & Gaonac’h, D. More dissociations and interactions within central executive functioning: A comprehensive latent-variable analysis. Acta Physiol. (Oxf.) 129(1), 32–48 (2008).
de Frias, C. M., Dixon, R. A. & Strauss, E. Structure of four executive functioning tests in healthy older adults. Neuropsychology 20(2), 206 (2006). DOI: 10.1037/0894-4105.20.2.206
Glisky, E. L. et al. Differences between young and older adults in unity and diversity of executive functions. Aging Neuropsychol. Cogn. 8, 1–26 (2020). DOI: 10.1080/13825585.2020.1830936
Traykov, L. et al. Executive functions deficit in mild cognitive impairment. Cogn. Behav. Neurol. 20(4), 219–224 (2007). DOI: 10.1097/WNN.0b013e31815e6254
Zhang, Y., Han, B., Verhaeghen, P. & Nilsson, L. Executive functioning in older adults with mild cognitive impairment: MCI has effects on planning, but not on inhibition. Aging Neuropsychol. Cogn. 14(6), 557–570 (2007). DOI: 10.1080/13825580600788118
Brandt, J. et al. Selectivity of executive function deficits in mild cognitive impairment. Neuropsychology 23(5), 607 (2009). DOI: 10.1037/a0015851
Elliott, R. Executive functions and their disorders: Imaging in clinical neuroscience. Br. Med. Bull. 65(1), 49–59 (2003). DOI: 10.1093/bmb/65.1.49
Badre, D. & Nee, D. E. Frontal cortex and the hierarchical control of behavior. Trends Cogn. Sci. 22(2), 170–188 (2018). DOI: 10.1016/j.tics.2017.11.005
Berg, E. A. A simple objective technique for measuring flexibility in thinking. J. Gen. Psychol. 39(1), 15–22 (1948). DOI: 10.1080/00221309.1948.9918159
Similar publications
Sorry the service is unavailable at the moment. Please try again later.
This website uses cookies to improve user experience. Read more
Save & Close
Accept all
Decline all
Show detailsHide details
Cookie declaration
About cookies
Strictly necessary
Performance
Strictly necessary cookies allow core website functionality such as user login and account management. The website cannot be used properly without strictly necessary cookies.
This cookie is used by Cookie-Script.com service to remember visitor cookie consent preferences. It is necessary for Cookie-Script.com cookie banner to work properly.
Performance cookies are used to see how visitors use the website, eg. analytics cookies. Those cookies cannot be used to directly identify a certain visitor.
Used to store the attribution information, the referrer initially used to visit the website
Cookies are small text files that are placed on your computer by websites that you visit. Websites use cookies to help users navigate efficiently and perform certain functions. Cookies that are required for the website to operate properly are allowed to be set without your permission. All other cookies need to be approved before they can be set in the browser.
You can change your consent to cookie usage at any time on our Privacy Policy page.
