Capturing fine-level structure using unsupervised clustering method with multiple data types

Several methods exist to detect shared genetic ancestry or to identify population substructure using SNP-based or haplotype-based information (Price et al. 2006, Lawson et al. 2012). Here, we propose an unsupervised clustering method built on the ipPCA algorithm (Intarapanich et al. 2009). Our method supports both ordinal and categorical data, and it can be applied to panels of single locus and/or multiple loci data, or gene-based integrative summaries (Fouladi et al. 2015). Our method involves an iterative process using binary and ternary splits based on multivariate Gaussian mixture modeling of PCs and Clustering EM estimation as in (Lebret et al. 2015). To evaluate its performance, we examined different simulated scenarios of 2-4 populations, 500-8,000 individuals, 5,000-20,000 independent SNPs in HWE, and FST=[0.0007,0.006] (Balding and Nichols 1995), with 100 replicates for each scenario. SNPs were treated as categorical or continuous including ancestry-corrected SNPs. Haplotype-based runs used HapMap 3 data: CHB, CHD, and JPT. In simulated scenarios of extremely subtle structure (FST=[0.0009,0.006]), a population classification accuracy of 92% or greater was obtained, which was superior to ipPCA. Also in case of the HapMap populations, promising results to detect fine structure were obtained. We are convinced that our method has a potential to detect fine-level structure and it will be important in molecular reclassification studies of patients once underlying population structure has been removed.