The Genetic and Functional Basis of Olfactory Evolution in Monogamous and Promiscuous Deer Mice

Mammals have well-developed olfactory systems that mediate a wide range of stereotypic behaviors. Altering the reception of chemosignals can lead to marked changes in behavior and thus may be a repeated target of evolution. Yet, we still know little about the molecular logic by which biological information is extracted, how this logic evolves, or how it contributes to transitions in social behavior. North American deer mice (genus Peromyscus) occupy a large range of habitats, each with distinct conspecific competitors and predators, and have evolved divergent mating systems, from highly promiscuous to both socially and genetically monogamous. Therefore, deer mice represent an exciting system, as a complement to classical laboratory models, for studying the genetic mechanisms underpinning complex behaviors and social recognition. Here, we use a reference-quality, chromosome-level de novo genome assembly for Peromyscus maniculatus bairdii to first show contrasting evolutionary trajectories between the olfactory and vomeronasal (VNO) subsystems in rodents. Next, by focusing on the evolutionary dynamic VNO, we identify genes with high divergence levels in gene expression, specifically members of the Vmn2r class of vomeronasal receptors, as candidates for adaptation. Using the activation of these receptors as readouts, we identify pheromone receptors for which differential regulation correlates with the transition between promiscuity and monogamy. Together, our results provide new insights into the evolution of the mammalian olfactory system and identify differences in the olfactory repertoire that likely contribute to adaptation and the transition in the social behaviors of deer mice, opening new avenues for further investigation into the neural mechanisms underlying behavioral evolution.