Small to train, small to test: Dealing with low sample size in model evaluation

[en] Sample size is a key issue in species distribution modelling. While many studies focused on the relevance of sample size for model calibration, the importance of the size of the dataset used for model evaluation has received much less attention. Here, we highlight two previously published approaches to address the problem, and which are relatively simple to implement: the pooling evaluation and the implementation of null models. We discuss the importance of these or other potential approaches that are critical for model evaluation in rare species, which represent the bulk of biodiversity, and for which accurate models are most necessary in a conservation context.

Disciplines :

Environmental sciences & ecology

Author, co-author :

Collart, Flavien ; Université de Liège - ULiège ; Department of Ecology and Evolution, University of Lausanne, Lausanne, Switzerland

Guisan, Antoine; Department of Ecology and Evolution, University of Lausanne, Lausanne, Switzerland ; Institute of Earth Surface Dynamics, University of Lausanne, Lausanne, Switzerland

Language :

English

Title :

Small to train, small to test: Dealing with low sample size in model evaluation

Publication date :

July 2023

Journal title :

Ecological Informatics

ISSN :

1574-9541

eISSN :

1878-0512

Publisher :

Elsevier B.V.

Volume :

Pages :

102106

Peer reviewed :

Peer Reviewed verified by ORBi

Additional URL :

https://api.elsevier.com/content/article/PII:S1574954123001358?httpAccept=text/xml

Funders :

SNF - Schweizerischer Nationalfonds zur Förderung der wissenschaftlichen Forschung [CH]

Funding text :

FC was funded by a grant from the Swiss National Science Foundation to AG (SNSF; Grant number 197777 ).

Available on ORBi :

since 20 June 2023

Statistics

Number of views

25 (5 by ULiège)

Number of downloads

94 (1 by ULiège)

More statistics

Scopus citations^®

Scopus citations^®
without self-citations

OpenCitations

Bibliography

Araújo, M.B., Anderson, R.P., Márcia Barbosa, A., Beale, C.M., Dormann, C.F., Early, R., Garcia, R.A., Guisan, A., Maiorano, L., Naimi, B., O'Hara, R.B., Zimmermann, N.E., Rahbek, C., Standards for distribution models in biodiversity assessments. Sci. Adv., 5(1), 2019, eaat4858, 10.1126/sciadv.aat4858.
Barber, R.A., Ball, S.G., Morris, R.K.A., Gilbert, F., Target-group backgrounds prove effective at correcting sampling bias in Maxent models. Divers. Distrib. 28:1 (2022), 128–141, 10.1111/ddi.13442.
Bean, W.T., Stafford, R., Brashares, J.S., The effects of small sample size and sample bias on threshold selection and accuracy assessment of species distribution models. Ecography 35:3 (2012), 250–258, 10.1111/j.1600-0587.2011.06545.x.
Bohl, C.L., Kass, J.M., Anderson, R.P., A new null model approach to quantify performance and significance for ecological niche models of species distributions. J. Biogeogr. 46:6 (2019), 1101–1111, 10.1111/jbi.13573.
Breiner, F.T., Guisan, A., Bergamini, A., Nobis, M.P., Overcoming limitations of modelling rare species by using ensembles of small models. Methods Ecol. Evol. 6:10 (2015), 1210–1218, 10.1111/2041-210X.12403.
Breiner, F.T., Nobis, M.P., Bergamini, A., Guisan, A., Optimizing ensembles of small models for predicting the distribution of species with few occurrences. Methods Ecol. Evol. 9:4 (2018), 802–808, 10.1111/2041-210X.12957.
Broennimann, O., Di Cola, V., Guisan, A., Ecospat: Spatial Ecology Miscellaneous Methods. (R package version 3.4.). 2022.
Collart, F, Hedenäs, L, Broennimann, O, Guisan, A, Vanderpoorten, A, Intraspecific differentiation: Implications for niche and distribution modelling. J. Biogeogr. 48 (2021), 415–426, 10.1111/jbi.14009.
Courchamp, F., Angulo, E., Rivalan, P., Hall, R.J., Signoret, L., Bull, L., Meinard, Y., Rarity value and species extinction: the anthropogenic Allee effect. PLoS Biol., 4(12), 2006, e415, 10.1371/journal.pbio.0040415.
Dubuis, A., Pottier, J., Rion, V., Pellissier, L., Theurillat, J.-P., Guisan, A., Predicting spatial patterns of plant species richness: a comparison of direct macroecological and species stacking modelling approaches. Divers. Distrib. 17:6 (2011), 1122–1131, 10.1111/j.1472-4642.2011.00792.x.
Elith, J., Graham, C., Valavi, R., Abegg, M., Bruce, C., Ferrier, S., Ford, A., Guisan, A., Hijmans, R.J., Huettmann, F., Lohmann, L., Loiselle, B., Moritz, C., Overton, J., Peterson, A.T., Phillips, S., Richardson, K., Williams, S., Wiser, S.K., Zimmermann, N.E., Presence-only and presence-absence data for comparing species distribution modeling methods. Biodivers. Inform., 15(2), 2020, 10.17161/bi.v15i2.13384 Article 2.
Enquist, B.J., Feng, X., Boyle, B., Maitner, B., Newman, E.A., Jørgensen, P.M., Roehrdanz, P.R., Thiers, B.M., Burger, J.R., Corlett, R.T., Couvreur, T.L.P., Dauby, G., Donoghue, J.C., Foden, W., Lovett, J.C., Marquet, P.A., Merow, C., Midgley, G., Morueta-Holme, N., McGill, B.J., The commonness of rarity: global and future distribution of rarity across land plants. Sci. Adv., 5(11), 2019, eaaz0414, 10.1126/sciadv.aaz0414.
Fernandes, R.F., Scherrer, D., Guisan, A., Effects of simulated observation errors on the performance of species distribution models. Divers. Distrib. 25:3 (2019), 400–413, 10.1111/ddi.12868.
Fielding, A.H., Bell, J.F., A review of methods for the assessment of prediction errors in conservation presence/absence models. Environ. Conserv. 24:1 (1997), 38–49, 10.1017/S0376892997000088.
Fourcade, Y., Besnard, A.G., Secondi, J., Paintings predict the distribution of species, or the challenge of selecting environmental predictors and evaluation statistics. Glob. Ecol. Biogeogr. 27:2 (2018), 245–256, 10.1111/geb.12684.
Guisan, A., Tingley, R., Baumgartner, J.B., Naujokaitis-Lewis, I., Sutcliffe, P.R., Tulloch, A.I.T., Regan, T.J., Brotons, L., McDonald-Madden, E., Mantyka-Pringle, C., Martin, T.G., Rhodes, J.R., Maggini, R., Setterfield, S.A., Elith, J., Schwartz, M.W., Wintle, B.A., Broennimann, O., Austin, M., Buckley, Y.M., Predicting species distributions for conservation decisions. Ecol. Lett. 16:12 (2013), 1424–1435, 10.1111/ele.12189.
Guisan, A., Thuiller, W., Zimmermann, N.E., Habitat Suitability and Distribution Models. 2017, Cambridge University Press, 10.1017/9781139028271.
Hallman, T.A., Robinson, W.D., Deciphering ecology from statistical artefacts: competing influence of sample size, prevalence and habitat specialization on species distribution models and how small evaluation datasets can inflate metrics of performance. Divers. Distrib. 26:3 (2020), 315–328, 10.1111/ddi.13030.
Harrell, F.E. Jr., Lee, K.L., Mark, D.B., Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Stat. Med. 15:4 (1996), 361–387, 10.1002/(SICI)1097-0258(19960229)15:4<361::AID-SIM168>3.0.CO;2-4.
Hernandez, P.A., Graham, C.H., Master, L.L., Albert, D.L., The effect of sample size and species characteristics on performance of different species distribution modeling methods. Ecography 29:5 (2006), 773–785, 10.1111/j.0906-7590.2006.04700.x.
Hijmans, R.J., Phillips, S., Leathwick, J., Elith, J., Dismo: Species Distribution Modeling. (R package version 1.3-9). 2022.
Işik, K., Rare and endemic species: why are they prone to extinction?. Turk. J. Bot. 35:4 (2011), 411–417, 10.3906/bot-1012-90.
Jeliazkov, A., Gavish, Y., Marsh, C.J., Geschke, J., Brummitt, N., Rocchini, D., Haase, P., Kunin, W.E., Henle, K., Sampling and modelling rare species: conceptual guidelines for the neglected majority. Glob. Chang. Biol. 28:12 (2022), 3754–3777, 10.1111/gcb.16114.
Jiménez-Valverde, A., Sample size for the evaluation of presence-absence models. Ecol. Indic., 114, 2020, 106289, 10.1016/j.ecolind.2020.106289.
Kass, J.M., Muscarella, R., Galante, P.J., Bohl, C.L., Pinilla-Buitrago, G.E., Boria, R.A., Soley-Guardia, M., Anderson, R.P., ENMeval 2.0: redesigned for customizable and reproducible modeling of species’ niches and distributions. Methods Ecol. Evol. 12:9 (2021), 1602–1608, 10.1111/2041-210X.13628.
Leroy, B., Meynard, C.N., Bellard, C., Courchamp, F., Virtualspecies, an R package to generate virtual species distributions. Ecography 39:6 (2016), 599–607, 10.1111/ecog.01388.
Liu, C., Newell, G., White, M., The effect of sample size on the accuracy of species distribution models: considering both presences and pseudo-absences or background sites. Ecography., 2018, 10.1111/ecog.03188.
Lomba, A., Pellissier, L., Randin, C., Vicente, J., Moreira, F., Honrado, J., Guisan, A., Overcoming the rare species modelling paradox: a novel hierarchical framework applied to an Iberian endemic plant. Biol. Conserv. 143:11 (2010), 2647–2657, 10.1016/J.BIOCON.2010.07.007.
Meynard, C.N., Kaplan, D.M., Using virtual species to study species distributions and model performance. J. Biogeogr. 40:1 (2013), 1–8, 10.1111/jbi.12006.
Meynard, C.N., Leroy, B., Kaplan, D.M., Testing methods in species distribution modelling using virtual species: What have we learnt and what are we missing?. Ecography, 2019, 10.1111/ecog.04385 ecog.04385.
Mondanaro, A., Di Febbraro, M., Castiglione, S., Melchionna, M., Serio, C., Girardi, G., Belfiore, A.M., Raia, P., ENphylo: a new method to model the distribution of extremely rare species. Methods Ecol. Evol. 14:3 (2023), 911–922, 10.1111/2041-210X.14066.
Osborne, O.G., Fell, H.G., Atkins, H., van Tol, J., Phillips, D., Herrera-Alsina, L., Mynard, P., Bocedi, G., Gubry-Rangin, C., Lancaster, L.T., Creer, S., Nangoy, M., Fahri, F., Lupiyaningdyah, P., Sudiana, I.M., Juliandi, B., Travis, J.M.J., Papadopulos, A.S.T., Algar, A.C., Fauxcurrence: simulating multi-species occurrences for null models in species distribution modelling and biogeography. Ecography, 2022(7), 2022, e05880, 10.1111/ecog.05880.
Peduzzi, P., Concato, J., Kemper, E., Holford, T.R., Feinstein, A.R., A simulation study of the number of events per variable in logistic regression analysis. J. Clin. Epidemiol. 49:12 (1996), 1373–1379, 10.1016/S0895-4356(96)00236-3.
Peterson, A.T., Papeş, M., Soberón, J., Rethinking receiver operating characteristic analysis applications in ecological niche modeling. Ecol. Model. 213:1 (2008), 63–72, 10.1016/j.ecolmodel.2007.11.008.
Phillips, S.J., Dudík, M., Elith, J., Graham, C.H., Lehmann, A., Leathwick, J., Ferrier, S., Sample selection bias and presence-only distribution models: implications for background and pseudo-absence data. Ecol. Appl. 19:1 (2009), 181–197, 10.1890/07-2153.1.
Ponder, W.F., Carter, G.A., Flemons, P., Chapman, R.R., Evaluation of museum collection data for use in biodiversity assessment. Conserv. Biol. 15:3 (2001), 648–657, 10.1046/j.1523-1739.2001.015003648.x.
Raes, N., ter Steege, H., A null-model for significance testing of presence-only species distribution models. Ecography 30:5 (2007), 727–736, 10.1111/j.2007.0906-7590.05041.x.
Stockwell, D.R.B., Peterson, A.T., Effects of sample size on accuracy of species distribution models. Ecol. Model. 148:1 (2002), 1–13, 10.1016/S0304-3800(01)00388-X.
ter Steege, H., Pitman, N.C.A., Sabatier, D., Baraloto, C., Salomão, R.P., Guevara, J.E., Phillips, O.L., Castilho, C.V., Magnusson, W.E., Molino, J.-F., Monteagudo, A., Núñez Vargas, P., Montero, J.C., Feldpausch, T.R., Coronado, E.N.H., Killeen, T.J., Mostacedo, B., Vasquez, R., Assis, R.L., Silman, M.R., Hyperdominance in the Amazonian tree Flora. Science, 342(6156), 2013, 1243092, 10.1126/science.1243092.
Valavi, R., Guillera-Arroita, G., Lahoz-Monfort, J.J., Elith, J., Predictive performance of presence-only species distribution models: a benchmark study with reproducible code. Ecol. Monogr., 92(1), 2022, e01486, 10.1002/ecm.1486.
van Proosdij, A.S.J., Sosef, M.S.M., Wieringa, J.J., Raes, N., Minimum required number of specimen records to develop accurate species distribution models. Ecography 39:6 (2016), 542–552, 10.1111/ecog.01509.
Vincent, H., Bornand, C.N., Kempel, A., Fischer, M., Rare species perform worse than widespread species under changed climate. Biol. Conserv., 246, 2020, 108586, 10.1016/j.biocon.2020.108586.
Warren, D.L., Dornburg, A., Zapfe, K., Iglesias, T.L., The effects of climate change on Australia's only endemic Pokémon: measuring bias in species distribution models. Methods Ecol. Evol. 12:6 (2021), 985–995, 10.1111/2041-210X.13591.
Wisz, M.S., Hijmans, R.J., Li, J., Peterson, A.T., Graham, C.H., Guisan, A., Elith, J., Dudík, M., Ferrier, S., Huettmann, F., Leathwick, J.R., Lehmann, A., Lohmann, L., Loiselle, B.A., Manion, G., Moritz, C., Nakamura, M., Nakazawa, Y., Overton, J. McC, Zimmermann, N.E., Effects of sample size on the performance of species distribution models. Divers. Distrib. 14:5 (2008), 763–773, 10.1111/j.1472-4642.2008.00482.x.
Zurell, D., Berger, U., Cabral, J.S., Jeltsch, F., Meynard, C.N., Münkemüller, T., Nehrbass, N., Pagel, J., Reineking, B., Schröder, B., Grimm, V., The virtual ecologist approach: simulating data and observers. Oikos 119:4 (2010), 622–635, 10.1111/j.1600-0706.2009.18284.x.
Zurell, D., Franklin, J., König, C., Bouchet, P.J., Dormann, C.F., Elith, J., Fandos, G., Feng, X., Guillera-Arroita, G., Guisan, A., Lahoz-Monfort, J.J., Leitão, P.J., Park, D.S., Peterson, A.T., Rapacciuolo, G., Schmatz, D.R., Schröder, B., Serra-Diaz, J.M., Thuiller, W., Merow, C., A standard protocol for reporting species distribution models. Ecography, 2020, 10.1111/ecog.04960 ecog.04960.