Uncertainty-Aware Evaluation of Deep Learning Object Detectors under Scarce and Evolving Test Datasets

In data-scarce domains, building reliable deep learning models often requires the relabeling, merging, or expansion of existing datasets. While these steps improve dataset quality and diversity, they complicate model evaluation: changes in evaluation outcomes may arise from dataset changes rather than real model improvement, making standard evaluation protocols difficult to interpret. We demonstrate this challenge in the context of trypanosome parasite detection, where model comparisons across three dataset versions yield inconsistent conclusions. Conventional metrics such as mean average precision (mAP) fail to reveal critical reliability issues, particularly when test data evolves. To address this, we propose a complementary evaluation approach based on predictive uncertainty. By assessing how well models distinguish true from false positives, including on out-of-distribution samples, we obtain a more stable and informative signal of model quality across dataset versions. Our findings show that uncertainty-aware evaluation exposes overlooked failure modes, enables more meaningful comparisons across evolving datasets, and highlights models that maintain reliable confidence estimates under distribution shift.