A Methodology to Evaluate Strategies Predicting Rankings on Unseen Domains

[en] Frequently, multiple entities (methods, algorithms, procedures, solutions, etc.) can be developed for a common task and applied across various domains that differ in the distribution of scenarios encountered. For example, in computer vision, the input data provided to image analysis methods depend on the type of sensor used, its location, and the scene content. However, a crucial difficulty remains: can we predict which entities will perform best in a new domain based on assessments on known domains, without having to carry out new and costly evaluations? This paper presents an original methodology to address this question, in a leave-one-domain-out fashion, for various application-specific preferences. We illustrate its use with 30 strategies to predict the rankings of 40 entities (unsupervised background subtraction methods) on 53 domains (videos).

Research Center/Unit :

Montefiore Institute - Montefiore Institute of Electrical Engineering and Computer Science - ULiège
TELIM
VIULab

Disciplines :

Computer science

Author, co-author :

Pierard, Sébastien ; Université de Liège - ULiège > Département d'électricité, électronique et informatique (Institut Montefiore) > Télécommunications

Deliège, Adrien ; Université de Liège - ULiège > Département d'électricité, électronique et informatique (Institut Montefiore) > Télécommunications

Halin, Anaïs ; Université de Liège - ULiège > Montefiore Institute of Electrical Engineering and Computer Science

Van Droogenbroeck, Marc ; Université de Liège - ULiège > Département d'électricité, électronique et informatique (Institut Montefiore) > Télécommunications

Language :

English

Title :

A Methodology to Evaluate Strategies Predicting Rankings on Unseen Domains

Publication date :

July 2025

Event name :

IEEE International Conference on Multimedia & Expo 2025

Event organizer :

IEEE

Event place :

Nantes, France

Event date :

Du 30 Juin au 4 Juillet

Audience :

International

Main work title :

Procceedings of ICMEW 2025

Publisher :

Institute of Electrical and Electronics Engineers (IEEE)

Pages :

1-6

Peer review/Selection committee :

Peer reviewed

Name of the research project :

ReconnAIssance
ARIAC
Destina

Funders :

SPW - Service Public de Wallonie
SPW EER - Service Public de Wallonie. Economie, Emploi, Recherche
ULiège FSR - Université de Liège. Fonds spéciaux pour la recherche

Funding number :

8573; 2010235

Available on ORBi :

since 21 May 2025

Statistics

Number of views

89 (20 by ULiège)

Number of downloads

133 (3 by ULiège)

More statistics

Scopus citations^®

Scopus citations^®
without self-citations

OpenCitations

OpenAlex citations

Bibliography

Sébastien Piérard, Anaïs Halin, Anthony Cioppa, Adrien Deliege, and Marc Van Droogenbroeck, "Foundations of the theory of performancebased ranking," in IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Nashville, TN, USA, Jun. 2025.
Sébastien Piérard, Anaïs Halin, Anthony Cioppa, Adrien Deliege, and Marc Van Droogenbroeck, "The Tile: A 2D map of ranking scores for two-class classification," arXiv, vol. abs/2412.04309, 2024.
Anaïs Halin, Sébastien Piérard, Anthony Cioppa, and Marc Van Droogenbroeck, "A hitchhiker's guide to understanding performances of two-class classifiers," arXiv, vol. abs/2412.04377, 2024.
Nil Goyette, Pierre-Marc Jodoin, Fatih Porikli, Janusz Konrad, and Prakash Ishwar, "Changedetection.net: A new change detection benchmark dataset," in IEEE Int. Conf. Comput. Vis. Pattern Recognit. Work. (CVPRW), Providence, RI, USA, Jun. 2012, pp. 1-8.
Yi Wang, Pierre-Marc Jodoin, Fatih Porikli, Janusz Konrad, Yannick Benezeth, and Prakash Ishwar, "CDnet 2014: An expanded change detection benchmark dataset," in IEEE/CVF Conf. Comput. Vis. Pattern Recognit. Work. (CVPRW), Columbus, OH, USA, Jun. 2014, pp. 393-400.
Anaïs Halin, Sébastien Piérard, Renaud Vandeghen, Benoit Gérin, Maxime Zanella, Martin Colot, Jan Held, Anthony Cioppa, Emmanuel Jean, Gianluca Bontempi, Saïd Mahmoudi, Benoit Macq, and Marc Van Droogenbroeck, "Physically interpretable probabilistic domain characterization," in Asian Conf. Comput. Vis. Work. (ACCV Work.), 2025, vol. 15482 of Lect. Notes Comput. Sci., pp. 17-35.
Kyle Cranmer, Johann Brehmer, and Gilles Louppe, "The frontier of simulation-based inference," Proc. National Acad. Sci. (PNAS), vol. 117, no. 48, pp. 30055-30062, May 2020.
Thierry Bouwmans, Sajid Javed, Maryam Sultana, and Soon Ki Jung, "Deep neural network concepts for background subtraction:a systematic review and comparative evaluation," Neural Networks, vol. 117, pp. 8-66, Sept. 2019.
Thierry Bouwmans and Belmar Garcia-Garcia, "Visual surveillance of human activities: Background subtraction challenges and methods," in From Visual Surveillance to Internet of Things: Technology and Applications, pp. 1-24. Taylor & Francis Group, Oct. 2019.
Thierry Bouwmans and Belmar Garcia-Garcia, "Visual surveillance of natural environments: Background subtraction challenges and methods," in From Visual Surveillance to Internet of Things: Technology and Applications, pp. 1-20. Taylor & Francis, Oct. 2019.
Marc Braham and Marc Van Droogenbroeck, "Deep background subtraction with scene-specific convolutional neural networks," in IEEE Int. Conf. Syst. Signals Image Process. (IWSSIP), Bratislava, Slovakia, May 2016, pp. 1-4.
Olivier Barnich and Marc Van Droogenbroeck, "ViBe: A universal background subtraction algorithm for video sequences," IEEE Trans. Image Process., vol. 20, no. 6, pp. 1709-1724, Jun. 2011.
Anthony Cioppa, Marc Van Droogenbroeck, and Marc Braham, "Realtime semantic background subtraction," in IEEE Int. Conf. Image Process. (ICIP), Abu Dhabi, United Arab. Emir., Oct. 2020, pp. 3214-3218.
Jérome Leens, Sébastien Piérard, Olivier Barnich, Marc Van Droogenbroeck, and Jean-Marc Wagner, "Combining color, depth, and motion for video segmentation," in Computer Vision Systems. 2009, vol. 5815 of Lect. Notes Comput. Sci., pp. 104-113, Springer.
Pierre-Marc Jodoin, Sébastien Piérard, Yi Wang, and Marc Van Droogenbroeck, "Overview and benchmarking of motion detection methods," in Background Modeling and Foreground Detection for Video Surveillance, chapter 24. Chapman and Hall/CRC, Jul. 2014.
Sébastien Piérard, Marc Braham, and Marc Van Droogenbroeck, "An exploration of the performances achievable by combining unsupervised background subtraction algorithms," arXiv, vol. abs/2202.12563, 2022.
Marc Braham, Sébastien Piérard, and Marc Van Droogenbroeck, "Semantic background subtraction," in IEEE Int. Conf. Image Process. (ICIP), Beijing, China, Sept. 2017, pp. 4552-4556.
Sébastien Piérard and Marc Van Droogenbroeck, "A perfect estimation of a background image does not lead to a perfect background subtraction: Analysis of the upper bound on the performance," in Int. Conf. Image Anal. Process. (ICIAP), Work. Scene Backgr. Model. Initial. (SBMI). 2015, vol. 9281 of Lect. Notes Comput. Sci., pp. 527-534, Springer.
Sébastien Piérard and Marc Van Droogenbroeck, "Summarizing the performances of a background subtraction algorithm measured on several videos," in IEEE Int. Conf. Image Process. (ICIP), Abu Dhabi, United Arab Emirates, Oct. 2020, pp. 3234-3238.
David Cossock and Tong Zhang, "Subset ranking using regression," in Learn. Theory, 2006, vol. 4005 of Lect. Notes Comput. Sci., pp. 605-619.
Ping Li, Qiang Wu, and Christopher Burges, "McRank: Learning to rank using multiple classification and gradient boosting," in Adv. Neural Inf. Process. Syst. (NeurIPS), Vancouver, Can., Dec. 2007, vol. 20, pp. 1-8, Curran Assoc. Inc.
Chris Burges, Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deeds, Nicole Hamilton, and Greg Hullender, "Learning to rank using gradient descent," in Int. Conf. Mach. Learn. (ICML), Bonn, Germany, 2005, pp. 89-96, ACM Press.
Zhe Cao, Tao Qin, Tie-Yan Liu, Ming-Feng Tsai, and Hang Li, "Learning to rank: From pairwise approach to listwise approach," in Int. Conf. Mach. Learn. (ICML), Corvalis, OR, USA, Jun. 2007, p. 129-136, ACM.
T. Liu, "Learning to rank for ir," Foundation and Trends in IR, vol. 3, no. 3, pp. 225-331, 2009.
Niek Tax, Sander Bockting, and Djoerd Hiemstra, "A cross-benchmark comparison of 87 learning to rank methods," Inf. Process. & Manag., vol. 51, no. 6, pp. 757-772, Nov. 2015.
MMSegmentation Contributors, "MMSegmentation: OpenMMLab semantic segmentation toolbox and benchmark," https://github.com/ open-mmlab/mmsegmentation, 2020.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba, "Scene parsing through ADE20K dataset," in IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Honolulu, HI, USA, Jul. 2017, pp. 5122-5130.
Holger Caesar, Jasper Uijlings, and Vittorio Ferrari, "COCO-stuff: Thing and stuff classes in context," in IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Salt Lake City, UT, USA, Jun. 2018, pp. 1209-1218.
Bowen Cheng, Ishan Misra, Alexander G. Schwing, Alexander Kirillov, and Rohit Girdhar, "Masked-attention mask transformer for universal image segmentation," in IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), New Orleans, LA, USA, Jun. 2022, pp. 1280-1289.