[en] There exist many background subtraction algorithms to detect motion in videos. To help comparing them, datasets with ground-truth data such as CDNET or LASIESTA have been proposed. These datasets organize videos in categories that represent typical challenges for background subtraction. The evaluation procedure promoted by their authors consists in measuring performance indicators for each video separately and to average them hierarchically, within a category first, then between categories, a procedure which we name “summarization”. While the summarization by averaging performance indicators is a valuable effort to standardize the evaluation procedure, it has no theoretical justification and it breaks the intrinsic relationships between summarized indicators. This leads to interpretation inconsistencies. In this paper, we present a theoretical approach to summarize the performances for multiple videos that preserves the relationships between performance indicators. In addition, we give formulas and an algorithm to calculate summarized performances. Finally, we showcase our observations on CDNET 2014.
Research Center/Unit :
Montefiore Institute of Electrical Engineering and Computer Science - Montefiore Institute ; Telim
Disciplines :
Computer science
Author, co-author :
Pierard, Sébastien ; Université de Liège - ULiège > Dép. d'électric., électron. et informat. (Inst.Montefiore) > Télécommunications
Van Droogenbroeck, Marc ; Université de Liège - ULiège > Dép. d'électric., électron. et informat. (Inst.Montefiore) > Télécommunications
Language :
English
Title :
Summarizing the performances of a background subtraction algorithm measured on several videos
Publication date :
October 2020
Event name :
IEEE International Conference on Image Processing (ICIP)
Event organizer :
IEEE
Event place :
Abu Dhabi, United Arab Emirates
Event date :
from 25-10-2020 to 28-10-2020
Audience :
International
Main work title :
Proceedings of the IEEE International Conference on Image Processing (ICIP)
T. Bouwmans, F. El Baf, and B. Vachon, "Statistical background modeling for foreground detection: A survey, " in Handbook of Pattern Recognition and Computer Vision (volume 4), chapter 3, pp. 181-199. World Scientific Publishing, January 2010.
T. Bouwmans, "Traditional and recent approaches in background modeling for foreground detection: An overview, " Computer Science Review, vol. 11-12, pp. 31-66, May 2014.
T. Bouwmans, S. Javed, M. Sultana, and S. Jung, "Deep neural network concepts for background subtraction: A systematic review and comparative evaluation, " Neural Networks, vol. 117, pp. 8-66, September 2019.
S. Elhabian, K. El-Sayed, and S. Ahmed, "Moving object detection in spatial domain using background removal techniques-State-of-art, " Recent Patents on Computer Science, vol. 1, pp. 32-54, January 2008.
P.-M. Jodoin, S. Piérard, Y. Wang, and M. Van Droogenbroeck, "Overview and benchmarking of motion detection methods, " in Background Modeling and Foreground Detection for Video Surveillance, chapter 24. Chapman and Hall/CRC, July 2014.
A. Sobral and A. Vacavant, "A comprehensive review of background subtraction algorithms evaluated with synthetic and real videos, " Computer Vision and Image Understanding, vol. 122, pp. 4-21, May 2014.
N. Vaswani, T. Bouwmans, S. Javed, and P. Narayanamurthy, "Robust subspace learning: Robust PCA, robust subspace tracking, and robust subspace recovery, " IEEE Signal Processing Magazine, vol. 35, no. 4, pp. 32-55, July 2018.
N. Goyette, P.-M. Jodoin, F. Porikli, J. Konrad, and P. Ishwar, "changedetection. net: A new change detection benchmark dataset, " in IEEE International Conference on Computer Vision and Pattern RecognitionWorkshops (CVPRW), June 2012.
N. Goyette, P.-M. Jodoin, F. Porikli, J. Konrad, and P. Ishwar, "A novel video dataset for change detection benchmarking, " IEEE Transactions on Image Processing, vol. 23, no. 11, pp. 4663-4679, November 2014.
C. Cuevas, E. Yanez, and N. Garcia, "Labeled dataset for integral evaluation of moving object detection algorithms: LASIESTA, " Computer Vision and Image Understanding, vol. 152, pp. 103-117, November 2016.
J. Davis and M. Goadrich, "The relationship between precision-recall and ROC curves, " in International Conference on Machine Learning (ICML), Pittsburgh, Pennsylvania, June 2006, pp. 233-240.
G. Forman and M. Scholz, "Apples-to-apples in crossvalidation studies: Pitfalls in classifier performance measurement, " ACM SIGKDD Explorations Newsletter, vol. 12, no. 1, pp. 49-57, November 2010.
K. Boyd, V. Costa, J. Davis, and C. Page, "Unachievable region in precision-recall space and its effect on empirical evaluation, " in International Conference on Machine Learning (ICML), Edinburgh, UK, June-July 2012, pp. 639-646.
C. Goutte and E. Gaussier, "A probabilistic interpretation of precision, recall and F-score, with implication for evaluation, " in Advances in Information Retrieval (Proceedings of ECIR), Santiago de Compostela, Spain, March 2005, vol. 3408 of Lecture Notes in Computer Science, pp. 345-359, Springer.
J. Bertrand, Calcul des probabilités, Gauthier-Villars et fils, 1889.
P. Flach, "The geometry of ROC space: Understanding machine learning metrics through ROC isometrics, " in International Conference on Machine Learning (ICML), Washington, USA, August 2003, pp. 194-201.
T. Fawcett, "An introduction to ROC analysis, " Pattern Recognition Letters, vol. 27, no. 8, pp. 861-874, June 2006.
