multiview distillation; multimodal distillation; knowledge distillation; deep learning; player detection; thermal camera; fisheye camera; real-time detection; football; ViBe
Abstract :
[en] Monitoring the occupancy of public sports facilities is essential to assess their use and to motivate their construction in new places. In the case of a football field, the area to cover is large, thus several regular cameras should be used, which makes the setup expensive and complex. As an alternative, we developed a system that detects players from a unique cheap and wide-angle fisheye camera assisted by a single narrow-angle thermal camera. In this work, we train a network in a knowledge distillation approach in which the student and the teacher have different modalities and a different view of the same scene. In particular, we design a custom data augmentation combined with a motion detection algorithm to handle the training in the region of the fisheye camera not covered by the thermal one. We show that our solution is effective in detecting players on the whole field filmed by the fisheye camera. We evaluate it quantitatively and qualitatively in the case of an online distillation, where the student detects players in real time while being continuously adapted to the latest video conditions.
Research center :
Montefiore Institute - Montefiore Institute of Electrical Engineering and Computer Science - ULiège Telim
Disciplines :
Electrical & electronics engineering
Author, co-author :
Cioppa, Anthony ✱; Université de Liège - ULiège > Dép. d'électric., électron. et informat. (Inst.Montefiore) > Télécommunications
Deliège, Adrien ✱; Université de Liège - ULiège > Dép. d'électric., électron. et informat. (Inst.Montefiore) > Télécommunications
Noor, Ul Huda
Gade, Rikke
Van Droogenbroeck, Marc ; Université de Liège - ULiège > Dép. d'électric., électron. et informat. (Inst.Montefiore) > Télécommunications
Moeslund, Thomas B.
✱ These authors have contributed equally to this work.
Language :
English
Title :
Multimodal and multiview distillation for real-time player detection on a football field
Publication date :
June 2020
Event name :
IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) - CVSports
Event organizer :
IEEE
Event place :
Seattle, United States - Washington
Event date :
from 14-06-2020 to 19-06-2020
Audience :
International
Journal title :
IEEE Conference on Computer Vision and Pattern Recognition. Proceedings
ISSN :
1063-6919
eISSN :
2575-7075
Publisher :
IEEE Computer Society, Washington, United States - District of Columbia
Pages :
3846-3855
Peer reviewed :
Peer reviewed
Name of the research project :
DeepSport
Funders :
Fonds pour la formation à la Recherche dans l'Industrie et dans l'Agriculture (Communauté française de Belgique) - FRIA DGTRE - Région wallonne. Direction générale des Technologies, de la Recherche et de l'Énergie
M. Archana and M. Kalaiselvi Geetha. An efficient ball and player detection in broadcast tennis video. In Intelligent Systems Technologies and Applications, pages 427-436, Cham, 2016. Springer International Publishing. 2
Olivier Barnich and Marc Van Droogenbroeck. ViBe: A universal background subtraction algorithm for video sequences. IEEE Transactions on Image Processing, 20(6): 1709-1724, June 2011. 4
Massimo Bertozzi, Luca Castangia, Stefano Cattani, Antonio Prioletti, and Pietro Versari. 360 detection and tracking algorithm of both pedestrian and vehicle using fisheye images. In IEEE Intelligent Vehicles Symposium (IV), pages 132-137, June 2015. 2
Jean-Yves Bouguet. Camera calibration toolbox for Matlab, 2014. 3
Kendrick Boyd, Vítor Santos Costa, Jesse Davis, and C. David Page. Unachievable region in precision-recall space and its effect on empirical evaluation. In Proceedings of the 29th International Coference on International Conference on Machine Learning (ICML), ICML'12, page 1619-1626, Madison, WI, USA, 2012. Omnipress. 6
Matija Buric, Marina Ivasic-Kos, and Miran Pobar. Player tracking in sports videos. In IEEE International Conference on Cloud Computing Technology and Science (CloudCom), pages 334-340, Dec. 2019. 2
Anthony Cioppa, Adrien Deliège, Maxime Istasse, Christophe De Vleeschouwer, and Marc Van Droogenbroeck. ARTHuS: Adaptive Real-Time Human Segmentation in Sports Through Online Distillation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2019. 3, 6, 7
Anthony Cioppa, Adrien Deliège, and Marc Van Droogenbroeck. A bottom-up approach based on semantics for the interpretation of the main camera stream in soccer games. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pages 1846-1855, June 2018. 3
Congxia Dai, Yunfei Zheng, and Xin Li. Layered representation for pedestrian detection and tracking in infrared imagery. In IEEE Conference on Computer Vision and Pattern Recognition(CVPR) Workshops, Sep. 2005. 2
Mark Everingham, Luc Van Gool, Christopher K. I. Williams, JohnWinn, and Andrew Zisserman. The PASCAL visual object classes (VOC) challenge. International Journal of Computer Vision, 88(2): 303-338, June 2010. 6
Hayden Faulkner and Anthony Dick. AFL player detection and tracking. In International Conference on Digital Image Computing: Techniques and Applications (DICTA), pages 1-8, Nov. 2015. 2
Rikke Gade, Anders Jorgensen, and Thomas B. Moeslund. Occupancy analysis of sports arenas using thermal imaging. In International Conference on Computer Vision Theory and Applications, pages 277-283. SCITEPRESS Digital Library, 2012. 2
Rikke Gade, Anders Jorgensen, and Thomas B. Moeslund. Long-term occupancy analysis using graph-based optimisation in thermal imagery. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3698-3705, US, 2013. IEEE Computer Society Press. 2
Rikke Gade and Thomas B. Moeslund. Thermal cameras and applications: A survey. Machine Vision and Applications, 25(1): 245-262, 2014. 2
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask R-CNN. In IEEE International Conference on Computer Vision (ICCV), pages 2980-2988, Oct. 2017. 3
Duyoung Heo, Eunju Lee, and Byoung Chul Ko. Pedestrian detection at night using deep neural networks and saliency maps. Journal of Imaging Science and Technology, 61: 60403-1-60403-9(9), 2017. 2
Christian Herrmann, Thomas Müller, Dieter Willersinn, and Jürgen Beyerer. Real-time person detection in low-resolution thermal infrared imagery with MSER and CNNs. In SPIE Security+Defence, volume 9987, 2016. 2
Noor Ul Huda, Kasper Halkjar Jensen, Rikke Gade, and Thomas B. Moeslund. Estimating the number of soccer players using simulation-based occlusion handling. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pages 1937-1946, US, 2018. IEEE. 2
Noor Ul Huda, Bolette D. Hansen, Rikke Gade, and Thomas B. Moeslund. Occupancy analysis of soccer fields using wide-angle lens. In International Conference on Signal-Image Technology Internet-Based Systems (SITIS), pages 354-359, Dec. 2017. 2
Noor Ul Huda, Bolette D. Hansen, Rikke Gade, and Thomas B. Moeslund. The effect of diverse dataset for transfer learning in thermal person detection. Sensors, 20(7): 1982, Apr 2020. 2, 5
Zdravko Ivankovic, Branko Markoski, Miodrag Ivkovic, Dragica Radosav, and Predrag Pecev. Adaboost in basketball player identification. In IEEE International Symposium on Computational Intelligence and Informatics (CINTI), pages 151-156, Nov. 2012. 2
Hyungtae Kim, Eunjung Chae, Gwanghyun Jo, and Joonki Paik. Fisheye lens-based surveillance camera for wide fieldof-view monitoring. In IEEE International Conference on Consumer Electronics (ICCE), pages 505-506, 2015. 2
Hyungtae Kim, Jaehoon Jung, and Joonki Paik. Fisheye lens camera based surveillance system for wide field of view monitoring. Optik, 127(14): 5636-5646, 2016. 2
Dan Levi and Shai Silberstein. Tracking and motion cues for rear-view pedestrian detection. In IEEE International Conference on Intelligent Transportation Systems, pages 664-671, Sep. 2015. 2
Wei Li, Dequan Zheng, Tiejun Zhao, and Mengda Yang. An effective approach to pedestrian detection in thermal imagery. In International Conference on Natural Computation, pages 325-329, May 2012. 2
Zahid Mahmood, Tauseef Ali, and Shahid Khattak. Automatic player detection and recognition in images using adaboost. In International Bhurban Conference on Applied Sciences Technology (IBCAST), pages 64-69, Jan. 2012. 2
Ezio Malis and Manuel Vargas. Deeper understanding of the homography decomposition for vision-based control. Research Report RR-6303, INRIA, 2007. 3
Van Tuan Nguyen, Thanh Binh Nguyen, and Sun-Tae Chung. ConvNets and AGMM based real-time human detection under fisheye camera for embedded surveillance. In International Conference on Information and Communication Technology Convergence (ICTC), pages 840-845. IEEE, 2016. 2
Cristina Palmero, Albert Clapés, Chris Bahnsen, Andreas Mogelmose, Thomas B. Moeslund, and Sergio Escalera. Multi-modal RGB-depth-thermal human body segmentation. International Journal of Computer Vision, 118(2): 217-239, 2016. 3
Miran Pobar and Marina Ivasic-Kos. Mask R-CNN and optical flow based method for detection and marking of handball actions. In International Congress on Image and Signal Processing, BioMedical Engineering and Informatics (CISPBMEI), pages 1-6, Oct. 2018. 2
Upendra M. Rao and Umesh C. Pati. A novel algorithm for detection of soccer ball and player. In International Conference on Communications and Signal Processing (ICCSP), pages 344-348, Apr. 2015. 2
Joseph Redmon and Ali Farhadi. YOLOv3: An incremental improvement. CoRR, abs/1804. 02767, 2018. 5
Vito Renò, Nicola Mosca, Massimiliano Nitti, Tiziana Dorazio, Donato Campagnoli, Andrea Prati, and Ettore Stella. Tennis player segmentation for semantic behavior analysis. In IEEE International Conference on Computer Vision (ICCV) Workshop, pages 718-725, Dec. 2015. 2
Melike Sah and Cem Direko?glu. Evaluation of image representations for player detection in field sports using convolutional neural networks. In International Conference on Theory and Application of Fuzzy Systems and Soft Computing (ICAFS), pages 107-115, Cham, 2019. Springer International Publishing. 2
Mamoru Saito, Katsuhisa Kitaguchi, Gun Kimura, and Masafumi Hashimoto. People detection and tracking from fish-eye image based on probabilistic appearance model. In SICE Annual Conference 2011, pages 435-440, Sep. 2011. 2
Graham Thomas, Rikke Gade, Thomas B. Moeslund, Peter Carr, and Adrian Hilton. Computer vision for sports: Current applications and research topics. Computer Vision and Image Understanding, 159: 3-18, 2017. Computer Vision in Sports. 2
Paulius Tumas, Arturas Jonkus, and Arturas Serackis. Acceleration of HOG based pedestrian detection in FIR camera video stream. In Open Conference of Electrical, Electronic and Information Sciences (eStream), pages 1-4, Apr. 2018. 2
Lin Wang and Kuk-Jin Yoon. Knowledge distillation and student-teacher learning for visual intelligence: A review and new outlooks. CoRR, 2020. 3
Tsaipei Wang, Chia-Wei Chang, and Yu-Shan Wu. Template-based people detection using a single downwardviewing fisheye camera. In International Symposium on Intelligent Signal Processing and Communication Systems (ISPACS), pages 719-723, Nov. 2017. 2
Tsaipei Wang and Chih-Hao Liao. People detection in downward-viewing fisheye camera networks using fuzzy integral. In IEEE International Conference on Fuzzy Systems (FUZZ-IEEE), pages 1-5, June 2019. 2
Yukun Yang, Min Xu, WannengWu, Ruiheng Zhang, and Yu Peng. 3D multiview basketball players detection and localization based on probabilistic occupancy. In Digital Image Computing: Techniques and Applications (DICTA), pages 1-8, Dec. 2018. 2
Hui Zhang, Baojun Zhao, Linbo Tang, Jianke Li, and Jianke Li. Variational-based contour tracking in infrared imagery. In International Congress on Image and Signal Processing, pages 1-5, Oct 2009. 2
Lijing Zhang, Yao Lu, Ge Song, and Hanfeng Zheng. RCCNN: Reverse connected convolutional neural network for accurate player detection. In Pacific Rim International Conference on Artificial Intelligence (PRICAI): Trends in Artificial Intelligence, pages 438-446, Cham, 2018. Springer International Publishing. 2
