SoccerNet-v3; Soccer; Football; Artificial intelligence; Deep learning; Dataset; Human tracking; Player tracking; Camera calibration; Player detection; Sports; Sport
Abstract :
[en] AbstractSoccer videos are a rich playground for computer vision, involving many elements, such as players, lines, and specific objects. Hence, to capture the richness of this sport and allow for fine automated analyses, we release SoccerNet-v3, a major extension of the SoccerNet dataset, providing a wide variety of spatial annotations and cross-view correspondences. SoccerNet’s broadcast videos contain replays of important actions, allowing us to retrieve a same action from different viewpoints. We annotate those live and replay action frames showing same moments with exhaustive local information. Specifically, we label lines, goal parts, players, referees, teams, salient objects, jersey numbers, and we establish player correspondences between the views. This yields 1,324,732 annotations on 33,986 soccer images, making SoccerNet-v3 the largest dataset for multi-view soccer analysis. Derived tasks may benefit from these annotations, like camera calibration, player localization, team discrimination and multi-view re-identification, which can further sustain practical applications in augmented reality and soccer analytics. Finally, we provide Python codes to easily download our data and access our annotations.
Research Center/Unit :
Montefiore Institute - Montefiore Institute of Electrical Engineering and Computer Science - ULiège Telim
Disciplines :
Electrical & electronics engineering
Author, co-author :
Cioppa, Anthony ✱; Université de Liège - ULiège > Montefiore Institute of Electrical Engineering and Computer Science
Deliège, Adrien ✱; Université de Liège - ULiège > Département d'électricité, électronique et informatique (Institut Montefiore) > Télécommunications
Giancola, Silvio ✱
Ghanem, Bernard
Van Droogenbroeck, Marc ; Université de Liège - ULiège > Montefiore Institute of Electrical Engineering and Computer Science
✱ These authors have contributed equally to this work.
Language :
English
Title :
Scaling up SoccerNet with multi-view spatial localization and re-identification
Applications et Recherche pour une Intelligence Artificielle de Confiance (ARIAC)
Funders :
F.R.S.-FNRS - Fonds de la Recherche Scientifique [BE] SPW - Service Public de Wallonie [BE] KAUST - King Abdullah University of Science and Technology [SA]
Funding number :
2010235
Funding text :
This work was supported by the Service Public de Wallonie (SPW) Recherche under the DeepSport project and Grant N°. 2010235 (ARIAC by https://DigitalWallonia4.ai), the FRIA, and KAUST Office of Sponsored Research through the Visual Computing Center (VCC) funding.
Lange, D. Market size of the European professional soccer market from 2006/07 to 2019/20. https://www.statista.com/statistics/261223/european-soccer-market-total-revenue/ (2021).
Moeslund, T. B., Thomas, G. & Hilton, A. Computer vision in sports (Springer, 2014).
Thomas, G., Gade, R., Moeslund, T. B., Carr, P. & Hilton, A. Computer vision for sports: current applications and research topics. Comp. Vision and Image Understanding 159, 3–18, 10.1016/j.cviu.2017.04.011 (2017). DOI: 10.1016/j.cviu.2017.04.011
Cioppa, A. et al. A context-aware loss function for action spotting in soccer videos. In IEEE Int. Conf. Comput. Vis. Pattern Recogn. (CVPR), 13126–13136, https://doi.org/10.1109/CVPR42600.2020.01314 (2020).
Deliège, A. et al. SoccerNet-v2: a dataset and benchmarks for holistic understanding of broadcast soccer videos. In IEEE Int. Conf. Comput. Vis. Pattern Recogn. Work. (CVPRW), 4508–4519, https://doi.org/10.1109/CVPRW53098.2021.00508 (2021).
Giancola, S. & Ghanem, B. Temporally-aware feature pooling for action spotting in video broadcasts. In IEEE Int. Conf. Comput. Vis. Pattern Recogn. Work. (CVPRW), 4485–4494, https://doi.org/10.1109/CVPRW53098.2021.00506 (2021).5
Richly, K., Moritz, F. & Schwarz, C. Utilizing artificial neural networks to detect compound events in spatio-temporal soccer data. In Proc. SIGKDD Work. MiLeTS, 1–7 (2017).
Tomei, M., Baraldi, L., Calderara, S., Bronzin, S. & Cucchiara, R. RMS-Net: regression and masking for soccer event spotting. In IEEE Int. Conf. Pattern Recogn. (ICPR), 7699–7706, https://doi.org/10.1109/ICPR48806.2021.9412268 (2020).
Khaustov, V. & Mozgovoy, M. Recognizing events in spatiotemporal soccer data. Applied Sciences 10, 1–12, 10.3390/app10228046 (2020). DOI: 10.3390/app10228046
Zhou, X., Kang, L., Cheng, Z., He, B. & Xin, J. Feature combination meets attention: Baidu soccer embeddings and transformer based temporal detection. Preprint at https://doi.org/10.48550/arXiv.2106.14447 (2021).
Cioppa, A., Deliège, A., Istasse, M., De Vleeschouwer, C. & Van Droogenbroeck, M. ARTHuS: adaptive real-time human segmentation in sports through online distillation. In IEEE Int. Conf. Comput. Vis. Pattern Recogn. Work. (CVPRW), 2505–2514, https://doi.org/10.1109/CVPRW.2019.00306 (2019).
Cioppa, A. et al. Multimodal and multiview distillation for real-time player detection on a football field. In IEEE Int. Conf. Comput. Vis. Pattern Recogn. Work. (CVPRW), 3846–3855, https://doi.org/10.1109/CVPRW50498.2020.00448 (2020).
Hurault, S., Ballester, C. & Haro, G. Self-supervised small soccer player detection and tracking. In Int. Work. Multimedia Content Analysis in Sports, 9–18, https://doi.org/10.1145/3422844.3423054 (2020).
Manafifard, M., Ebadi, H. & Abrishami Moghaddam, H. A survey on player tracking in soccer videos. Comp. Vision and Image Understanding 159, 19–46, 10.1016/j.cviu.2017.02.002 (2017). DOI: 10.1016/j.cviu.2017.02.002
Kamble, P. R., Keskar, A. G. & Bhurchandi, K. M. A deep learning ball tracking system in soccer videos. Opto-Electronics Review 27, 58–69, 10.1016/j.opelre.2019.02.003 (2019). DOI: 10.1016/j.opelre.2019.02.003
Suzuki, G., Takahashi, S., Ogawa, T. & Haseyama, M. Team tactics estimation in soccer videos based on a deep extreme learning machine and characteristics of the tactics. IEEE Access 7, 153238–153248, 10.1109/ACCESS.2019.2946378 (2019). DOI: 10.1109/ACCESS.2019.2946378
Arbués Sangüesa, A., Martín, A., Fernández, J., Ballester, C. & Haro, G. Using player’s body-orientation to model pass feasibility in soccer. In IEEE Int. Conf. Comput. Vis. Pattern Recogn. Work. (CVPRW), 3875–3884, https://doi.org/10.1109/CVPRW50498.2020.00451 (2020).
Decroos, T., Bransen, L., Van Haaren, J. & Davis, J. Actions speak louder than goals: valuing player actions in soccer. In ACM Int. Conf. Knowl. Disc. and Data Mining (KDD), 1851–1861, https://doi.org/10.1145/3292500.3330758 (2019).
Cioppa, A., Deliège, A. & Van Droogenbroeck, M. A bottom-up approach based on semantics for the interpretation of the main camera stream in soccer games. In IEEE Int. Conf. Comput. Vis. Pattern Recogn. Work. (CVPRW), 1846–1855, https://doi.org/10.1109/CVPRW.2018.00229 (2018).
Agyeman, R., Muhammad, R. & Choi, G. S. Soccer video summarization using deep learning. In IEEE Conf. Multimedia Inf. Process. Retr. (MIPR), 270–273, https://doi.org/10.1109/MIPR.2019.00055 (2019).
Sanabria, M., Sherly, Precioso, F. & Menguy, T. A deep architecture for multimodal summarization of soccer games. In Int. Work. Multimedia Content Anal. Sports (MMSports), 16–24, https://doi.org/10.1145/3347318.3355524 (2019).
He, K., Zhang, X., Ren, S. & Sun, J. Deep residual learning for image recognition. In IEEE Int. Conf. Comput. Vis. Pattern Recogn. (CVPR), 770–778, https://doi.org/10.1109/CVPR.2016.90 (2016).
Tan, M. & Le, Q. V. EfficientNet: rethinking model scaling for convolutional neural networks. In Int. Conf. Mach. Learn. (ICML), 6105–6114 (2019).
Deng, J. et al. ImageNet: a large-scale hierarchical image database. In IEEE Int. Conf. Comput. Vis. Pattern Recogn. (CVPR), 248–255, https://doi.org/10.1109/CVPR.2009.5206848 (2009).
Lin, T.-Y. et al. Microsoft COCO: common objects in context. In Eur. Conf. Comput. Vision (ECCV), vol. 8693 of Lect. Notes Comput. Sci. 740–755, https://doi.org/10.1007/978-3-319-10602-1_48 (Springer, 2014).
Homayounfar, N., Fidler, S. & Urtasun, R. Sports field localization via deep structured models. In IEEE Int. Conf. Comput. Vis. Pattern Recogn. (CVPR), 4012–4020, https://doi.org/10.1109/CVPR.2017.427 (2017).
Biermann, H. et al. A unified taxonomy and multimodal dataset for events in invasion games. Preprint at https://doi.org/10.48550/arXiv.2108.11149 (2021).
Giancola, S., Amine, M., Dghaily, T. & Ghanem, B. SoccerNet: a scalable dataset for action spotting in soccer videos. In IEEE Int. Conf. Comput. Vis. Pattern Recogn. Work. (CVPRW), 1711–1721, https://doi.org/10.1109/CVPRW.2018.00223 (2018).
Pappalardo, L. et al. A public data set of spatio-temporal match events in soccer competitions. Scientific Data 6, 1–15, 10.1038/s41597-019-0247-7 (2019). DOI: 10.1038/s41597-019-0247-7
Pappalardo, L. et al. Metadata record for: a public data set of spatio-temporal match events in soccer competitions, figshare, 10.6084/m9.figshare.9711164.v2 (2020).
Yu, J. et al. Comprehensive dataset of broadcast soccer videos. In IEEE Conf. Multimedia Inf. Process. Retr. (MIPR), 418–423, https://doi.org/10.1109/MIPR.2018.00090 (2018).
Jiang, Y., Cui, K., Chen, L., Wang, C. & Xu, C. SoccerDB: A large-scale database for comprehensive video understanding. In Int. Work. Multimedia Content Anal. Sports (MMSports), 1–8, https://doi.org/10.1145/3422844.3423051 (2020).
He, K., Gkioxari, G., Dollár, P. & Girshick, R. Mask R-CNN. In IEEE Int. Conf. Comput. Vision (ICCV), 2980–2988, https://doi.org/10.1109/ICCV.2017.322 (2017).
Sha, L. et al. End-to-end camera calibration for broadcast videos. In IEEE Int. Conf. Comput. Vis. Pattern Recogn. (CVPR), 13627–13636, https://doi.org/10.1109/CVPR42600.2020 (2020).
Cioppa, A. et al. Camera calibration and player localization in SoccerNet-v2 and investigation of their representations for action spotting. In IEEE Int. Conf. Comput. Vis. Pattern Recogn. Work. (CVPRW), 4537–4546, https://doi.org/10.1109/CVPRW53098.2021.00511 (2021).
Kurach, K. et al. Google research football: a novel reinforcement learning environment. AAAI Conf. Artificial Intell. 34, 4501–4510, 10.1609/aaai.v34i04.5878 (2020). DOI: 10.1609/aaai.v34i04.5878
Rematas, K., Kemelmacher-Shlizerman, I., Curless, B. & Seitz, S. Soccer on your tabletop. In IEEE Int. Conf. Comput. Vis. Pattern Recogn. (CVPR), 4738–4747, https://doi.org/10.1109/CVPR.2018.00498 (2018).
Morra, L. et al. Slicing and dicing soccer: automatic detection of complex events from spatio-temporal data. In Int. Conf. Image Anal. and Recognit. (ICIAR), vol. 12131 of Lect. Notes Comput. Sci. 107–121, https://doi.org/10.1007/978-3-030-50347-5_11 (2020).
Cioppa, A. et al. SoccerNet-v3: scaling up SoccerNet with multi-view spatial localization and re-identification, figshare, 10.6084/m9.figshare.c.5668645 (2022).
European Commission. Proposal for a regulation of the European parliament and of the council laying down harmonised rules on artificial intelligence (artificial intelligence ACT) and amending certain union legislative ACTs. https://eur-lex.europa.eu/legal-content/EN/TXT/HTML/?uri=CELEX:52021PC0206&from=EN (2021).
