Towards an AI-Powered Video Assistant Referee System (VARS) for Association Football

[en] Over the past decade, the technology used by referees in football has improved substantially, enhancing the fairness and accuracy of decisions. This progress has culminated in the implementation of the Video Assistant Referee (VAR), an innovation that enables backstage referees to review incidents on the pitch from multiple points of view. However, the VAR is currently limited to professional leagues due to its expensive infrastructure and the lack of referees worldwide. In this paper, we present the Video Assistant Referee System (VARS) that leverages the latest findings in multi-view video analysis. Our VARS achieves a new state-of-the-art on the SoccerNet-MVFoul dataset by recognizing the type of foul in $50\%$ of instances and the appropriate sanction in 46% of cases. Finally, we conducted a comparative study to investigate human performance in classifying fouls and their corresponding severity and compared these findings to our VARS. The results of our study highlight the potential of our VARS to reach human performance and support football refereeing across all levels of professional federations.

Research Center/Unit :

Montefiore Institute - Montefiore Institute of Electrical Engineering and Computer Science - ULiège
TELIM
VIULab

Disciplines :

Electrical & electronics engineering

Author, co-author :

Held, Jan ; Université de Liège - ULiège > Montefiore Institute of Electrical Engineering and Computer Science

Cioppa, Anthony ; Université de Liège - ULiège > Montefiore Institute of Electrical Engineering and Computer Science

Giancola, Silvio

Hamdi, Abdullah

Devue, Christel ; Université de Liège - ULiège > Département de Psychologie > Psychologie et neurosciences cognitives

Ghanem, Bernard

Van Droogenbroeck, Marc ; Université de Liège - ULiège > Département d'électricité, électronique et informatique (Institut Montefiore) > Télécommunications

Language :

English

Title :

Towards an AI-Powered Video Assistant Referee System (VARS) for Association Football

Publication date :

2025

Main work title :

Lecture Notes in Computer Science

Publisher :

Springer Nature Switzerland

ISBN/EAN :

978-3-03-195911-0
978-3-03-195910-3

Pages :

295-309

Peer reviewed :

Peer reviewed

Additional URL :

https://link.springer.com/content/pdf/10.1007/978-3-031-95911-0_21

Funders :

FRIA - Fonds pour la Formation à la Recherche dans l'Industrie et dans l'Agriculture
F.R.S.-FNRS - Fonds de la Recherche Scientifique

Available on ORBi :

since 23 June 2025

Statistics

Number of views

65 (3 by ULiège)

Number of downloads

129 (0 by ULiège)

More statistics

Scopus citations^®

Scopus citations^®
without self-citations

OpenCitations

OpenAlex citations

Bibliography

Cabado, B., et al.: Beyond the Premier: Assessing action spotting transfer capability across diverse domains. In: IEEE Conference on Computer Vision and Pattern Recognition Work (CVPRW), vol. 27, pp. 3386–3398 (2024)
Cioppa, A., et al.: A context-aware loss function for action spotting in soccer videos. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 13123–13133 (2020)
Cioppa, A., Deliège, A., Ul Huda, N., Gade, R., Van Droogenbroeck, M., Moeslund, T.B.: Multimodal and multiview distillation for real-time player detection on a football field. In: IEEE International Conference on computer Vision and Pattern Recognition Work. (CVPRW), CVsports, pp. 3846–3855, Seattle, WA, USA (2020)
Cioppa, A., et al.: SoccerNet-tracking: multiple object tracking dataset and benchmark in soccer videos. In: IEEE International Conference on Computer Vision and Pattern Recognition Working (CVPRW), CVsports, pp. 3490–3501 (2022)
Cioppa, A., et al.: SoccerNet 2023 challenges results. Sports Eng. 27(2), 1–18 (2024)
De Dios Crespo, J.: The Contribution of VARs to Fairness in Sport, chap. 2, pp. 23–35. Routledge, New York City (2021)
Deutscher Fußball-Bund (DFB): Anzahl aktiver schiedsrichter/-innen bis 2022 (2022). https://www.dfb.de/verbandsstruktur/dfb-zentrale/
Fan, H., Xiong, B., Mangalam, K., Li, Y., Yan, Z., Malik, J., Feichtenhofer, C.: Multiscale vision transformers. In: IEEE International Conference on Computer Vision (ICCV), pp. 6804–6815 (2021)
Gautam, S., et al.: SoccerNet-echoes: a soccer game audio commentary dataset. In: International Symposium on Multimedia (ISM), pp. 71–78 (2024)
Giancola, S., Amine, M., Dghaily, T., Ghanem, B.: SoccerNet: a scalable dataset for action spotting in soccer videos. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition Working (CVPRW), pp. 1792–179210 (2018)
Giancola, S., et al.: Towards active learning for action spotting in association football videos. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition Working (CVPRW), pp. 5098–5108 (2023)
Giancola, S., Ghanem, B.: Temporally-aware feature pooling for action spotting in soccer broadcasts. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition Working (CVPRW), pp. 4485–4494 (2021)
Held, J., Cioppa, A., Giancola, S., Hamdi, A., Ghanem, B., Van Droogenbroeck, M.: VARS: Video assistant referee system for automated soccer decision making from multiple views. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition Working (CVPRW), pp. 5086–5097 (2023)
Held, J., Itani, H., Cioppa, A., Giancola, S., Ghanem, B., Van Droogenbroeck, M.: X-VARS: introducing explainability in football refereeing with multi-modal large language models. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition Working (CVPRW), vol. 9, pp. 3267–3279 (2024)
Holder, U., Ehrmann, T., König, A.: Monitoring experts: insights from the introduction of video assistant referee (VAR) in elite football. J. Bus. Econ. 92(2), 285–308 (2021)
IFAB: Laws of the game. Technical report, The International Football Association Board, Zurich, Switzerland (2022)
Kay, W., et al.: The kinetics human action video dataset (2017). arXiv:abs/1705.06950
Leduc, A., Cioppa, A., Giancola, S., Ghanem, B., Van Droogenbroeck, M.: SoccerNet-Depth: a scalable dataset for monocular depth estimation in sports videos. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition Working (CVPRW), vol. 12, pp. 3280–3282 (2024)
Li, Y., et al.: MViTv2: improved multiscale vision transformers for classification and detection. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4794–4804 (2022)
López, A.M.: Average time needed for a video assistant referee (VAR) intervention in Brazil in 2019 and 2020 (2023). https://www.statista.com/statistics/1010093/average-time-video-assistant-referee-checking-brazil/
MacMahon, C., Helsen, W.F., Starkes, J.L., Weston, M.: Decision-making skills and deliberate practice in elite association football referees. J. Sports Sci. 25(1), 65–78 (2007)
Magera, F., Hoyoux, T., Barnich, O., Van Droogenbroeck, M.: A universal protocol to benchmark camera calibration for sports. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition Working (CVPRW), pp. 3335–3346 (2024)
Mkhallati, H., Cioppa, A., Giancola, S., Ghanem, B., Van Droogenbroeck, M.: SoccerNet-caption: dense video captioning for soccer broadcasts commentaries. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition Working (CVPRW), pp. 5074–5085 (2023)
Somers, V., De Vleeschouwer, C., Alahi, A.: Body part-based representation learning for occluded person re-identification. In: IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 1613–1623 (2023)
Somers, V., et al.: SoccerNet game state reconstruction: end-to-end athlete tracking and identification on a Minimap. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition Working (CVPRW), pp. 3293–3305 (2024)
Spitz, J., Put, K., Wagemans, J., Williams, A.M., Helsen, W.F.: Visual search behaviors of association football referees during assessment of foul play situations. Cogn. Res.: Principles Implications 1(1), 1–11 (2016). https://doi.org/10.1186/s41235-016-0013-8
Vandeghen, R., Cioppa, A., Van Droogenbroeck, M.: Semi-supervised training to improve player and ball detection in soccer. In: IEEE International Conference on Computer Vision and Pattern Recognition Working (CVPRW), CVsports, pp. 3480–3489 (2022)
Yang, Z., Wang, L.: Learning relationships for multi-view 3D object recognition. In: IEEE International Conference on Computer Vision (ICCV), pp. 7504–7513 (2019)
Zeppenfeld, B.: Anzahl aktiver schiedsrichter / schiedsrichterinnen des deutschen fußball bundes (DFB) von 2018/2019 bis 2022/2023. https://de.statista.com/statistik/daten/studie/1243626/umfrage/dfb-anzahl-aktiver-schiedsrichter/(2023)