Computer Science - Computer Vision and Pattern Recognition; Computer Science - Learning; Online distillation; Knowledge distillation; Continual learning; Autonomous vehicle; Test time adaptation
Abstract :
[en] In recent years, online distillation has emerged as a powerful technique for
adapting real-time deep neural networks on the fly using a slow, but accurate
teacher model. However, a major challenge in online distillation is
catastrophic forgetting when the domain shifts, which occurs when the student
model is updated with data from the new domain and forgets previously learned
knowledge. In this paper, we propose a solution to this issue by leveraging the
power of continual learning methods to reduce the impact of domain shifts.
Specifically, we integrate several state-of-the-art continual learning methods
in the context of online distillation and demonstrate their effectiveness in
reducing catastrophic forgetting. Furthermore, we provide a detailed analysis
of our proposed solution in the case of cyclic domain shifts. Our experimental
results demonstrate the efficacy of our approach in improving the robustness
and accuracy of online distillation, with potential applications in domains
such as video surveillance or autonomous driving. Overall, our work represents
an important step forward in the field of online distillation and continual
learning, with the potential to significantly impact real-world applications.
Research Center/Unit :
Montefiore Institute - Montefiore Institute of Electrical Engineering and Computer Science - ULiège Telim
Henry, Maxim ; Université de Liège - ULiège > Montefiore Institute of Electrical Engineering and Computer Science
Ghanem, Bernard
Van Droogenbroeck, Marc ; Université de Liège - ULiège > Département d'électricité, électronique et informatique (Institut Montefiore) > Télécommunications
✱ These authors have contributed equally to this work.
Language :
English
Title :
Online Distillation with Continual Learning for Cyclic Domain Shifts
Publication date :
03 April 2023
Event name :
2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)
Event date :
17-24 June 2023
Journal title :
IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops
ISSN :
2160-7508
eISSN :
2160-7516
Publisher :
IEEE
Pages :
2437-2446
Peer reviewed :
Peer reviewed
Funders :
SPW - Public Service of Wallonia MecaTech F.R.S.-FNRS - Fonds de la Recherche Scientifique
Funding number :
2010235; C8650
Funding text :
A. Cioppa is funded by the F.R.S.- FNRS. A. Halin is funded by the Walloon region (Ser- vice Public de Wallonie Recherche, Belgium) under grant No. 2010235 (ARIAC by DIGITALWALLONIA4.AI). M. Henry is funded by PIT MecaTech, Belgium, under grant No. C8650 (ReconnAIssance). This work was par- tially supported by the King Abdullah University of Science and Technology (KAUST) Office of Sponsored Research (OSR) under Award No. OSR-CRG2021-4648.
scite shows how a scientific paper has been cited by providing the context of the citation, a classification describing whether it supports, mentions, or contrasts the cited claim, and a label indicating in which section the citation was made.
Bibliography
Motasem Alfarra, Zhipeng Cai, Adel Bibi, Bernard Ghanem, and Matthias Müller. SimCS: Simulation for online domainincremental continual segmentation. CoRR, abs/2211. 16234, 2022.
Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) to forget. In Eur. Conf. Comput. Vis. (ECCV), volume 11207 of Lect. Notes Comput. Sci., pages 144-161. Springer Int. Publ., 2018.
Rahaf Aljundi, Lucas Caccia, Eugene Belilovsky, Massimo Caccia, Laurent Charlin, and Tinne Tuytelaars. Online continual learning with maximally interfered retrieval. In Adv. Neural Inf. Process. Syst. (NeurIPS), 2019.
Fatemeh Azimi, Sebastian Palacio, Federico Raue, Jorn Hees, Luca Bertinetto, and Andreas Dengel. Self-supervised test-time adaptation on video data. In IEEE/CVF Winter Conf. Appl. Comput. Vis. (WACV), pages 2603-2612, Waikoloa, HI, USA, Jan. 2022. Inst. Electr. Electron. Eng. (IEEE).
Olivier Barnich and Marc Van Droogenbroeck. ViBe: A universal background subtraction algorithm for video sequences. IEEE Trans. Image Process., 20(6): 1709-1724, Jun. 2011.
Lucas Caccia, Rahaf Aljundi, Nader Asadi, Tinne Tuytelaars, Joelle Pineau, and Eugene Belilovsky. New insights on reducing abrupt representation change in online continual learning. In Int. Conf. Learn. Represent. (ICLR), 2022.
Zhipeng Cai, Ozan Sener, and Vladlen Koltun. Online continual learning with natural distribution shifts: An empirical study with visual data. In IEEE Int. Conf. Comput. Vis. (ICCV), pages 8261-8270, Montréal, Can., Oct. 2021. Inst. Electr. Electron. Eng. (IEEE).
Arslan Chaudhry, Puneet K. Dokania, Thalaiyasingam Ajanthan, and Philip H. S. Torr. Riemannian walk for incremental learning: Understanding forgetting and intransigence. In Eur. Conf. Comput. Vis. (ECCV), volume 11215 of Lect. Notes Comput. Sci., pages 556-572. Springer Int. Publ., 2018.
Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K. Dokania, Philip H. S. Torr, and Marc'Aurelio Ranzato. Continual learning with tiny episodic memories. In Int. Conf. Mach. Learn. (ICML), 2019.
Anthony Cioppa, Adrien Deliege, Maxime Istasse, Christophe De Vleeschouwer, and Marc Van Droogenbroeck. ARTHuS: Adaptive real-time human segmentation in sports through online distillation. In IEEE Int. Conf. Comput. Vis. Pattern Recognit. Work. (CVPRW), CVsports, pages 2505-2514, Long Beach, CA, USA, Jun. 2019. Inst. Electr. Electron. Eng. (IEEE).
Anthony Cioppa, Adrien Deliège, Noor Ul Huda, Rikke Gade, Marc Van Droogenbroeck, and Thomas B. Moeslund. Multimodal and multiview distillation for real-time player detection on a football field. In IEEE Int. Conf. Comput. Vis. Pattern Recognit. Work. (CVPRW), CVsports, pages 3846-3855, Seattle, WA, USA, Jun. 2020.
Anthony Cioppa, Adrien Deliège, and Marc Van Droogenbroeck. A bottom-up approach based on semantics for the interpretation of the main camera stream in soccer games. In IEEE Int. Conf. Comput. Vis. Pattern Recognit. Work. (CVPRW), CVsports, pages 1846-1855, Salt Lake City, UT, USA, Jun. 2018.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), pages 3213-3223, Las Vegas, NV, USA, Jun. 2016. Inst. Electr. Electron. Eng. (IEEE).
Natalia Díaz-Rodríguez, Vincenzo Lomonaco, David Filliat, and Davide Maltoni. Don't forget, there is more than forgetting: New metrics for continual learning. In Continual learning W., Neural Inf. Process. Syst. (NeurIPS), 2018.
Abolfazl Farahani, Sahar Voghoei, Khaled Rasheed, and Hamid R. Arabnia. A brief review of domain adaptation. Trans. Comput. Sci. Comput. Intell., pages 877-894, 2021.
Robert M. French. Catastrophic forgetting in connectionist networks. Trends Cogn. Sci., 3(4): 128-135, Apr. 1999.
Qiankun Gao, Chen Zhao, Bernard Ghanem, and Jian Zhang. R-DFCIL: Relation-guided representation learning for datafree class incremental learning. In Eur. Conf. Comput. Vis. (ECCV), volume 13683 of Lect. Notes Comput. Sci., pages 423-439. Springer Nat. Switz., 2022.
Prachi Garg, Rohit Saluja, Vineeth N. Balasubramanian, Chetan Arora, Anbumani Subramanian, and C. V. Jawahar. Multi-domain incremental learning for semantic segmentation. In IEEE/CVF Winter Conf. Appl. Comput. Vis. (WACV), pages 2080-2090, Waikoloa, HI, USA, Jan. 2022. Inst. Electr. Electron. Eng. (IEEE).
Yasir Ghunaim, Adel Bibi, Kumail Alhamoud, Motasem Alfarra, Hasan Abed Al Kader Hammoud, Ameya Prabhu, Philip H. S. Torr, and Bernard Ghanem. Real-time evaluation in online continual learning: A new hope. CoRR, abs/2302. 01047, 2023.
Jiangpeng He, Runyu Mao, Zeman Shao, and Fengqing Zhu. Incremental learning in online scenario. In IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), pages 13923-13932, Seattle, WA, USA, Jun. 2020. Inst. Electr. Electron. Eng. (IEEE).
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), pages 770-778, Las Vegas, NV, USA, Jun. 2016. Inst. Electr. Electron. Eng. (IEEE).
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In Int. Conf. Learn. Represent. (ICLR), 2019.
Oguzhan Fatih Kar, Teresa Yeo, Andrei Atanov, and Amir Zamir. 3D common corruptions and data augmentation. In IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), pages 18941-18952, New Orleans, LA, USA, Jun. 2022. Inst. Electr. Electron. Eng. (IEEE).
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks. Proc. National Acad. Sci. (PNAS), 114(13): 3521-3526, Mar. 2017.
Jinlong Li, Runsheng Xu, Jin Ma, Qin Zou, Jiaqi Ma, and Hongkai Yu. Domain adaptive object detection for autonomous driving under foggy weather. In IEEE/CVF Winter Conf. Appl. Comput. Vis. (WACV), pages 612-622, Waikoloa, HI, USA, Jan. 2023. Inst. Electr. Electron. Eng. (IEEE).
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE Trans. Pattern Anal. Mach. Intell., 40(12): 2935-2947, Dec. 2018.
Ming Liang and Xiaolin Hu. Recurrent convolutional neural network for object recognition. In IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), pages 3367-3375, Boston, MA, USA, Jun. 2015. Inst. Electr. Electron. Eng. (IEEE).
Hyesu Lim, Byeonggeun Kim, Jaegul Choo, and Sungha Choi. TTN: A domain-shift aware batch normalization in test-time adaptation. In Int. Conf. Learn. Represent. (ICLR), pages 1-19, 2023.
David Lopez-Paz and Marc'Aurelio Ranzato. Gradient episodic memory for continual learning. In Adv. Neural Inf. Process. Syst. (NeurIPS), 2017.
Yawei Luo, Liang Zheng, Tao Guan, Junqing Yu, and Yi Yang. Taking a closer look at domain shift: Category-level adversaries for semantics consistent domain adaptation. In IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), pages 2502-2511, Long Beach, CA, USA, Jun. 2019. Inst. Electr. Electron. Eng. (IEEE).
Michael McCloskey and Neal J. Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. Psychol. Learn. Motiv., pages 109-165, 1989.
Theodoros Panagiotakopoulos, Pier Luigi Dovesi, Linus Härenstam-Nielsen, and Matteo Poggi. Online domain adaptation for semantic segmentation in ever-changing conditions. In Eur. Conf. Comput. Vis. (ECCV), volume 13694 of Lect. Notes Comput. Sci., pages 128-146. Springer Nat. Switz., 2022.
Sébastien Piérard, Anthony Cioppa, Anäis Halin, Renaud Vandeghen, Maxime Zanella, Benôit Macq, Säid Mahmoudi, and Marc Van Droogenbroeck. Mixture domain adaptation to improve semantic segmentation in real-world surveillance. In IEEE/CVF Winter Conf. Appl. Comput. Vis. Work. (WACVW), pages 22-31, Waikoloa, HI, USA, Jan. 2023. Inst. Electr. Electron. Eng. (IEEE).
Ameya Prabhu, Philip H. S. Torr, and Puneet K. Dokania. GDumb: A simple approach that questions our progress in continual learning. In Eur. Conf. Comput. Vis. (ECCV), volume 12347 of Lect. Notes Comput. Sci., pages 524-540. Springer Int. Publ., 2020.
Christos Sakaridis, Dengxin Dai, and Luc Van Gool. ACDC: The adverse conditions dataset with correspondences for semantic driving scene understanding. In IEEE Int. Conf. Comput. Vis. (ICCV), pages 10745-10755, Montreal, QC, Canada, Oct. 2021. Inst. Electr. Electron. Eng. (IEEE).
Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. CoRR, abs/1511. 05952, 2015.
Jürgen Schmidhuber. Deep learning in neural networks: An overview. Neural Networks, 61: 85-117, Jan. 2015.
James Smith, Yen-Chang Hsu, Jonathan Balloch, Yilin Shen, Hongxia Jin, and Zsolt Kira. Always be dreaming: A new approach for data-free class-incremental learning. In IEEE Int. Conf. Comput. Vis. (ICCV), pages 9354-9364, Montreal, QC, Canada, Oct. 2021. Inst. Electr. Electron. Eng. (IEEE).
Baochen Sun and Kate Saenko. Deep CORAL: Correlation alignment for deep domain adaptation. In Eur. Conf. Comput. Vis. (ECCV), volume 9915 of Lect. Notes Comput. Sci., pages 443-450. Springer Int. Publ., 2016.
Tao Sun, Mattia Segu, Janis Postels, Yuxuan Wang, Luc Van Gool, Bernt Schiele, Federico Tombari, and Fisher Yu. SHIFT: A synthetic driving dataset for continuous multi-task domain adaptation. In IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), pages 21339-21350, New Orleans, LA, USA, Jun. 2022. Inst. Electr. Electron. Eng. (IEEE).
Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. CoRR, abs/2006. 10726, 2020.
Qin Wang, Olga Fink, Luc Van Gool, and Dengxin Dai. Continual test-time domain adaptation. In IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), pages 7191-7201, New Orleans, LA, USA, Jun. 2022. Inst. Electr. Electron. Eng. (IEEE).
Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M. Alvarez, and Ping Luo. SegFormer: Simple and efficient design for semantic segmentation with transformers. In Adv. Neural Inf. Process. Syst. (NeurIPS), volume 34, pages 12077-12090, 2021.
Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, and Trevor Darrell. BDD100K: A diverse driving dataset for heterogeneous multitask learning. In IEEE Int. Conf. Comput. Vis. Pattern Recognit. (CVPR), pages 2633-2642, Seattle, WA, USA, Jun. 2020. Inst. Electr. Electron. Eng. (IEEE).
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In Int. Conf. Mach. Learn. (ICML), 2017.
Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), pages 6230-6239, Honolulu, HI, USA, Jul. 2017. Inst. Electr. Electron. Eng. (IEEE).
Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip H. S. Torr, and Li Zhang. Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers. In IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), pages 6877-6886, Nashville, TN, USA, Jun. 2021. Inst. Electr. Electron. Eng. (IEEE).
Similar publications
Sorry the service is unavailable at the moment. Please try again later.
This website uses cookies to improve user experience. Read more
Save & Close
Accept all
Decline all
Show detailsHide details
Cookie declaration
About cookies
Strictly necessary
Performance
Strictly necessary cookies allow core website functionality such as user login and account management. The website cannot be used properly without strictly necessary cookies.
This cookie is used by Cookie-Script.com service to remember visitor cookie consent preferences. It is necessary for Cookie-Script.com cookie banner to work properly.
Performance cookies are used to see how visitors use the website, eg. analytics cookies. Those cookies cannot be used to directly identify a certain visitor.
Used to store the attribution information, the referrer initially used to visit the website
Cookies are small text files that are placed on your computer by websites that you visit. Websites use cookies to help users navigate efficiently and perform certain functions. Cookies that are required for the website to operate properly are allowed to be set without your permission. All other cookies need to be approved before they can be set in the browser.
You can change your consent to cookie usage at any time on our Privacy Policy page.