[en] When used by autonomous vehicles for trajectory planning or obstacle avoidance, depth estimation methods need to be reliable. Therefore, estimating the quality of the depth outputs is critical. In this paper, we show how M4Depth, a state-of-the-art depth estimation method designed for unmanned aerial vehicle (UAV) applications, can be enhanced to perform joint depth and uncertainty estimation. For that, we present a solution to convert the uncertainty estimates related to parallax generated by M4Depth into uncertainty estimates related to depth, and show that it outperforms the standard probabilistic approach. Our experiments on various public datasets demonstrate that our method performs consistently, even in zero-shot transfer. Besides, our method offers a compelling value when compared to existing multi-view depth estimation methods as it performs similarly on a multi-view depth estimation benchmark despite being 2.5 times faster and causal, as opposed to other methods. The code of our method is publicly available at the following URL: https://github.com/michael-fonder/M4DepthU .
Research Center/Unit :
Montefiore Institute - Montefiore Institute of Electrical Engineering and Computer Science - ULiège Telim
Fonder, Michaël ; Université de Liège - ULiège > Montefiore Institute of Electrical Engineering and Computer Science
Van Droogenbroeck, Marc ; Université de Liège - ULiège > Département d'électricité, électronique et informatique (Institut Montefiore) > Télécommunications ; Université de Liège - ULiège > Montefiore Institute of Electrical Engineering and Computer Science
Language :
English
Title :
A technique to jointly estimate depth and depth uncertainty for unmanned aerial vehicles
Publication date :
June 2023
Event name :
International Conference on Systems, Signals and Image Processing
Event place :
Ohrid, North Macedonia
Event date :
June 27-29, 2023
Event number :
30
Audience :
International
Main work title :
International Conference on Systems, Signals and Image Processing (IWSSIP)
Publisher :
IEEE
Peer reviewed :
Peer reviewed
Name of the research project :
ARIAC
Funders :
SPW - Public Service of Wallonia
Funding number :
2010235
Funding text :
This work was partly supported by the Walloon Region (Service Public de Wallonie Recherche, Belgium) under grant n°2010235 (ARIAC by DigitalWallonia.ai)
The Montefiore Institute Dataset of Aerial Images and Records (Mid-Air), is a multi-purpose synthetic dataset for low altitude drone flights. It provides a large amount of synchronized data corresponding to flight records for multi-modal vision sensors and navigation sensors mounted on board of a flying quadcopter.
Commentary :
The code of our method is publicly available at the following URL: https://github.com/michael-fonder/M4DepthU
scite shows how a scientific paper has been cited by providing the context of the citation, a classification describing whether it supports, mentions, or contrasts the cited claim, and a label indicating in which section the citation was made.
Bibliography
F. Mumuni, A. Mumuni, and C. K. Amuzuvi, "Deep learning of monocular depth, optical flow and ego-motion with geometric guidance for UAV navigation in dynamic environments, " Mach. Learn. Appl., vol. 10, pp. 2-15, Dec. 2022. [Online]. Available: https://doi. org/10. 1016/j. mlwa. 2022. 100416
X. Yang, H. Luo, Y. Wu, Y. Gao, C. Liao, and K.-T. Cheng, "Reactive obstacle avoidance of monocular quadrotors with online adapted depth prediction network, " Neurocomputing, vol. 325, pp. 142-158, Jan. 2019. [Online]. Available: https://doi. org/10. 1016/j. neucom. 2018. 10. 019
D. Wang, W. Li, X. Liu, N. Li, and C. Zhang, "UAV environmental perception and autonomous obstacle avoidance: A deep learning and depth camera combined solution, " Comput. Electron. Agric., vol. 175, pp. 1-11, Aug. 2020. [Online]. Available: https://doi. org/10. 1016/j. compag. 2020. 105523
M. Fonder, D. Ernst, and M. Van Droogenbroeck, "Parallax inference for robust temporal monocular depth estimation in unstructured environments, " Sensors, vol. 22, no. 23, pp. 1-22, Dec. 2022. [Online]. Available: https://doi. org/10. 3390/s22239374
M. Fonder and M. Van Droogenbroeck, "Mid-air: A multi-modal dataset for extremely low altitude drone flights, " in IEEE Int. Conf. Comput. Vis. Pattern Recognit. Work. (CVPRW), UAVision. Long Beach, CA, USA: Inst. Electr. Electron. Eng. (IEEE), Jun. 2019, pp. 553-562. [Online]. Available: https://doi. org/10. 1109/CVPRW. 2019. 00081
A. Geiger, P. Lenz, and R. Urtasun, "Are we ready for autonomous driving? The KITTI vision benchmark suite, " in IEEE Int. Conf. Comput. Vis. Pattern Recognit. (CVPR), Providence, RI, USA, Jun. 2012, pp. 3354-3361. [Online]. Available: https://doi. org/10. 1109/CVPR. 2012. 6248074
J. Gawlikowski, C. R. N. Tassi, M. Ali, J. Lee, M. Humt, J. Feng, A. Kruspe, R. Triebel, P. Jung, R. Roscher, M. Shahzad, W. Yang, R. Bamler, and X. X. Zhu, "A survey of uncertainty in deep neural networks, " CoRR, vol. abs/2107. 03342, 2021. [Online]. Available: https://doi. org/10. 48550/arXiv. 2107. 03342
A. Kendall and Y. Gal, "What uncertainties do we need in Bayesian deep learning for computer vision?" in Adv. Neural Inf. Process. Syst. (NeurIPS), Long Beach, CA, USA, Dec. 2017, pp. 5574-5584.
T. Ke, T. Do, K. Vuong, K. Sartipi, and S. I. Roumeliotis, "Deep multi-view depth estimation with predicted uncertainty, " in IEEE Int. Conf. Robot. Autom. (ICRA). Xian, China: Inst. Electr. Electron. Eng. (IEEE), May 2021, pp. 9235-9241. [Online]. Available: https://doi. org/10. 1109/ICRA48506. 2021. 9560873
E. Ilg, Ö. Çiçek, S. Galesso, A. Klein, O. Makansi, F. Hutter, and T. Brox, "Uncertainty estimates and multi-hypotheses networks for optical flow, " in Eur. Conf. Comput. Vis. (ECCV), ser. Lect. Notes Comput. Sci., vol. 11211. Springer Int. Publ., 2018, pp. 677-693. [Online]. Available: https://doi. org/10. 1007/978-3-030-01234-2_40
P. Schröppel, J. Bechtold, A. Amiranashvili, and T. Brox, "A benchmark and a baseline for robust multi-view depth estimation, " CoRR, vol. abs/2209. 06681, 2022. [Online]. Available: https://doi. org/10. 48550/arXiv. 2209. 06681
J. Zhang, S. Li, Z. Luo, T. Fang, and Y. Yao, "Vis-MVSNet: Visibility-aware multi-view stereo network, " Int. J. Comput. Vis., vol. 131, no. 1, pp. 199-214, Oct. 2022. [Online]. Available: https://doi. org/10. 1007/s11263-022-01697-3
M. Poggi, F. Aleotti, F. Tosi, and S. Mattoccia, "On the uncertainty of self-supervised monocular depth estimation, " in IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR). Seattle, WA, USA: Inst. Electr. Electron. Eng. (IEEE), Jun. 2020, pp. 3224-3234. [Online]. Available: https://doi. org/10. 1109/cvpr42600. 2020. 00329
W. Su, Q. Xu, and W. Tao, "Uncertainty guided multi-view stereo network for depth estimation, " IEEE Trans. Circuits Syst. Video Technol., vol. 32, no. 11, pp. 7796-7808, Nov. 2022. [Online]. Available: https://doi. org/10. 1109/TCSVT. 2022. 3183836
C. Liu, J. Gu, K. Kim, S. G. Narasimhan, and J. Kautz, "Neural RGB-D sensing: Depth and uncertainty from a video camera, " in IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR). Long Beach, CA, USA: Inst. Electr. Electron. Eng. (IEEE), Jun. 2019, pp. 10 978-10 987. [Online]. Available: https://doi. org/10. 1109/CVPR. 2019. 01124
W. Zhao, S. Liu, Y. Wei, H. Guo, and Y.-J. Liu, "A confidence-based iterative solver of depths and surface normals for deep multi-view stereo, " in IEEE Int. Conf. Comput. Vis. (ICCV). Montreal, QC, Canada: Inst. Electr. Electron. Eng. (IEEE), Oct. 2021, pp. 6148-6157. [Online]. Available: https://doi. org/10. 1109/iccv48922. 2021. 00611
M. Mehltretter and C. Heipke, "Aleatoric uncertainty estimation for dense stereo matching via CNN-based cost volume analysis, " ISPRS J. Photogramm. Remote Sens., vol. 171, pp. 63-75, Jan. 2021. [Online]. Available: https://doi. org/10. 1016/j. isprsjprs. 2020. 11. 003
C. Homeyer, O. Lange, and C. Schnörr, "Multi-view monocular depth and uncertainty prediction with deep SfM in dynamic environments, " in Int. J. Pattern Recognit. Artif. Intell., ser. Lect. Notes Comput. Sci., vol. 13363. Springer Int. Publ., 2022, pp. 373-385. [Online]. Available: https://doi. org/10. 1007/978-3-031-09037-0_31
M. Klodt and A. Vedaldi, "Supervising the new with the old: Learning SFM from SFM, " in Eur. Conf. Comput. Vis. (ECCV), ser. Lect. Notes Comput. Sci., vol. 11214. Springer Int. Publ., 2018, pp. 713-728. [Online]. Available: https://doi. org/10. 1007/978-3-030-01249-6_43
X. Yang, J. Chen, Y. Dang, H. Luo, Y. Tang, C. Liao, P. Chen, and K.-T. Cheng, "Fast depth prediction and obstacle avoidance on a monocular drone using probabilistic convolutional neural network, " IEEE Trans. Intell. Transp. Syst., vol. 22, no. 1, pp. 156-167, Jan. 2021. [Online]. Available: https://doi. org/10. 1109/TITS. 2019. 2955598
W. Wang, D. Zhu, X. Wang, Y. Hu, Y. Qiu, C. Wang, Y. Hu, A. Kapoor, and S. Scherer, "TartanAir: A dataset to push the limits of visual SLAM, " in IEEE/RSJ Int. Conf. Intell. Robot. Syst. (IROS). Las Vegas, NV, USA: Inst. Electr. Electron. Eng. (IEEE), Oct. 2020, pp. 4909-4916. [Online]. Available: https://doi. org/10. 1109/IROS45743. 2020. 9341801
D. Eigen, C. Puhrsch, and R. Fergus, "Depth map prediction from a single image using a multi-scale deep network, " in Adv. Neural Inf. Process. Syst. (NeurIPS), 2014, pp. 2366-2374.
O. Mac Aodha, A. Humayun, M. Pollefeys, and G. J. Brostow, "Learning a confidence measure for optical flow, " IEEE Trans. Pattern Anal. Mach. Intell., vol. 35, no. 5, pp. 1107-1120, May 2013. [Online]. Available: https://doi. org/10. 1109/TPAMI. 2012. 171
C. Kondermann, D. Kondermann, B. Jähne, and C. Garbe, "An adaptive confidence measure for optical flows based on linear subspace projections, " in Pattern Recognit., ser. Lect. Notes Comput. Sci., vol. 4713. Springer, 2007, pp. 132-141. [Online]. Available: https://doi. org/10. 1007/978-3-540-74936-3_14
A. S. Wannenwetsch, M. Keuper, and S. Roth, "ProbFlow: Joint optical flow and uncertainty estimation, " in IEEE Int. Conf. Comput. Vis. (ICCV). Venice, Italy: Inst. Electr. Electron. Eng. (IEEE), Oct. 2017, pp. 1182-1191. [Online]. Available: https://doi. org/10. 1109/ICCV. 2017. 133
J. Kybic and C. Nieuwenhuis, "Bootstrap optical flow confidence and uncertainty measure, " Comput. Vis. Image Underst., vol. 115, no. 10, pp. 1449-1462, Oct. 2011. [Online]. Available: https://doi. org/10. 1016/j. cviu. 2011. 06. 008
Y. Yao, Z. Luo, S. Li, T. Fang, and L. Quan, "MVSNet: Depth inference for unstructured multi-view stereo, " in Eur. Conf. Comput. Vis. (ECCV), ser. Lect. Notes Comput. Sci., vol. 11212. Springer, 2018, pp. 785-801. [Online]. Available: https://doi. org/10. 1007/978-3-030-01237-3_47
Z. Yu and S. Gao, "Fast-MVSNet: Sparse-to-dense multi-view stereo with learned propagation and gauss-newton refinement, " in IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR). Seattle, WA, USA: Inst. Electr. Electron. Eng. (IEEE), Jun. 2020, pp. 1946-1955. [Online]. Available: https://doi. org/10. 1109/cvpr42600. 2020. 00202
Similar publications
Sorry the service is unavailable at the moment. Please try again later.
This website uses cookies to improve user experience. Read more
Save & Close
Accept all
Decline all
Show detailsHide details
Cookie declaration
About cookies
Strictly necessary
Performance
Strictly necessary cookies allow core website functionality such as user login and account management. The website cannot be used properly without strictly necessary cookies.
This cookie is used by Cookie-Script.com service to remember visitor cookie consent preferences. It is necessary for Cookie-Script.com cookie banner to work properly.
Performance cookies are used to see how visitors use the website, eg. analytics cookies. Those cookies cannot be used to directly identify a certain visitor.
Used to store the attribution information, the referrer initially used to visit the website
Cookies are small text files that are placed on your computer by websites that you visit. Websites use cookies to help users navigate efficiently and perform certain functions. Cookies that are required for the website to operate properly are allowed to be set without your permission. All other cookies need to be approved before they can be set in the browser.
You can change your consent to cookie usage at any time on our Privacy Policy page.
