Deep generative model; Deep learning; Geological prior information; Geophysical inversion; Stochastic gradient descent; Variational autoencoder; Conceptual frameworks; Generative functions; Geological setting; Geophysical imaging; Gradient-based method; Markov Chain Monte-Carlo; Training parameters; Information Systems; Computers in Earth Sciences
Abstract :
[en] When solving inverse problems in geophysical imaging, deep generative models (DGMs) may be used to enforce the solution to display highly structured spatial patterns which are supported by independent information (e.g. the geological setting) of the subsurface. In such case, inversion may be formulated in a latent space where a low-dimensional parameterization of the patterns is defined and where Markov chain Monte Carlo or gradient-based methods may be applied. However, the generative mapping between the latent and the original (pixel) representations is usually highly nonlinear which may cause some difficulties for inversion, especially for gradient-based methods. In this contribution we review the conceptual framework of inversion with DGMs and propose that this nonlinearity is caused mainly by changes in topology and curvature induced by the generative function. As a result, we identify a conflict between two goals: the accuracy of the generated patterns and the feasibility of gradient-based inversion. In addition, we show how some of the training parameters of a variational autoencoder, which is a particular instance of a DGM, may be chosen so that a tradeoff between these two goals is achieved and acceptable inversion results are obtained with a stochastic gradient-descent scheme. A series of test cases using synthetic models with channel patterns of different complexity and cross-borehole traveltime tomographic data involving both a linear and a nonlinear forward operator show that the proposed method provides useful results and performs better compared to previous approaches using DGMs with gradient-based inversion.
Disciplines :
Geological, petroleum & mining engineering
Author, co-author :
Lopez-Alvis, Jorge ; Urban and Environmental Engineering, Applied Geophysics, University of Liège, Belgium ; Department of Geology, Ghent University, Belgium
Laloy, Eric; Engineered and Geosystems Analysis, Institute for Environment, Health and Safety, Belgian Nuclear Research Center, Belgium
Nguyen, Frédéric ; Université de Liège - ULiège > Département ArGEnCo > Géophysique appliquée
Hermans, Thomas; Department of Geology, Ghent University, Belgium
Language :
English
Title :
Deep generative models in inversion: The impact of the generator's nonlinearity and development of a new approach based on a variational autoencoder
H2020 - 722028 - ENIGMA - European training Network for In situ imaGing of dynaMic processes in heterogeneous subsurfAce environments
Funders :
EU - European Union [BE]
Funding text :
This work has received funding from the European Union's Horizon 2020 research and innovation program under the Marie Sklodowska-Curie grant agreement number 722028 (ENIGMA ITN). We thank the anonymous reviewers and the editor for their valuable comments that greatly improved the manuscript.
Armstrong, M., Galli, A., Beucher, H., Loc'h, G., Renard, D., Doligez, B., Eschard, R., Geffroy, F., Plurigaussian Simulations in Geosciences. 2011, Springer Berlin Heidelberg, Berlin, Heidelberg URL http://link.springer.com/10.1007/978-3-642-19607-2.
Arvanitidis, G., Hansen, L.K., Hauberg, S., Latent Space Oddity: on the Curvature of Deep Generative Models. Jan. 2018 arXiv:1710.11379 [stat]ArXiv: 1710.11379. URL http://arxiv.org/abs/1710.11379.
Aster, R., Borchers, B., Thurber, C., Parameters Estimation and Inverse Problems. second ed., 2013, Academic press.
Bergmann, U., Jetchev, N., Vollgraf, R., Learning Texture Manifolds with the Periodic Spatial GAN. Sep. 2017 arXiv:1705.06566 [cs, stat]ArXiv: 1705.06566. URL http://arxiv.org/abs/1705.06566.
Bora, A., Jalal, A., Price, E., Dimakis, A.G., Compressed Sensing Using Generative Models. Mar. 2017 arXiv:1703.03208 [cs, math, stat]ArXiv: 1703.03208. URL http://arxiv.org/abs/1703.03208.
Caers, J., Hoffman, T., The probability perturbation method: a new look at bayesian inverse modeling. Math. Geol. 38:1 (Jan. 2006), 81–100 URL http://link.springer.com/10.1007/s11004-005-9005-9.
Canchumuni, S.W., Emerick, A.A., Pacheco, M.A.C., Towards a robust parameterization for conditioning facies models using deep variational autoencoders and ensemble smoother. Comput. Geosci. 128 (Jul. 2019), 87–102 URL https://linkinghub.elsevier.com/retrieve/pii/S0098300419300378.
Caterina, D., Hermans, T., Nguyen, F., Case studies of incorporation of prior information in electrical resistivity tomography: comparison of different approaches. Near Surf. Geophys. 12 (Aug. 2014), 451–465 URL http://nsg.eage.org/publication/publicationdetails/?publication=76904.
Chaudhari, P., Soatto, S., Stochastic Gradient Descent Performs Variational Inference, Converges to Limit Cycles for Deep Networks. Jan. 2018 arXiv:1710.11029 [cond-mat, stat]ArXiv: 1710.11029. URL http://arxiv.org/abs/1710.11029.
Chen, N., Klushyn, A., Kurle, R., Jiang, X., Bayer, J., van der Smagt, P., Metrics for Deep Generative Models. Feb. 2018 arXiv:1711.01204 [cs, stat]ArXiv: 1711.01204. URL http://arxiv.org/abs/1711.01204.
Domingos, P., A few useful things to know about machine learning. Commun. ACM, 55(10), Oct. 2012, 78 URL http://dl.acm.org/citation.cfm?doid=2347736.2347755.
Falorsi, L., de Haan, P., Davidson, T.R., De Cao, N., Weiler, M., Forré, P., Cohen, T.S., Explorations in Homeomorphic Variational Auto-Encoding. Jul. 2018 arXiv:1807.04689 [cs, stat]ArXiv: 1807.04689. URL http://arxiv.org/abs/1807.04689.
Fefferman, C., Mitter, S., Narayanan, H., Testing the manifold hypothesis. J. Am. Math. Soc. 29:4 (Feb. 2016), 983–1049 URL http://www.ams.org/jams/2016-29-04/S0894-0347-2016-00852-4/.
Giroux, B., Larouche, B., Task-parallel implementation of 3D shortest path raytracing for geophysical applications. Comput. Geosci. 54 (Apr. 2013), 130–141 URL https://linkinghub.elsevier.com/retrieve/pii/S0098300412004128.
Goodfellow, I., Bengio, Y., Courville, A., Deep Learning. 2016, MIT Press.
Goodfellow, I.J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y., Generative Adversarial Networks. Jun. 2014 arXiv:1406.2661 [cs, stat]ArXiv: 1406.2661. URL http://arxiv.org/abs/1406.2661.
Hand, P., Voroninski, V., Global Guarantees for Enforcing Deep Generative Priors by Empirical Risk. Dec. 2018 arXiv:1705.07576 [cs, math]ArXiv: 1705.07576. URL http://arxiv.org/abs/1705.07576.
Hansen, T.M., Cordua, K.S., Mosegaard, K., Inverse problems with non-trivial priors: efficient solution through sequential Gibbs sampling. Comput. Geosci. 16:3 (Jun. 2012), 593–611 URL http://link.springer.com/10.1007/s10596-011-9271-1.
Hermans, T., Vandenbohede, A., Lebbe, L., Martin, R., Kemna, A., Beaujean, J., Nguyen, F., Imaging artificial salt water infiltration using electrical resistivity tomography constrained by geostatistical data. J. Hydrol. 438–439 (2012), 168–180.
Higgins, I., Matthey, L., Pal, A., Burgess, C., Glorot, X., Botvinick, M., Mohamed, S., Lerchner, A., Beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework, vol. 13, 2017.
Kim, J., Zhang, B.-T., Data Interpolations in Deep Generative Models under Non-simply-connected Manifold Topology. Jan. 2019 arXiv:1901.08553 [cs, stat]ArXiv: 1901.08553. URL http://arxiv.org/abs/1901.08553.
Kingma, D.P., Ba, J., Adam: A Method for Stochastic Optimization. Jan. 2017 arXiv:1412.6980 [cs]ArXiv: 1412.6980. URL http://arxiv.org/abs/1412.6980.
Kingma, D.P., Welling, M., Auto-Encoding Variational Bayes. May 2014 arXiv:1312.6114 [cs, stat]ArXiv: 1312.6114. URL http://arxiv.org/abs/1312.6114.
Kleinberg, R., Li, Y., Yuan, Y., An Alternative View: when Does SGD Escape Local Minima?. Aug. 2018 arXiv:1802.06175 [cs]ArXiv: 1802.06175. URL http://arxiv.org/abs/1802.06175.
Laloy, E., Hérault, R., Jacques, D., Linde, N., Training-image based geostatistical inversion using a spatial generative adversarial neural network. Water Resour. Res. 54:1 (Jan. 2018), 381–406, 10.1002/2017WR022148 URL.
Laloy, E., Hérault, R., Lee, J., Jacques, D., Linde, N., Inversion using a new low-dimensional representation of complex binary geological media based on a deep neural network. Adv. Water Resour. 110 (Dec. 2017), 387–405 URL https://linkinghub.elsevier.com/retrieve/pii/S0309170817306243.
Laloy, E., Linde, N., Ruffino, C., Hérault, R., Gasso, G., Jacques, D., Gradient-based deterministic inversion of geophysical data with generative adversarial networks: is it feasible?. Comput. Geosci., 133, Dec. 2019, 104333 URL https://linkinghub.elsevier.com/retrieve/pii/S009830041831207X.
Lange, K., Frydendall, J., Cordua, K.S., Hansen, T.M., Melnikova, Y., Mosegaard, K., A frequency matching method: solving inverse problems by use of geologically realistic prior information. Math. Geosci. 44:7 (Oct. 2012), 783–803 URL http://link.springer.com/10.1007/s11004-012-9417-2.
Linde, N., Renard, P., Mukerji, T., Caers, J., Geological realism in hydrogeological and geophysical inverse modeling: a review. Adv. Water Resour. 86 (Dec. 2015), 86–101.
Liu, N., Oliver, D.S., Ensemble Kalman filter for automatic history matching of geologic facies. J. Petrol. Sci. Eng. 47:3 (Jun. 2005), 147–161 URL https://www.sciencedirect.com/science/article/pii/S0920410505000550.
Luo, X., Stordal, A.S., Lorentzen, R.J., Nævdal, G., Iterative ensemble smoother as an approximate solution to a regularized minimum-average-cost problem: theory and applications. 05 SPE J. 20 (Oct. 2015), 962–982, 10.2118/176023-PA URL.
Mariethoz, G., Renard, P., Straubhaar, J., The Direct Sampling method to perform multiple-point geostatistical simulations: performing multiple-points simulations. Water Resour. Res., 46(11), Nov. 2010, 10.1029/2008WR007621 URL.
Metz, L., Poole, B., Pfau, D., Sohl-Dickstein, J., Unrolled Generative Adversarial Networks. May 2017 arXiv:1611.02163 [cs, stat]ArXiv: 1611.02163. URL http://arxiv.org/abs/1611.02163.
Mo, S., Zabaras, N., Shi, X., Wu, J., Feb, Integration of adversarial autoencoders with residual dense convolutional networks for estimation of non-Gaussian hydraulic conductivities. Water Resour. Res., 56(2), 2020 URL https://onlinelibrary.wiley.com/doi/abs/10.1029/2019WR026082.
Mosser, L., Dubrule, O., Blunt, M.J., Stochastic Seismic Waveform Inversion Using Generative Adversarial Networks as a Geological Prior. Jun. 2018 arXiv:1806.03720 [physics, stat]ArXiv: 1806.03720. URL http://arxiv.org/abs/1806.03720.
Naitzat, G., Zhitnikov, A., Lim, L.-H., Topology of Deep Neural Networks. Apr. 2020 arXiv:2004.06093 [cs, math, stat]ArXiv: 2004.06093. URL http://arxiv.org/abs/2004.06093.
Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., Lerer, A., Automatic Differentiation in PyTorch, vol. 4, 2017.
Rezaee, H., Marcotte, D., Calibration of categorical simulations by evolutionary gradual deformation method. Comput. Geosci. 22:2 (Apr. 2018), 587–605 http://link.springer.com/10.1007/s10596-017-9711-7.
Richardson, A., Generative Adversarial Networks for Model Order Reduction in Seismic Full-Waveform Inversion. Jun. 2018 arXiv:1806.00828 [physics]ArXiv: 1806.00828. URL http://arxiv.org/abs/1806.00828.
Rücker, C., Günther, T., Wagner, F.M., pyGIMLi: an open-source library for modelling and inversion in geophysics. Comput. Geosci. 109 (Dec. 2017), 106–123.
Salakhutdinov, R., Learning deep generative models. Annual Review of Statistics and Its Application 2:1 (Apr. 2015), 361–385 URL http://www.annualreviews.org/doi/10.1146/annurev-statistics-010814-020120.
Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., Chen, X., Improved Techniques for Training GANs. Jun. 2016 arXiv:1606.03498 [cs]ArXiv: 1606.03498. URL http://arxiv.org/abs/1606.03498.
Seo, J.K., Kim, K.C., Jargal, A., Lee, K., Harrach, B., A learning-based method for solving ill-posed nonlinear inverse problems: a simulation study of lung EIT. SIAM J. Imag. Sci. 12:3 (Jan. 2019), 1275–1295 URL https://epubs.siam.org/doi/10.1137/18M1222600.
Shao, H., Kumar, A., Fletcher, P.T., The Riemannian Geometry of Deep Generative Models. Nov. 2017 arXiv:1711.08014 [cs, stat]ArXiv: 1711.08014. URL http://arxiv.org/abs/1711.08014.
Smith, S.L., Le, Q.V., A Bayesian Perspective on Generalization and Stochastic Gradient Descent. Feb. 2018 arXiv:1710.06451 [cs, stat]ArXiv: 1710.06451. URL http://arxiv.org/abs/1710.06451.
Strebelle, S., Conditional simulation of complex geological structures using multiple-point statistics. Math. Geol. 34:1 (2002), 1–21 URL http://www.springerlink.com/index/8G2MEAGU5K0U07PK.pdf.
Zahner, T., Lochbühler, T., Mariethoz, G., Linde, N., Image synthesis with graph cuts: a fast model proposal mechanism in probabilistic inversion. Geophys. J. Int. 204:2 (Feb. 2016), 1179–1190 URL https://academic.oup.com/gji/article-lookup/doi/10.1093/gji/ggv517.
