State-of-the-art stochastic data assimilation methods for high-dimensional non-Gaussian problems

[en] This paper compares several commonly used state-of-the-art ensemble-based data assimilation methods in a coherent mathematical notation. The study encompasses different methods that are applicable to high-dimensional geophysical systems, like ocean and atmosphere and provide an uncertainty estimate. Most variants of Ensemble Kalman Filters, Particle Filters and second-order exact methods are discussed, including Gaussian Mixture Filters, while methods that require an adjoint model or a tangent linear formulation of the model are excluded. The detailed description of all the methods in a mathematically coherent way provides both novices and experienced researchers with a unique overview and new insight in the workings and relative advantages of each method, theoretically and algorithmically, even leading to new filters. Furthermore, the practical implementation details of all ensemble and particle filter methods are discussed to show similarities and differences in the filters aiding the users in what to use when. Finally, pseudo-codes are provided for all of the methods presented in this paper.

Research center :

GHER

Disciplines :

Earth sciences & physical geography

Author, co-author :

Vetra-Carvalho, Sanita

van Leeuwen, Peter Jan

Nerger, Lars

Barth, Alexander ; Université de Liège - ULiège > Département d'astrophys., géophysique et océanographie (AGO) > GeoHydrodynamics and Environment Research (GHER)

Altaf, M. Umer

Brasseur, Pierre

Kirchgessner, Paul

Beckers, Jean-Marie ; Université de Liège - ULiège > Département d'astrophys., géophysique et océanographie (AGO) > GeoHydrodynamics and Environment Research (GHER)

Language :

English

Title :

State-of-the-art stochastic data assimilation methods for high-dimensional non-Gaussian problems

Publication date :

21 March 2018

Journal title :

Tellus. Series A

ISSN :

0280-6495

eISSN :

1600-0870

Publisher :

Taylor & Francis, United Kingdom

Volume :

Peer reviewed :

Peer Reviewed verified by ORBi

European Projects :

FP7 - 283580 - SANGOMA - Stochastic Assimilation for the Next Generation Ocean Model Applications

Name of the research project :

Sangoma

Funders :

UE - Union Européenne [BE]
CE - Commission Européenne [BE]

Available on ORBi :

since 29 January 2019

Statistics

Number of views

58 (3 by ULiège)

Number of downloads

103 (2 by ULiège)

More statistics

Scopus citations^®

Scopus citations^®
without self-citations

OpenCitations

Bibliography

Ades, M., and van Leeuwen, P. J., 2013. An exploration of the equivalent weights particle filter. Q. J. R. Meteorol. Soc. 139, 820–840.
Anderson, J., 2003. A local least squares framework for ensemble filtering. Mon. Wea. Rev. 131, 634–642.
Anderson, J. L., 2001. An ensemble adjustment Kalman filter for data assimilation. Mon. Wea. Rev. 129, 2884–2903.
Anderson, J. L., 2007. Exploring the need for localization in ensemble data assimilation using a hierarchical ensemble filter. Physica D 230, 99–111.
Anderson, J. L., 2009. Spatially and temporally varying adaptive covariance inflation for ensemble filters. Tellus 61A, 72–83.
Anderson, J. L., 2010. A non-Gaussian ensemble filter update for data assimilation. Mon. Wea. Rev. 138(11), 4186–4198.
Anderson, J. L., 2012. Localization and sampling error correction in ensemble Kalman filter data assimilation. Mon. Wea. Rev. 140, 2359–2371.
Anderson, J. L., and Anderson, S. L., 1999. A Monte Carlo implementation of the non-linear filtering problem to produce ensemble assimilations and forecasts. Mon. Wea. Rev. 126, 2741–2758.
Bannister, R. N., 2017. A review of operational methods of variational and ensemble-variational data assimilation. Q. J. R. Meteorol. Soc. 143, 607–633.
Bengtsson, T., Snyder, C., and Nychka, D., 2003. Toward a nonlinear ensemble filter for high-dimensional systems. J. Geophys. Res. 108, 8775–8785.
Bengtsson, T., Bickel, P., and Li, B., 2008. Curse-of-dimensionality revisited: collapse of the particle filter in very large scale systems. IMS Collections: Prob. Stat. Essays Honor David A. Freedman 2, 316–334.
Bishop, C. H., and Hodyss, D., 2009. Ensemble covariances adaptively localized with ECO-RAP. Part 2: a strategy for the atmosphere. Tellus 61A, 97–111.
Bishop, C. H., Etherton, B. J., and Majumdar, S. J., 2001. Adaptive sampling with the ensemble transform Kalman filter. Part I: theoretical aspects. Mon. Wea. Rev. 129, 420–436.
Blockley, E. W., Martin, M. J., McLaren, A. J., Ryan, A. G., Waters, A., and co-authors. 2014. Recent development of the Met Office operational ocean forecasting system: an overview and assessment of the new global FOAM forecasts. Geosci. Mod. Dev. 7, 2613–2638.
Bocquet, M., 2011. Ensemble Kalman filtering without the intrinsic need for inflation. Nonl. Proc. Geophy. 18, 735–750.
Bocquet, M., Pires, C. A., and Wu, L., 2010. Beyond Gaussian statistical modelling in geophysical data assimilation. Mon. Wea. Rev. 138, 2997–3023.
Bolić, M., Djurić, P. M., and Hong, S., 2003. New resampling algorithms for particle filters. Acoustics, Speech, and Signal Processing, 2003. Proceedings. (ICASSP ’03). 2003 IEEE International Conference, Vol. 2, IEEE, pp. II--589–592.
Brankart, J.-M., Testut, C.-E., Brasseur, P., and Verron, J., 2003. Implementation of a multivariate data assimilation scheme for isopycnic coordinate ocean models: application to a 1993–1996 hindcast of the North Atlantic ocean circulation. J. Geophys. Res. 108(C3), 3074.
Browne, P. A., and Wilson, S., 2015. A simple method for integrating a complex model into an ensemble data assimilation system using MPI. Env. Modell. Software 68, 122–128.
Brusdal, K., Brankart, J. M., Halberstadt, G., Evensen, G., Brasseur, P., and co-authors. 2003. A demonstration of ensemble based assimilation methods with a layered OGCM from the perspective of operational ocean forecasting systems. J. Mar. Syst. 40–41, 253–289.
Buehner, M., and Charron, M., 2007. Spectral and spatial localization of background-error correlations for data assimilation. Q. J. R. Meteorol. Soc. 133, 615–630.
Burgers, G., van Leeuwen, P. J., and Evensen, G., 1998. Analysis scheme in the ensemble Kalman filter. Mon. Wea. Rev. 126(6), 1719–1724.
Campbell, W. F., Bishop, C. H., and Hodyss, D., 2010. Vertical covariance localization for satellite radiances in ensemble Kalman filters. Mon. Wea. Rev. 138, 282–290.
Chorin, A. J., and Tu, X., 2009. Implicit sampling for particle filters. PNAS 106, 17249–17254.
Chorin, A. J., Morzfeld, M., and Tu, X., 2010. Interpolation and iteration for nonlinear filters. Commun. Appl. Math. Comput. Sci. 5, 221–240.
Chustagulprom, N., Reich, S., and Reinhardt, M., 2016. A hybrid ensemble transform filter for nonlinear and spatially extended dynamical systems. SIAM/ASA J. Uncert. Quant. 4, 552–591.
Cohn, S. E., Da Silva, A., Guo, J., Sienkiewicz, M., and Lamich, D., 1998. Assessing the effects of data selection with the DAO physical-space statistical analysis system. Mon. Wea. Rev. 126, 2913–2926.
Desroziers, G., Berre, L., Chapnik, B., and Poli, P., 2005. Diagnosis of observation, background and analysis-error statistics in observation space. Q. J. R. Meteorol. Soc. 131, 3385–3396.
Doucet, A., Godsill, S., and Andrieu, C., 2000. On sequential Monte Carlo sampling methods for Bayesian filtering. Stat. Comput. 10, 197–208.
Doucet, A., de Freitas, N., Gordon, N., 2001. Sequential Monte-Carlo Methods in Practice. Springer-Verlag, New York.
Evensen, G., 1993. Open boundary conditions for the extended Kalman filter with a quasi-geostrophic ocean model. J. Geophys. Res. 98(C9), 16529–16546.
Evensen, G., 1994. Sequential data assimilation with a nonlinear quasi-geostrophic model using Monte-Carlo methods to forecast error statistics. J. Geophys. Res. 99, 10143–10162.
Evensen, G., 2003. The ensemble Kalman filter: theoretical formulation and practical implementation. Ocean Dyn. 53, 343–367.
Flowerdew, J., 2015. Towards a theory of optimal localisation. Tellus A 67, 25257.
Frei, M., and Künsch, H. R., 2013. Bridging the ensemble Kalman and particle filters. Biometrica 100, 781–800.
Gaspari, G., and Cohn, S., 1999. Construction of correlation functions in two and three dimensions. Q. J. R. Meteorol. Soc. 125, 723–757.
Golub, G. H., and Van Loan, C. F., 1996. Matrix computations, 3rd ed. The Johns Hopkins University Press, Baltimore and London.
Gordon, N. J., Salmond, D. J., and Smith, A. F. M., 1993. Novel approach to nonlinear/non-Gaussian Bayesian state estimation. IEEE Proc. 140, 107–113.
Greybush, S. J., Kalnay, E., Miyoshi, T., Ide, K., and Hunt, B. R., 2011. Balance and ensemble Kalman filter localization techniques. Mon. Wea. Rev. 139, 511–522.
Gropp, W., Lusk, E., and Skjellum, A., 1994. Using MPI: Portable Parallel Programming with the Message-Passing Interface The MIT Press, Cambridge, Massachusetts.
Hamill, T. M., 2001. Interpretation of rank histograms for verifying ensemble forecasts. Mon. Wea. Rev. 129, 550–560.
Hamill, T. M., 2006. Ensemble-based atmospheric data assimilation. In: Predictability of Weather and Climate (eds. T., Palmer and R., Hagedorn). Cambridge University Press, New York, pp. 124–156. chapter 6.
Hamill, T. M., Whitaker, J. S., and Snyder, C., 2001. Distance-dependent filtering of background error covariance estimates in an ensemble Kalman filter. Mon. Wea. Rev. 129, 2776–1790.
Hoteit, I., Pham, D.-T., Triantafyllou, G., and Korres, G., 2008. A new approximate solution of the optimal nonlinear filter for data assimilation in meteorology and oceanography. Mon. Wea. Rev. 136, 317–334.
Houtekamer, P. L., and Mitchell, H. L., 1998. Data assimilation using an ensemble Kalman filter technique. Mon. Wea. Rev. 126, 796–811.
Houtekamer, P. L., and Mitchell, H. L., 2001. A sequential ensemble Kalman filter for atmospheric data assimilation. Mon. Wea. Rev. 129, 123–137.
Houtekamer, P. L., and Zhang, F., 2016. Review of the ensemble Kalman filter for atmospheric data assimilation. Mon. Wea. Rev. 144, 4489–4532.
Hunt, B. R., Kostelich, E. J., and Szunyogh, I., 2007. Efficient data assimilation for spatiotemporal chaos: a local ensemble transform Kalman filter. Physica D 230, 112–126.
Ide, K., Courtier, P., Ghil, M., and Lorenc, A. C., 1997. Unified notation for data assimilation: pperational, l sequential and variational. J. Meteor. Soc. Jpn. 75, 181–189.
Janjić, T., Nerger, L., Albertella, A., Schroeter, J., and Skachko, S., 2011. On domain localization in ensemble-based Kalman filter algorithms. Mon. Wea. Rev. 139, 2046–2060.
Jazwinski, A. H., 1970. Stochastic Processes and Filtering Theory Academic Press, New York.
Kalman, R. E., 1960. A new approach to linear filtering and prediction problems. Trans. AMSE --J. Basic Eng. 82(D), 35–45.
Kepert, J. D., 2006. Localisation, balance and choice of analysis variable in an ensemble Kalman filter. Q. J. R. Meteorol. Soc. 135(642), 1157–1176.
Kepert, J. D., 2009. Covariance localisation and balance in an ensemble Kalman filter. Q. J. R. Meteorol. Soc. 135, 1157–1176.
Keppenne, C. L., and Rienecker, M. M., 2002. Initial testing of a massively parallel ensemble Kalman filter with the Poseidon isopycnal ocean circulation model. Mon. Wea. Rev. 130, 2951–2965.
Keppenne, C. L., and Rienecker, M. M., 2003. Assimilation of temperature into an isopycnal ocean general circulation model using a parallel ensemble Kalman filter. J. Mar. Syst. 40–41, 363–380.
Kirchgessner, P., Nerger, L., and Bunse-Gerstner, A., 2014. On the choice of an optimal localization radius in ensemble Kalman filter methods. Mon. Wea. Rev. 142, 2165–2175.
Law, K. J. H., and Stuart, A. M., 2012. Evaluating data assimilation algorithms. Mon. Wea. Rev. 140, 3757–3782.
Lawson, W. G., and Hansen, J. A., 2004. Implications of stochastic and deterministic filters as ensemble-based data assimilation methods in varying regimes of error growth. Mon. Wea. Rev. 132, 1966–1989.
Lei, J., and Bickel, P., 2011. A moment matching ensemble filter for nonlinear non-Gaussian data assimilation. Mon. Wea. Rev. 139, 3964–3973.
Lei, L., and Anderson, J., 2014. Empirical localization of observations for serial ensemble Kalman filter data assimilation in an atmospheric general circulation model. Mon. Wea. Rev. 142, 1835–1851.
Lermusiaux, P. F. J., 2007. Adaptive modelling, adaptive data assimilation and adaptive sampling. Physica D 230, 172–196.
Lermusiaux, P. F. J., and Robinson, A. R., 1999. Data assimilation via error subspaces statistical estimation, part I: theory and schemes. Mon. Wea. Rev. 127, 1385–1407.
Lermusiaux, P. F. J., Robinson, A. R., Haley, P. J., and Leslie, W. G., 2002. Advanced interdisciplinary data assimilation: filtering and smoothing via error subspace statistical estimation. Proceedings of “The OCEANS 2002 MTS/IEEE conference, Holland”, Vol. 230, pp. 795–802.
Livings, D., 2005. Aspects of the ensemble Kalman filter [Master’s thesis] Department of Mathematics, University of Reading, UK.
Livings, D., Dance, S. L., and Nichols, N. K., 2008. Unbiased ensemble square root filters. Physica D 237, 1021–1028.
Lorenc, A. C., 2003. The potential of the ensemble Kalman filter for NWP - a comparison with 4D-Var. Q. J. R. Meteorol. Soc. 129, 3183–3203.
Metref, S., Cosme, E., Snyder, C., and Brasseur, P., 2014a. A non-Gaussian analysis scheme using rank histograms for ensemble data assimilation. Nonlin. Processes Geophys. 21, 869–885.
Metref, S., Cosme, E., Snyder, C., and Brasseur, P., 2014b. A non-Gaussian analysis scheme using rank histogram for ensemble data assimilation. Nonlinear Proc. Geophys. 21, 869–885.
Minamide, M., and Zhang, F., 2017. Adaptive observation error inflation for assimilating all-sky satellite radiance. Mon. Wea. Rev. 145, 1063–1081.
Miyoshi, T., and Yamane, S., 2007. Local ensemble transform Kalman filter with an AGCM at a T159/L48 resolution. Mon. Wea. Rev. 135, 3841–3861.
Morzfeld, M., Tu, X., Atkins, E., and Chorin, A. J., 2012. A random map implementation of implicit filters. J. Comput. Phys. 231, 2049–2066.
Morzfeld, M., Hodyss, D., and Snyder, C., 2017. What the collapse of the ensemble Kalman filter tells us about particle filters. Tellus A 69, 1–14.
Nakano, S., Ueno, G., and Higuchi, T., 2007. Merging particle filter for sequential data assimilation. Nonlinear Process. Geophys. 14, 395–408.
Nerger, L., 2004. Parallel Filter Algorithms for Data Assimilation in Oceanography Number 487 in Reports on Polar and Marine Research, Alfred Wegener Institute for Polar and Marine Research, Bremerhaven, Germany, [PhD Thesis]. University of Bremen, Germany.
Nerger, L., 2015. On serial observation processing on localized ensemble Kalman filters. Mon. Wea. Rev. 143, 1554–1567.
Nerger, L., and Hiller, W., 2013. Software for ensemble-based data assimilation systems - implementation strategies and scalability. Comput. Geosci. 55, 110–118.
Nerger, L., Hiller, W., and Schröter, J., 2005a. PDAF -- the parallel data assimilation framework: experiences with Kalman filtering. In: Use of High Performance Computing in Meteorology: Proceedings of the Eleventh ECMWF Workshop on the Use of High Performance Computing in Meteorology, Reading, UK, 25--29 October 2004 (eds. W., Zwieflhofer and G., Mozdzynski). World Scientific, Singapore, pp. 63–83.
Nerger, L., Hiller, W., and Schröter, J., 2005b. A comparison of error subspace Kalman filters. Tellus 57A, 715–735.
Nerger, L., Danilov, S., Hiller, W., and Schröter, J., 2006. Using sea level data to constrain a finite-element primitive-equation ocean model with a local SEIK filter. Ocean Dyn. 56, 634–649.
Nerger, L., Janjić, T., Schroeter, J., and Hiller, W., 2012a. A unification of ensemble square root filters. Mon. Wea. Rev. 140, 2335–2345.
Nerger, L., Janjić, T., Schröter, J., and Hiller, W., 2012b. A regulated localization scheme for ensemble-based Kalman filters. Q. J. Roy. Meteor. Soc. 138, 802–812.
Ott, E., Hunt, B. R., Szunyogh, I., Zimin, A. V., Kostelich, E. J., and co-authors. 2004. A local ensemble Kalman filter for atmospheric data assimilation. Tellus 56A, 415–428.
Patil, D. J., Hunt, B. R., Kalnay, E., Yorke, J. A., and Ott, E., 2001. Local low dimensionality of atmospheric dynamics. Phys. Rev. Lett. 86(26), 5878–5881.
Penny, S., and Miyoshi, T., 2016. A local particle filter for high-dimensional geophysical systems. Nonlinear Processes Geophys. 23, 391–405.
Perianez, A., Reich, H., and Potthast, R., 2014. Optimal localization for ensemble Kalman filter systems. J. Meteorol. Soc. Jpn. 92, 585–597.
Petrie, R. E., and Dance, S. L., 2010. Ensemble-based data assimilation and the localisation problem. Weather 65(3), 65–69.
Pham, D. T., 2001. Stochastic methods for sequential data assimilation in strongly nonlinear systems. Mon. Wea. Rev. 129, 1194–1207.
Pham, D. T., Verron, J., and Gourdeau, L., 1998a. Singular evolutive Kalman filters for data assimilation in oceanography. C. R. Acad. Sci. Ser. II 326(4), 255–260.
Pham, D. T., Verron, J., and Roubaud, M. C., 1998b. A singular evolutive extended Kalman filter for data assimilation in oceanography. J. Mar. Syst. 16, 323–340.
Poterjoy, J., 2016a. A localized particle filter for high-dimensional nonlinear systems. Mon. Wea. Rev. 144, 59–76.
Poterjoy, J., and Anderson, J. L., 2016b. Efficient sssimilation of simulated observations in a high-dimensional geophysical system using a localized particle filter. Mon. Wea. Rev. 144, 2007–2020.
Reich, S., 2013. A nonparametric ensemble transform method for Bayesian inference. SIAM Journal on Scientific Computing 4(35), 2013–2024.
Sakov, P., and Oke, P. R., 2008. Implications of the form of the ensemble transformation in the ensemble square root filters. Mon. Wea. Rev. 136, 1042–1053.
Sakov, P., Counillon, F., Bertino, L., Lisaeter, K. A., Oke, P. R., and co-authors. 2012. TOPAZ4: an ocean-sea ice data assimilation system for the North Atlantic and Arctic. Ocean Sci. 8, 633–656.
Silverman, B. W., 1986. Density estimation for statistics and data analysis Chapman and Hall, New Work.
Snyder, C., Bengtsson, T., Bickel, P., and Anderson, J., 2008. Obstacles to high-dimensional particle filtering. Mon. Wea. Rev. 136, 4629–4640.
Snyder, C., Bengtsson, T., and Morzfeld, M., 2015. Performance bounds for particle filters using the optimal proposal. Mon. Wea. Rev. 143(11), 4750–4761.
Stordal, A. S., Karlsen, H. A., Nævdal, G., Skaug, H. J., and Valles, B., 2011. Bridging the ensemble Kalman filter and particle filters: the adaptive Gaussian mixture filter. Comput. Geosci. 15, 293–305.
Testut, C.-E., Brasseur, P., Brankart, J.-M., and Verron, J., 2003. Assimilation of sea-surface temperature and altimetric observationsduring 1992–1993 inot an eddy permitting primitive equation model of the North Atlantic ocean. J. Mar. Sys. 40–41, 291–316.
Tippett, M. K., Anderson, J. L., Bishop, C. H., Hamill, T. M., and Whitaker, J. S., 2003. Ensemble square root filters. Mon. Wea. Rev. 131, 1485–1490.
Tödter, J., and Ahrens, B., 2015. A Second-Order Exact Ensemble Square Root Filter for Nonlinear Data Assimilation. Mon. Wea. Rev. 143(4), 1347–1367.
Tödter, J., Kirchgessner, P., Nerger, L., and Ahrens, B., 2016. Assessment of a nonlinear ensemble transform filter for high-dimensional data assimilation. Mon. Wea. Rev. 144, 409–427.
Tong, X. T., Majda, A. J., and Kelly, D., 2016. Nonlinear stability and ergodicity of ensemble based Kalman filters. Nonlinearity 29, 657–691.
van Leeuwen, P. J., 2003a. A variance-minimizing filter for nonlinear dynamics. Mon. Wea. Rev. 131, 2071–2084.
van Leeuwen, P. J., 2003b. Nonlinear ensemble data assimilation for the ocean. Recent Developments in data assimilation for atmosphere and ocean, ECMWF Seminar 8--12 September 2003, Reading, United Kingdom, pp. 265–286.
van Leeuwen, P. J., 2009. Particle filtering in geophysical systems. Mon. Wea. Rev. 137, 4089–4114.
van Leeuwen, P. J., 2010. Nonlinear data assimilation in Geosciences: an extremely efficient particle filter. Q. J. R. Meteorol. Soc. 136, 1991–1999.
van Leeuwen, P. J., 2011. Efficient nonlinear data-assimilation in geophysical fluid dynamics. Comput. Fluids 46, 52–58.
van Leeuwen, P. J., Cheng, Y., and Reich, S., 2015. Nonlinear data assimilation. Frontiers in Applied Dynamical Systems: Reviews and Tutorials 2 Springer.
van Leeuwen, P. J., and Evensen, G., 1996. Data assimilation and inverse methods in terms of a probabilistic formulation. Mon. Wea. Rev. 124, 2898–2913.
Verlaan, M., and Heemink, A. W., 2001. Nonlinearity in data assimilation applications: a practical method for analysis. Mon. Wea. Rev. 129(6), 1578–1589.
Vossepoel, F. C., and Van Leeuwen, P. J., 2006. Parameter estimation using a particle method: inferring mixing coefficients from sea-level observations. Mon. Wea. Rev. 135, 1006–1020.
Wang, Y., Jung, Y., Supine, T. A., and Xue, M., 2013. A hybrid MPI-OpenMP parallel algorithm and performance analysis for an ensemble square root filter designed for multiscale observations. J. Atm. and Oce. Tech. 30, 1382–1397.
Whitaker, J. S., and Hamill, T. M., 2002. Ensemble data assimilation without perturbed observations. Mon. Wea. Rev. 130, 1913–1924.
Wikle, C. K., and Berliner, L. M., 2006. A Bayesian tutorial for data assimilation. Physica D, 230(1):1–16.
Xiong, X., Navon, I. M., and Uzunoglu, B., 2006. A note on the particle filter with posterior Gaussian resampling. Tellus A 58(4), 456–460.
Zhen, Y., and Zhang, F., 2014. A probabilistic approach to adaptive covariance localization for serial ensemble square root filters. Mon. Wea. Rev. 142, 4499–4518.
Zhu, M., van Leeuwen, P. J., and Amezcua, J., 2016. Implicit equal-weights particle filter. Q. J. R. Meteorol. Soc. page personal communication.
Zupanski, M., 2005. Maximum likelihood ensemble filter: Theoretical aspects. Mon. Wea. Rev. 133, 1710–1726.
Zupanski, M., Michael, N. I., and Zupanski, D., 2008. The maximum likelihood ensemble filter as a non-differentiable minimization algorithm. Q. J. R. Meteorol. Soc. 134, 1039–1050.