[en] Recent applications of multi-model methods have demonstrated their potential in quantifying conceptual model uncertainty in groundwater modeling applications. To date, however, little is known about the value of conditioning to constrain the ensemble of conceptualizations, to differentiate among retained alternative conceptualizations, and to reduce conceptual model uncertainty. We address these questions by conditioning multi-model simulations on measurements of hydraulic conductivity and observations of system-state variables and evaluating the e ffects on (i) the posterior multi-model statistics and (ii) the contribution of conceptual model uncertainty to the predictive uncertainty. Multi-model aggregation and conditioning is performed by combining the generalized likelihood uncertainty estimation (GLUE) method and Bayesian model averaging (BMA). As an illustrative example we employ a 3-dimensional hypothetical system under steady-state conditions, for which uncertainty about the conceptualization is expressed by an ensemble (M) of 7 models with varying complexity. Results show that conditioning on heads allowed for the exclusion of the two simplest models, but that their information content is limited to further diff erentiate among the retained conceptualizations. Conditioning on increasing numbers of conductivity measurements allowed for a further reffinement of the ensemble M and resulted in an increased precision and accuracy of the multi-model predictions. For some groundwater flow components not included as conditioning data, however, the gain in accuracy and precision was partially o ffset by strongly deviating predictions of a single conceptualization. Identifying the conceptualization producing the most deviating predictions may guide data collection campaigns aimed at acquiring data to further eliminate such conceptualizations. Including groundwater flow and river discharge observations
further allowed for a better diff erentiation among alternative conceptualizations and drastic reductions of the predictive variances. Results strongly advocate the use of observations less commonly available than groundwater heads to reduce conceptual model uncertainty in groundwater modeling.
Research Center/Unit :
Aquapôle - ULiège
Disciplines :
Geological, petroleum & mining engineering
Author, co-author :
Rojas, Rodrigo; Joint Research Centre (JRC), European Commission (EC) > Institute for Environment and Sustainability (IES) > Land management and natural hazards unit
Feyen, Luc; Joint Research Centre (JRC), European Commission (EC) > Institute for Environment and Sustainability (IES) > Land management and natural hazards unit
Batelaan, Okke; Katholieke Universiteit Leuven - KUL > Department of Earth and Environmental Sciences > Applied geology and mineralogy
Dassargues, Alain ; Université de Liège - ULiège > Département Argenco : Secteur GEO3 > Hydrogéologie & Géologie de l'environnement
Language :
English
Title :
On the value of conditioning data to reduce conceptual model uncertainty in groundwater modeling
Publication date :
August 2010
Journal title :
Water Resources Research
ISSN :
0043-1397
eISSN :
1944-7973
Publisher :
American Geophysical Union, Washington, United States - District of Columbia
Abrahart, R., and L. See (2002), Multi-model data fusion for river flow forecasting: An evaluation of six alternative methods based on two contrasting catchments, Hydrol. Earth Syst. Sci., 6(4), 655-670.
Ajami, N., Q. Duan, X. Gao, and S. Sorooshian (2005), Multimodel combination techniques for hydrologic forecasting: Application to distributed model intercomparison project results, J. Hydrometeorol., 7(4), 755-768, doi:10.1175/JHM519.1.
Ajami, N., Q. Duan, and S. Sorooshian (2007), An integrated hydrologic Bayesian multimodel combination framework: Confronting input, parameter, and model structural uncertainty in hydrologic prediction, Water Resour. Res., 43, W01403, doi:10.1029/2005WR004745. (Pubitemid 46364281)
Ajami, N., G. Hornberger, and D. Sunding (2008), Sustainable water resource management under hydrological uncertainty, Water Resour. Res., 44, W11406, doi:10.1029/2007WR006736.
Alcolea, A., J. Carrera, and A. Medina (2006), Pilot points method incorporating prior information for solving the groundwater flow inverse problem, Adv. Water Resour., 29(11), 1678-1689, doi:10.1016/ j.advwatres.2005.12.009.
Anderman, E., M. Holl, and E. Poeter (1996), Two-dimensional advective transport in ground-water flow parameter estimation, Ground Water, 34(6), 1001-1009, doi:10.1111/j.1745-6584.1996.tb02165.x.
Beven, K. (2006), A manifesto for the equifinality thesis, J. Hydrol., 320(1-2), 18-36, doi:10.1016/j.jhydrol.2005.07.007.
Beven, K., and A. Binley (1992), The future of distributed models: Model calibration and uncertainty prediction, Hydrol. Processes, 6(5), 279-283, doi:10.1002/hyp.3360060305.
Beven, K., and J. Freer (2001), Equifinality, data assimilation, and uncertainty estimation in mechanistic modelling of complex environmental systems using the GLUE methodology, J. Hydrol., 249(1-4), 11-29, doi:10.1016/S0022- 1694(01)00421-8. (Pubitemid 32664993)
Binley, A., and K. Beven (2003), Vadose zone flow model uncertainty as conditioned on geophysical data, Ground Water, 41(2), 119-127, doi:10.1111/j.1745-6584.2003.tb02576.x. (Pubitemid 36280731)
Box, G. (1980), Sampling and Bayes' inference in scientific modelling and robustness, J. R. Stat. Soc., Ser. A, 143(4), 383-430.
Bredehoeft, J. (2003), From models to performance assessment: The conceptualization problem, Ground Water, 41(5), 571-577, doi:10.1111/ j.1745-6584.2003.tb02395.x.
Bredehoeft, J. (2005), The conceptualization model problem-Surprise, Hydrogeol. J., 13(1), 37-46, doi:10.1007/s10040-004-0430-5. (Pubitemid 40517750)
Brooks, S., and A. Gelman (1998), General methods for monitoring convergence of iterative simulations, J. Comput. Graphical Stat., 7(4), 434-455.
Cacuci, D. (2003), Sensitivity and Uncertainty Analysis: Theory, 1st ed., 304 pp., Chapman and Hall, Boca Raton, Fla.
Capilla, J., and C. Llopis-Albert (2009), Gradual conditioning of non- Gaussian transmissivity fields to flow and mass transport data: 1. Theory, J. Hydrol., 371(1-4), 66-74, doi:10.1016/j.jhydrol.2009.03.015.
Carrera, J., and S. Neuman (1986a), Estimation of aquifer parameters under transient and steady state conditions: 1. Maximum likelihood method incorporating prior information, Water Resour. Res., 22(2), 199-210. (Pubitemid 16477839)
Carrera, J., and S. Neuman (1986b), Estimation of aquifer parameters under transient and steady state conditions: 2. Uniqueness, stability, and solution algorithms, Water Resour. Res., 22(2), 211-227. (Pubitemid 16477840)
Carrera, J., and S. Neuman (1986c), Estimation of aquifer parameters under transient and steady state conditions: 3. Application to synthetic and field data, Water Resour. Res., 22(2), 228-242. (Pubitemid 16477841)
Chib, S., and E. Greenberg (1995), Understanding the Metropolis-Hastings algorithm, Am. Stat., 49(4), 327-335.
Cooley, R., and R. Naff (1990), Regression modeling of ground-water flow, U.S. Geol. Surv. Tech. Water Resour. Invest., Book 3, Chap. B4, 232 pp.
Cooley, R., L. Konikow, and R. Naff (1986), Nonlinear regression groundwater flow modeling of a deep regional aquifer system, Water Resour. Res., 22(13), 1759-1778.
Cowles, M., and B. Carlin (1996), Markov chain Monte Carlo convergence diagnostics: A comparative review, J. Am. Stat. Assoc., 91(434), 883-904.
Dagan, G. (1985), Stochastic modelling of groundwater flow by unconditional and conditional probabilities: The inverse problem, Water Resour. Res., 21(1), 65-72.
Dagan, G. (1989), Flow and Transport in Porous Formations, 1st ed., 470 pp., Springer, Berlin.
Dagan, G., and S. Neuman (1997), Subsurface Flow and Transport: A Stochastic Approach, 1st ed., 256 pp., Cambridge Univ. Press, Cambridge, U. K.
Delhomme, J. (1979), Spatial variability and uncertainty in groundwater flow model parameters: A geostatistical approach, Water Resour. Res., 15(2), 269-280.
Deutsch, C., and A. Journel (1998), GSLIB: Geostatistical Software Library and User's Guide, 2nd ed., 384 pp., Oxford Univ. Press, New York.
Draper, D. (1995), Assessment and propagation of model uncertainty, J. R. Stat. Soc., Ser. B, 57(1), 45-97.
Duan, Q., N. Ajami, X. Gao, and S. Sorooshian (2007), Multi-model ensemble hydrologic prediction using Bayesian model averaging, Adv. Water Resour., 30(5), 1371-1386, doi:10.1016/j.advwatres.2006.11.014. (Pubitemid 46323445)
Ezzedine, S., and Y. Rubin (1996), A geostatistical approach to the conditional estimation of spatially distributed solute concentration and notes on the use of tracer data in the inverse problem, Water Resour. Res., 32(4), 853-861.
Feyen, L., K. Beven, F. De Smedt, and J. Freer (2001), Stochastic capture zone delineation within the Generalized Likelihood Uncertainty Estimation methodology: Conditioning on head observations, Water Resour. Res., 37(3), 625-638.
Feyen, L., P. Ribeiro, F. De Smedt, and P. Diggle (2002), Bayesian methodology to stochastic capture zone determination: Conditioning on transmissivity measurements, Water Resour. Res., 38(9), 1164, doi:10.1029/ 2001WR000950.
Feyen, L., J. Gómez-Hernández, P. Ribeiro, K. Beven, and F. De Smedt (2003), A Bayesian approach to stochastic capture zone delineation incorporating tracer arrival times, conductivity measurements, and hydraulic head observations, Water Resour. Res., 39(5), 1126, doi:10.1029/ 2002WR001544.
Fienen, M., T. Clemo, and P. Kitanidis (2008), An interactive Bayesian geostatistical inverse protocol for hydraulic tomography, Water Resour. Res., 44, W00B01, doi:10.1029/2007WR006730.
Fienen, M., R. Hunt, D. Krabbenhoft, and T. Clemo (2009), Obtaining parsimonious hydraulic conductivity fields using head and transport observations: A Bayesian geostatistical parameter estimation approach, Water Resour. Res., 45, W08405, doi:10.1029/2008WR007431.
Foglia, L., M. Hill, S. Mehl, and P. Burlando (2009), Sensitivity analysis, calibration, and testing of a distributed hydrological model using errorbased weighting and one objective function, Water Resour. Res., 45, W06427, doi:10.1029/2008WR007255.
Freeze, R. (1975), A stochastic-conceptual analysis of one-dimensional groundwater flow in non-uniform, homogeneus media, Water Resour. Res., 11(5), 725-741.
Gallagher, M., and J. Doherty (2007), Parameter estimation and uncertainty analysis for a watershed model, Environ. Modell. Software, 22(7), 1000-1020, doi:10.1016/j.envsoft.2006.06.007. (Pubitemid 46209835)
Gelhar, L. (1993), Stochastic Subsurface Hydrology, 1st ed., 390 pp., Prentice Hall, Englewood Cliffs, N. J.
Gelman, A., J. Carlin, H. Stern, and D. Rubin (2004), Bayesian Data Analysis, 2nd ed., 696 pp., Chapman and Hall, New York.
Georgakakos, K., D. Seo, H. Gupta, J. Schaake, and M. Butts (2004), Towards the characterization of streamflow simulation uncertainty through multimodel ensembles, J. Hydrol., 298(1-4), 222-241, doi:10.1016/j.jhydrol.2004. 03.037. (Pubitemid 39098775)
Geyer, C. (1992), Practical Markov chain Monte Carlo, Stat. Sci., 7(4), 473-483.
Ghosh, J., M. Delampady, and T. Samanta (2006), An Introduction to Bayesian Analysis-Theory and Methods, 1st ed., 352 pp., Springer, New York.
Gilks, W., S. Richardson, and D. Spiegelhalter (1995), Markov Chain Monte Carlo in Practice, 1st ed., 512 pp., Chapman and Hall, Boca Raton, Fla.
Gómez-Hernández, J., A. Sahuquillo, and J. Capilla (1997), Stochastic simulation of transmissivity fields conditional to both transmissivity and piezometric data: I. Theory, J. Hydrol. 203(1-4) 162-174 doi:10.1016/S0022-1694(97)00098-X. (Pubitemid 28118236)
Goovaerts, P. (1997), Geostatistics for Natural Resources Evaluation, 1st ed., 483 pp., Oxford Univ. Press, New York.
Gutjahr, A., and J. Wilson (1989), Co-kriging for stochastic flow models, Transp. Porous Media, 4(6), 585-598.
Gutjahr, A., B. Bullard, S. Hatch, and L. Hughson (1994), Joint conditional simulations and the spectral approach for flow modeling, Stochastic Hydrol. Hydraul., 8(1), 79-108, doi:10.1007/BF01581391.
Hanna, S., and T. Yeh (1998), Estimation of co-conditional moments of transmissivity, hydraulic head and velocity fields, Adv. Water Resour. Res., 22(1), 87-95.
Harbaugh, A., E. Banta, M. Hill, and M. McDonald (2000), MODFLOW- 2000 U.S. Geological Survey modular ground-water model-User guide to modularization concepts and the ground-water flow process, U.S. Geol. Surv. Open File Rep., 00-92.
Harrar, W., T. Sonnenberg, and H. Henriksen (2003), Capture zone, travel time, and solute transport predictions using inverse modelling and different geological models, Hydrogeol. J., 11(5), 536-548, doi:10.1007/ s10040-003-0276-282
Hassan, A., H. Bekhit, and J. Chapman (2008), Uncertainty assessment of a stochastic groundwater flow model using GLUE analysis, J. Hydrol., 362(1-2), 89-109, doi:10.1016/j.jhydrol.2008.08.017.
Hastings, W. (1970), Monte Carlo sampling methods using Markov chains and their applications, Biometrika, 57(1), 97-109, doi:10.1093/biomet/ 57.1.97.
Hendricks Franssen, H., J. Gómez-Hernández, and A. Sahuquillo (2003), Coupled inverse modelling of groundwater flow and mass transport and the worth of concentration data, J. Hydrol., 281(4), 281-295, doi:10.1016/S0022-1694(03)00191-4. (Pubitemid 37332124)
Hill, M. (2006), The practical use of simplicity in developing ground water models, Ground Water, 44(6), 775-781, doi:10.1111/j.1745-6584.2006. 00227.x.
Hill, M., and C. Tiedeman (2007), Effective Groundwater Model Calibration: With Analysis of Data, Sensitivities, Predictions and Uncertainty, 1st ed., 480 pp., John Wiley, Hoboken, N. J.
Hill, M., R. Cooley, and D. Pollock (1998), A controlled experiment in ground water flow model calibration, Ground Water, 36(3), 520-535, doi:10.1111/j.1745-6584.1998.tb02824.x.
Hoeksema, R., and P. Kitanidis (1984), An application of the geostatistical approach to the inverse problem in Two-dimensional groundwater modeling, Water Resour. Res., 20(7), 1003-1020.
Hoeting, J., D. Madigan, A. Raftery, and C. Volinsky (1999), Bayesian model averaging: A tutorial, Stat. Sci., 14(4), 382-401.
Højberg, A., and J. Refsgaard (2005), Model uncertainty-Parameter uncertainty versus conceptual models, Water Sci. Technol., 52(6), 177-186.
Hsu, K., H. Moradkhani, and S. Sorooshian (2009), A sequential Bayesian approach for hydrologic model selection and prediction, Water Resour. Res., 45, W00B12, doi:10.1029/2008WR006824.
Hu, L. (2000), Gradual deformation and iterative calibration of Gaussianrelated stochastic models, Math. Geol., 32(1), 87-108.
Hunt, R., J. Doherty, and M. Tonkin (2007), Are models too simple? Arguments for increased parameterization, Ground Water, 45(3), 254-262, doi:10.1111/j.1745-6584.2007.00316.x. (Pubitemid 46675856)
Ijiri, Y., H. Saegusa, A. Sawada, M. Ono, K. Watanabe, K. Karasaki, C. Doughty, M. Shimo, and K. Fumimura (2009), Evaluation of uncertainties originating from the different modeling approaches applied to analyze regional groundwater flow in the Tono area of Japan, J. Contam. Hydrol., 103(3-4), 168-181, doi:10.1016/j.jconhyd.2008.10.010.
Jensen, J. (2003), Parameter and uncertainty estimation in groundwater modelling, Ph.D. thesis, Dep. of Civ. Eng., Aalborg Univ., Aalborg, Denmark.
Kass, R., and A. Raftery (1995), Bayes factors, J. Am. Stat. Assoc., 90(430), 773-795.
Kavetski, D., and G. Kuczera (2007), Model smoothing strategies to remove microscale discontinuities and spurious secondary optima in objective functions in hydrological calibration, Water Resour. Res., 43, W03411, doi:10.1029/2006WR005195. (Pubitemid 46682781)
Kitanidis, P., and E. Vomvoris (1983), A geostatistical approach to the inverse problem in groundwater modeling (steady state) and onedimensional simulations, Water Resour. Res., 19(3), 677-690.
Krzysztofowicz, R. (1999), Bayesian theory of probabilistic forecasting via deterministic hydrologic model, Water Resour. Res., 35(9), 2739-2750.
LaVenue, A., B. RamaRao, G. de Marsily, and M. Marietta (1995), Pilot points methodology for automated calibration of an ensemble of conditionally simulated transmissivity fields: 2. Application, Water Resour. Res., 31(3), 495-516. (Pubitemid 26437836)
Leamer, E. (1978), Specification Searches: Ad hoc Inference With Nonexperimental Data, 1st ed., 384 pp., John Wiley, New York.
Li, X., and F. Tsai (2009), Bayesian model averaging for groundwater head prediction and uncertainty analysis using multimodel and multimethod, Water Resour. Res., 45, W09403, doi:10.1029/2008WR007488.
Llopis-Albert, C., and J. Capilla (2009), Gradual conditioning of non- Gaussian transmissivity fields to flow and mass transport data: 2. Demonstration on a synthetic aquifer, J. Hydrol. 371(1-4) 53-65 doi:10.1016/j. jhydrol.2009.03.014.
Makowski, D., D. Wallach, and M. Tremblay (2002), Using a Bayesian approach to parameter estimation; comparison of the GLUE and MCMC methods, Agronomie, 22(2), 191-203, doi:10.1051/agro:2002007.
Mantovan, P., and E. Todini (2006), Hydrological forecasting uncertainty assessment: Incoherence of the GLUE methodology, J. Hydrol., 330(1-2), 368-381, doi:10.1016/j.jhydrol.2006.04.046. (Pubitemid 44528546)
McLaughlin, D., and E. Wood (1988), A distributed parameter approach for evaluating the accuracy of groundwater model predictions: 2. Application to groundwater flow, Water Resour. Res., 24(7), 1048-1060. (Pubitemid 19297451)
Metropolis, N., A. Rosenbluth, M. Rosenbluth, A. Teller, and E. Teller (1953), Equation of state calculations by fast computing machines, J. Chem. Phys., 21(6), 1087-1092, doi:10.1063/1.1699114.
Meyer, P., M. Ye, S. Neuman, and K. Cantrell (2004), Combined estimation of hydrogeologic conceptual model and parameter uncertainty, Rep. NUREG/CR-6843 PNNL-14534, U.S. Nucl. Regul. Comm., Washington, D. C.
Meyer, P., M. Ye, M. Rockhold, S. Neuman, and K. Cantrell (2007), Combined estimation of hydrogeologic conceptual model parameter and scenario uncertainty with application to uranium transport at the Hanford site 300 area, Rep. NUREG/CR-6940 PNNL-16396, U.S. Nucl. Regul. Comm., Washington, D. C.
Moore, C., and J. Doherty (2005), Role of the calibration process in reducing model predictive error, Water Resour. Res., 41, W05020, doi:10.1029/2004WR003501. (Pubitemid 40856470)
Morse, B., G. Pohll, J. Huntington, and R. Rodriguez (2003), Stochastic capture zone analysis of an arsenic-contaminated well using the generalized likelihood uncertainty estimator (GLUE) methodology, Water Resour. Res., 39(6), 1151, doi:10.1029/2002WR001470.
Neuman, S. (2003), Maximum likelihood Bayesian averaging of uncertain model predictions, Stochastic Environ. Res. Risk Assess., 17(5), 291-305, doi:10.1007/s00477-003-0151-7. (Pubitemid 38013163)
Neuman, S., and P. Wierenga (2003), A comprehensive strategy of hydrogeologic modeling and uncertainty analysis for nuclear facilities and sites, Rep. NUREG/CR-6805, U.S. Nucl. Regul. Comm., Washington, D. C.
Oliver, D., L. Cunha, and A. Reynolds (1997), Markov chain Monte Carlo methods for conditioning a permeability field to pressure data, Math. Geol., 29(1), 61-91, doi:10.1007/BF02769620.
Pappenberger, F., and K. Beven (2006), Ignorance is bliss: Or seven reasons not to use uncertainty analysis, Water Resour. Res., 42, W05302, doi:10.1029/2005WR004820. (Pubitemid 43955900)
Pasquier, P., and D. Marcotte (2006), Steady- and transient-state inversion in hydrogeology by successive flux estimation, Adv. Water Resour., 29(12), 1934-1952, doi:10.1016/j.advwatres.2006.02.001. (Pubitemid 44567364)
Poeter, E., and D. Anderson (2005), Multimodel ranking and inference in ground water modelling, Ground Water, 43(4), 597-605, doi:10.1111/ j.1745-6584.2005.0061.x.
Poeter, E., and M. Hill (1997), Inverse models: A necessary step in groundwater modeling, Ground Water, 35(2), 250-260, doi:10.1111/j.1745-6584. 1997.tb00082.x.
Poeter, E., and S. McKenna (1995), Reducing uncertainty associated with ground-water flow and transport predictions, Ground Water, 33(6), 899-904, doi:10.1111/j.1745-6584.1995.tb00034.x.
Raftery, A., and Y. Zhang (2003), Discussion: Performance of Bayesian model averaging, J. Am. Stat. Assoc., 98(464), 931-938.
RamaRao, B., A. LaVenue, G. de Marsily, and M. Marietta (1995), Pilot point methodology for automated calibration of an ensemble of conditionally simulated transmissivity fields: 1. Theory and computational experiments, Water Resour. Res., 31(3), 475-493. (Pubitemid 26437835)
Refsgaard, J., J. Van der Sluijs, A. Højberg, and P. Vanrolleghem (2005), Harmoni-CA guidance uncertainty analysis, technical report, Eur. Comm., Brussels.
Refsgaard, J., J. Van der Sluijs, J. Brown, and P. Van der Keur (2006), A framework for dealing with uncertainty due to model structure error, Adv. Water Resour., 29(11), 1586-1597, doi:10.1016/j.advwatres.2005. 11.013.
Refsgaard, J., J. Van der Sluijs, A. Højberg, and P. Vanrolleghem (2007), Uncertainty in the environmental modelling process-A framework and guidance, Environ. Modell. Software, 22(11), 1543-1556, doi:10.1016/j. envsoft.2007.02.004.
Renard, P. (2007), Stochastic hydrogeology: What professionals really need?, Ground Water, 45(5), 531-541, doi:10.1111/j.1745-6584.2007. 00340.x.
Robert, C. (2007), The Bayesian Choice-From Decision-Theoretic Foundations to Computational Implementation, 2nd ed., 577 pp., Springer, New York.
Rojas, R., L. Feyen, and A. Dassargues (2008), Conceptual model uncertainty in groundwater modeling: Combining generalized likelihood uncertainty estimation and Bayesian model averaging, Water Resour. Res., 44, W12418, doi:10.1029/2008WR006908.
Rojas, R., L. Feyen, and A. Dassargues (2009), Sensitivity analysis of prior model probabilities and the value of prior knowledge in the assessment of conceptual model uncertainty in groundwater modelling, Hydrol. Processes, 23(8), 1131-1146, doi:10.1002/hyp.7231.
Rojas, R., O. Batelaan, L. Feyen, and A. Dassargues (2010), Assessment of conceptual model uncertainty for the regional aquifer Pampa del Tamarugal-north Chile, Hydrol. Earth Syst. Sci., 14(2), 171-192.
Rubin, Y. (2003), Applied Stochastic Hydrogeology, 1st ed., 416 pp., Oxford Univ. Press, New York.
Rubin, Y., and G. Dagan (1987), Stochastic identification of transmissivity and effective recharge in steady groundwater flow: 1. Theory, Water Resour. Res., 23(7), 1185-1192. (Pubitemid 17621181)
Sahuquillo, A., J. Capilla, J. Gómez-Hernández, and J. Andreu (1992), Conditional simulation of transmissivity fields honouring piezometric data, in Hydraulic Engineering Software IV. Fluid Flow Modelling, edi-ted by W. Blain and E. Cabrera, pp. 210-214, Kluwer Acad., Dordrecht, Netherlands.
Saltelli, A., S. Tarantola, F. Campolango, and M. Ratto (2008), Sensitivity Analysis in Practice: A Guide to Assessing Scientific Models, 1st ed., 232 pp., John Wiley, Chichester, U. K.
Seifert, D., T. Sonnenberg, P. Scharling, and K. Hinsby (2008), Use of alternative conceptual models to assess the impact of a buried valley on groundwater vulnerability, Hydrogeol. J., 16(4), 659-674, doi:10.1007/s10040- 007-0252-3.
Singh, A., S. Mishra, and G. Ruskauff (2010), Model averaging techniques for quantifying conceptual model uncertainty, Ground Water, doi:10.1111/j.1745- 6584.2009.00642.x, in press.
Sohn, M., M. Small, and M. Pantazidou (2000), Reducing uncertainty in site characterization using Bayes Monte Carlo methods, J. Environ. Eng., 126(10), 893-902, doi:10.1061/(ASCE)0733-9372(2000)126:10 (893).
Sorensen, D., and D. Gianola (2002), Likelihood, Bayesian, and MCMC Methods in Quantitative Genetics, vol.1, 1st ed., 740 pp., Springer, New York.
Tiedeman, C., D. Goode, and P. Hsieh (1997), Numerical simulation of ground-water flow through glacial deposits and crystalline bedrock in the Mirror Lake area, Grafton County, New Hampshire, U.S. Geol. Surv. Prof. Pap., 1572.
Tiedeman, C., D. Goode, and P. Hsieh (1998), Characterizing a ground water basin in a New England mountain and valley terrain, Ground Water, 36(4), 611-620, doi:10.1111/j.1745-6584.1998.tb02835.x.
Tiedeman, C., M. Hill, F. D'Agnese, and C. Faunt (2003), Methods for using groundwater model predictions to guide hydrogeologic data collection, with application to the Death Valley regional groundwater flow system, Water Resour. Res., 39(1), 1010, doi:10.1029/2001WR001255.
Tiedeman, C., D. Ely, M. Hill, and G. O'Brien (2004), A method for evaluating the importance of system state observations to model predictions, with application to the Death Valley regional groundwater flow system, Water Resour. Res., 40, W12411, doi:10.1029/2004WR003313.
Tierney, L. (1994), Markov chains for exploring posterior distributions, Ann. Stat., 22(4), 1701-1728.
Tonkin, M., C. Tiedeman, D. Ely, and M. Hill (2007), OPR-PPR, a computer program for assessing data importance to model predictions using linear statistics, U.S. Geol. Surv. Tech. Methods, 6-E2.
Troldborg, L., J. Refsgaard, K. Jensen, and P. Engesgaard (2007), The importance of alternative conceptual models for simulation of concentrations in a multi-aquifer system, Hydrogeol. J., 15(5), 843-860, doi:10.1007/s10040-007- 0192-y. (Pubitemid 47103590)
Tsai, F., and X. Li (2008), Inverse groundwater modeling for hydraulic conductivity estimation using Bayesian model averaging and variance window, Water Resour. Res., 44, W09434, doi:10.1029/2007WR006576.
Tsai, F., and X. Li (2010), Reply to comment by Ming Ye et al. on "Inverse groundwater modeling for hydraulic conductivity estimation using Bayesian model averaging and variance window", Water Resour. Res., 46, W02802, doi:10.1029/2009WR008591.
Van der Sluijs, J. (2005), Uncertainty as a monster in the science-policy interface: Four coping strategies, Water Sci. Technol., 52(6), 87-92.
Vrugt, J., and B. Robinson (2007), Treatment of uncertainty using ensemble methods: Comparison of sequential data assimilation and Bayesian model averaging, Water Resour. Res., 43,W01411, doi:10.1029/2005WR004838. (Pubitemid 46364289)
Vrugt, J., M. Clarck, C. Diks, Q. Duan, and B. Robinson (2006), Multiobjective calibration of forecast ensembles using Bayesian model averaging, Geophys. Res. Lett., 33, L19817, doi:10.1029/2006GL027126. (Pubitemid 46535104)
Wagner, B. (1995), Sampling design methods for groundwater modeling under uncertainty, Water Resour. Res., 31(10), 2581-2591.
Walker, W., and V. Marchau (2003), Dealing with uncertainty in policy analysis and policy making, Integr. Assess., 4(1), 1-4.
Wasserman, L. (2000), Bayesian model selection and model averaging, J. Math. Psychol., 44(1), 92-107, doi:10.1006/jmps.1999.1278.
Wöhling, T., and J. Vrugt (2008), Combining multiobjective optimization and Bayesian model averaging to calibrate forecast ensembles of soil hydraulic models, Water Resour. Res., 44, W12432, doi:10.1029/ 2008WR007154.
Ye, M., S. Neuman, and P. Meyer (2004), Maximum likelihood Bayesian averaging of spatial variability models in unsaturated fractured tuff, Water Resour. Res., 40, W05113, doi:10.1029/2003WR002557.
Ye, M., S. Neuman, P. Meyer, and K. Pohlmann (2005), Sensitivity analysis and assessment of prior model probabilities in MLBMA with application to unsaturated fractured tuff, Water Resour. Res., 41, W12429, doi:10.1029/ 2005WR004260. (Pubitemid 43151130)
Ye, M., K. Pohlmann, and J. Chapman (2008a), Expert elicitation of recharge model probabilities for the Death Valley regional flow system, J. Hydrol., 354(1-4), 102-115, doi:10.1016/j.jhydrol.2008.03.001.
Ye, M., P. Meyer, and S. Neuman (2008b), On model selection criteria in multimodel analysis, Water Resour. Res., 44, W03428, doi:10.1029/ 2008WR006803.
Ye, M., K. Pohlman, J. Chapman, G. Pohll, and D. Reeves (2010a), A model-averaging method for assessing groundwater conceptual model uncertainty, Ground Water, doi:10.1111/j.1745-6584.2009.00633.x, in press.
Ye, M., D. Lu, S. Neuman, and P. Meyer (2010b), Comment on "Inverse groundwater modeling for hydraulic conductivity estimation using Bayesian model averaging and variance window" by Frank T.-C. Tsai and Xiaobao Li, Water Resour. Res., 46, W02801, doi:10.1029/ 2009WR008501.
