A global and efficient multi-objective auto-calibration

Q IWA Publishing 2007 Journal of Hydroinformatics | 09.4 | 2007

277

A global and efficient multi-objective auto-calibration and uncertainty estimation method for water quality catchment models A. van Griensven and T. Meixner

ABSTRACT Catchment water quality models have many parameters, several output variables and a complex structure leading to multiple minima in the objective function. General uncertainty/optimization methods based on random sampling (e.g. GLUE) or local methods (e.g. PEST) are often not applicable for theoretical or practical reasons. This paper presents “ParaSol”, a method that performs optimization and uncertainty analysis for complex models such as distributed water quality models. Optimization is done by adapting the Shuffled Complex Evolution algorithm (SCEUA) to account for multi-objective problems and for large numbers of parameters. The simulations performed by the SCE-UA are used further for uncertainty analysis and thereby focus the uncertainty analysis on solutions near the optimum/optima. Two methods have been developed that select “good” results out of these simulations based on an objective threshold. The first method is based on x 2 statistics to delineate the confidence regions around the optimum/optima and the second method uses Bayesian statistics to define high probability regions. The ParaSol method was applied to a simple bucket model and to a Soil and Water Assessment Tool (SWAT) model of Honey Creek, OH, USA. The bucket model case showed the success of the method in finding the minimum and the applicability of the statistics under importance sampling. Both cases showed that the confidence regions are very small when the

x 2 statistics are used and even smaller when using the Bayesian statistics. By comparing the ParaSol uncertainty results to those derived from 500,000 Monte Carlo simulations it was shown

A. van Griensven (corresponding author) Environmental Sciences, University of California, Riverside, CA 92507, USA now at: UNESCO-IHE Water Education Institute, Department of Hydroinformatics and Knowledge Management, PO Box 3015, 2601 DA Delft, The Netherlands E-mail: [email protected] and BIOMATH, Department of Applied Mathematics, Biometrics and Process Control, Ghent University, Coupure Links 653, B-9000 Ghent, Belgium T. Meixner Environmental Sciences, University of California, Riverside, CA 92507, USA Now at: College of Engineering, Department of Hydrology and Water Resources, University of Arizona, 845 North Park Avenue, Tucson, AZ 85721-0158, USA Tel: +1 520 626153 Fax: +1 520 6211422 E-mail: [email protected]

that the SCE-UA sampling used for ParaSol was more effective and efficient, as none of the Monte Carlo samples were close to the minimum or even within the confidence region defined by ParaSol. Key words

| auto-calibration, model, river basin, water quality

INTRODUCTION Hydrologic models are not yet able to describe all the macro-

values. Therefore it is dangerous to believe that a model is able

and microelements and corresponding processes of reality by

to predict the output variables of importance without being

means of parameters that are assessed a priori either

conditioned to reality by a calibration or even without a

experimentally, via field measurement, or using literature

verification on observations in cases of ungauged basins

doi: 10.2166/hydro.2007.104

278

A. van Griensven and T. Meixner | Multi-objective auto-calibration and uncertainty estimation method

(Gupta & Sorooshian 2003). Meanwhile, models are becoming

Journal of Hydroinformatics | 09.4 | 2007

therefore not applicable in cases with multiple minima in the

more complex through integration of sub-models, extensions

objective function (Vrugt et al. 2003a). Water quality catchment

to water quality calculations or by developments in obser-

models thus require methods that operate globally and may

vation methods, data storage systems or computer technology

consider the error on multiple output variables (output signals)

(Grayson & Bloschl 2001). Where in the past, manual

by means of a search method, as opposed to random sampling

calibrations were the rule, they are now more difficult due to

methods, to find the optimum. The single-objective optimization

this increased model complexity (Duan 2003). Thus, develop-

problem can be solved in an efficient and effective way by the

ment and implementation of automated methods for par-

Shuffled Complex Evolution algorithm (SCE-UA) (Duan et al.

ameter calibration are important topics in hydrological

1992; Duan 2003). Some extensions of the method exist that deal

modeling (Duan et al. 2003). A good automatic method should

with multi-signal problems by aggregation to a Global Optim-

be as independent as possible of assumptions (such as

ization Criterion (GOC) (van Griensven & Bauwens 2003;

assumptions of model linearity), be general (search over

Madsen 2003). These optimization methods are not associated

whole parameter space), effective (find the optimal or most

with uncertainty assessments.

probable parameter set) and be as efficient (require few model

Often, when calibrating or generating uncertainty esti-

executions) as possible (Duan et al. 1992; Beven 1993; Gupta &

mates for distributed hydrologic or water quality models it is

Sorooshian 2003). It is difficult to combine all of these

not practically feasible to use the most precise methods in all

properties in any one method.

cases due to computation time limitations. It is better to use

While optimization tools are useful to point out the best

an alternative method that leads to “approximately right”

solution, they do not provide information on the uncer-

solutions than to be “precisely wrong”. This statement means

tainty of parameters and model outputs. Experience has led

that it is better to use a robust and global method than a local

to the insight that several parameter combinations could

search method that is not effective in locating the global

give equally good results and has led some to doubt the

optimum (Vrugt et al. 2003b).

concept of an optimal solution and instead advocate for the

This paper describes a new multi-objective uncertainty

equifinality of models and their parameters sets (Beven &

method ParaSol (Parameter Solutions) that is efficient in

Binley 1992; Beven & Young 2003). Thus, instead of

optimizing a model and providing parameter uncertainty

searching for a single optimal solution for model parameters

estimates without being based on assumptions on prior

and output, several researchers have proposed alternative

parameter distributions for the sampling strategy. It is based

methods for finding a range, a distribution or a probability

on statistical techniques to define an objective, statistically

function of model parameters and outputs (Beven & Binley

based, threshold that is used to subdivide simulations into

1992; Gupta et al. 1998). These methods have not been

“good” and “bad” subsets. It should be noted that ParaSol only

applied to nor extensively investigated for complex dis-

provides uncertainty assessment for the model parameters;

tributed water quality models. The lack of uncertainty

more specifically the uncertainty in model parameters due to

analysis is thus an important gap in providing reliable model

insufficient observed data to identify the free parameters.

outputs for these types of models such as the Soil and Water

Other sources of uncertainty, such as errors in forcing input

Assessment Tool (SWAT) (Arnold et al. 1998).

data (rainfall, temperature, etc.), spatial data errors (GIS data),

Distributed water quality catchment models have many

model structure (spatial scaling, mathematical equations) or

parameters, several output variables and a complex structure

observed data errors, are represented by the residual variance

leading to multiple minima in the objective function representing

and thus are not dealt with directly in this study.

the model error. General uncertainty analysis methods based on random sampling such as GLUE (Beven & Binley 1992) are often not applicable on such models, for the practical reason that they require too many runs. Local methods, such as PEST

PARASOL METHOD

(Doherty 2000), are very efficient, but they are typically based on

The ParaSol method calculates objective functions (OF)

first-order approximations of the error function and are

based on model outputs and observation time series, it

279



aggregates these objective functions into a global optimiz-

assumptions made in determining the uncertainty bounds

ation criterion (GOC), minimizes the OF or a GOC using

for ParaSol. The following objective functions can be used.

the SCE-UA algorithm and performs uncertainty analysis

Sum of the squares of the residuals (SSQ). This method

with a choice between two statistical concepts.

aims at matching a simulated series to a measured time series

The shuffled complex evolution (SCE-UA) algorithm

SSQ ¼

X

½xn;sim 2 xn;obs 2

ð1Þ

n¼1;N

The SCE-UA algorithm is a global search algorithm for the minimization of a single function that is implemented to deal

with N the number of pairs consisting of the simulation

with up to 16 parameters (Duan et al. 1992). It combines the

xn,sim and the corresponding observation xn,obs.

direct search method of the simplex downhill descent pro-

The sum of the squares of the difference of the measured

cedure (Nelder & Mead 1965) with the concept of a controlled

and simulated values after ranking (SSQR). The SSQR

random search by a systematic evolution of points in the

method aims at the fitting of the frequency distributions of

direction of global improvement, competitive evolution

the observed and the simulated series.

(Holland 1995) and the concept of complex shuffling. In the

After independent ranking of the measured and the

first step (zero-loop), SCE-UA selects an initial “population” by

simulated values, new pairs are formed and the SSQR is

random sampling throughout the feasible parameters space for

calculated as

p parameters to be optimized (delineated by given parameter ranges). The population is portioned into several “complexes” that consist of 2p þ 1 points. Each complex evolves independently using the simplex downhill descent algorithm. The complexes are periodically shuffled to form new complexes in order to share information between the complexes.

SSQR ¼

X ½xj;sim 2 xj;obs 2

ð2Þ

j¼1;N

where j represents the rank. As opposed to the SSQ method, the time of occurrence of a given value of the variable is not accounted for in the

SCE-UA has been widely used in watershed model

SSQR method (van Griensven & Bauwens 2003). The

calibration and other areas of hydrology such as soil

method might be of use in a stochastic analysis, where the

erosion, subsurface hydrology, remote sensing and land

time of occurrence is not as important as the frequency of

surface modeling (Duan 2003). It has been generally found

occurrence.

to be robust, effective and efficient (Duan 2003). The SCEUA has also been applied with success on SWAT for the hydrologic parameters (Eckardt & Arnold 2001) and hydrologic and water quality parameters (van Griensven & Bauwens 2003).

Multi-objective optimization Since the SCE-UA minimizes a single function, it cannot be applied directly for multi-objective optimization. There are

Objective functions

several methods available in the literature to aggregate objective functions to a global optimization criterion

The application of an optimization algorithm involves a

(Madsen 2003; van Griensven & Bauwens 2003) for multi-

proper selection of a function that must be minimized or

objective calibration, but existing methods do not provide

maximized that replaces the expert perception of curve-

uncertainty analysis.

fitting during the manual calibration. There are numerous

A statistically based aggregation method is found within

possible objective functions, often called error functions,

Bayesian theory (Box & Tiao 1973). For a specific time series

and many reasons to select one rather than another; for

of N data Yobs ¼ [y1,obs, … , yn,obs, yN,obs] corresponding to

discussion on this topic see Gupta et al. (1998) and Legates

the simulations Ysim ¼

& McCabe (1999). The types of objective functions selected

residuals (yn,sim-yn,obs) are assumed to have a normal

for use with ParaSol are limited due to the statistical

distribution N(0, x 2) with constant variance. The variance

[y1,sim, … , yn,sim, yN,sim], the

280


For M independent objective functions, this gives

of the residuals is then estimated as

s2 ¼


SSQMIN N

ð3Þ

" # Y SSQm exp 2 2*s2m m¼1;M

pðujY obs Þ /

with SSQMIN being the sum of the squared errors for the

Applying Equation (3), Equation (9) can then be written

optimum solution of the objective function and N the number of observations (Box & Tiao 1973). Upon

as " # Y SSQm *nobsm exp 2 2*SSQm;min m¼1;M

the assumption that the initial parameter distribution is equal to the non-informational uniform distribution, the

ð9Þ

pðujY obs Þ /

probability that the parameter set x is the true parameter

ð10Þ

set representing reality set – or likelihood of a parameter set

where nobsm is the number of observations for the signal m.

x – consisting of the P parameters (x1, x2,… xP) when

In accordance with Equation (10), it is true that

conditioned by the observation yn,obs can be calculated as (Box & Tiao 1973) "

1 ðy 2y Þ2 Pðujyn;obs Þ ¼ pffiffiffiffiffiffiffi exp 2 n;sim 2 n;obs 2s 2ps2

ln½2pðujY obs /

#

M X SSQm *N m : SSQm;min m¼1

ð11Þ

ð4Þ We can thus optimize the likelihood for the GOC by minimizing a Global Optimization Criterion (GOC) defined

or

as follows: "

pðujyn;obs Þ / exp 2

ðyn;sim 2 yn;obs Þ 2s2

2

# :

ð5Þ

For a time series Yobs consisting of N observations, this gives

GOC ¼

M X SSQm *N m : SSQm;min m¼1

ð12Þ

With Equations (11) and (12), the probability can be related to the GOC according to

" # N Y 1 ðy 2y Þ2 pðujY obs Þ ¼ pffiffiffiffiffiffiffiN exp 2 n;sim 2 n;obs 2s 2ps2 n¼1

ð6Þ

pðujY obs Þ / exp½2GOC

ð13Þ

Thus the sum of the squares of the residuals receives a weight that is equal to the number of observations divided

or " pðujY obs Þ / exp 2

PN

n¼1

ðyn;sim 2 yn;obs Þ 2s2

2

by the minimum. However, the minima of the individual

# :

ð7Þ

objective functions (SSQ or SSQR) are initially not known. At each shuffling step in the SCE-UA optimization, an update is performed for the minima of the objective functions using the newly gathered information within the

It is then also true for the objective function SSQ1: " pðujY obs Þ / exp 2

SSQ1 2*s21

loop and, as a consequence, the GOC values are

# ð8Þ

recalculated. The main advantage of using Equation (12) to calculate the GOC is that it allows for a global uncertainty analysis

where SSQ1 is the sum of the squares of the residuals with

considering all objective functions as described below.

corresponding variance s1 for a certain output signal.

Note that, in all cases, integration of the probability over

In Equation (8), the objective function is related to a

the entire parameter space is equal to 1. Therefore, a rescaling

probability. According to Bayes’ theorem, a joint probability

or normalization is performed in order to assign absolute

can be calculated by multiplying independent probabilities.

probabilities through a weighting factor that is equal to

281



the integration of Equation (10) or (13) over the entire

statistics where the selected simulations correspond to the

parameter space for the sample population (Box & Tiao 1973).

confidence region (CR) or Bayesian statistics that are able to point out the high probability density region (HPD) for the parameters or the model outputs (Figure 2). A separation

Uncertainty analysis method The uncertainty analysis divides the simulations that have been performed by the SCE-UA optimization into “good” simulations and “not good” simulations and in this way is

of the “good” simulation allows for a further defining posterior parameter limits that are determined by the lower and upper values of the parameters in all the parameter combinations that have been selected.

similar to the GLUE methodology (Beven & Binley 1992). The simulations gathered by SCE-UA are very valuable as the

x2 method

algorithm samples over the entire parameter space with a focus of solutions near the optimum/optima. To increase the usefulness of the SCE-UA samples for uncertainty analysis, some adaptations were made to the original SCE-UA

For a single objective calibration for the SSQ, the SCE-UA will find a parameter set u p consisting of the p free parameters (u*1 ; u*2 ; … ; u*P ) that corresponds to the minimum

algorithm, to prevent being trapped in a localized minimum

of the sum of the squares (SSQ). According to x 2 statistics

and to allow for a better exploration of the full parameter range

(Bard 1974), we can define a threshold “c” for a “good”

and prevent the algorithm from focusing on a very narrow set

parameter set using the equation

of solutions. The most important modifications are: 1. At each shuffling step of the algorithm, the k worst results are replaced by random sampling (where k is equal to the number of complexes). This change prevents the method from collapsing around a local minimum. Similarly, Vrugt et al. (2003b) solved this problem of collapsing rapidly in the minimum by introducing randomness through integration of the Metropolis – Hastings algorithm (Metropolis et al. 1953; Hastings 1970) into the SCE-UA algorithm. As opposed to this, the randomness introduced in ParaSol involves the replacement of the worst results. 2. When a parameter value is under or above the parameter limits as set for the SCE-UA parameter space sampling,

c ¼ OFðu* Þ*ð1 þ

x2P;0:95 N2P

ð14Þ

whereby N is the number of observations, u p the vector with the best parameter values, x 2P,0.95 is the x 2 result for P parameter values and for a 95% confidence level. x 2P,0.95 gets a higher value for more free parameters P. For multiobjective calibration, the selections are made using the GOC of Equation (12) that normalizes the sum of the squares for the total of observations NT, equal to the sum of N1,…,Nm,…NM observations. A threshold for the GOC is for P parameters and a confidence level of 95% is calculated by c ¼ GOCðu* Þ*ð1 þ PM

x2P;0:95

m¼1

Nm 2 P

ð15Þ

the algorithm resets the parameter value to a value equal to the current minimum bound or maximum bound (these bounds are narrowed after each loop) instead of a

Thus all simulations with GOC , c are deemed acceptable.

randomly sampled value. In that way, the narrowing of the upper and lower boundaries is slowed. The ParaSol algorithm uses two techniques to divide the

Bayesian method

sample population of SCE-UA into “good” and “bad”

According to Bayes’ theorem (Bayes 1763), the probability

simulations. Both techniques are based on a threshold

p(ujYobs) of a parameter set u is described by Equation (13).

value for the objective function (or global optimization

After normalizing the probabilities (to ensure that the

criterion) to select the “good” simulations by considering all

integral over the entire parameter space is equal to

the simulations that give an objective function value below

1) a cumulative distribution can be made and hence a

this threshold. The threshold value can be defined by x 2

95% confidence region can be defined (Box & Tiao 1973).


282

As the parameter sets developed by SCE-UA were not sampled randomly but according to some importance sampling, it is necessary to develop a methodology that will avoid over-representation of the densely sampled regions. This problem is prevented by determining a weight for each parameter set by the following procedure:


Uncertainty analysis The selected good parameter sets (based on the “c” threshold that is calculated by one of the above methods) provide confidence limits on the parameters, defined by the lower and upper value in the selection. The corresponding model results for the selected good parameter sets provide

1. Dividing the P parameter ranges into K intervals.

the uncertainty bounds on the model outputs.

2. For each kth interval (between 1 and K) of the pth parameter (kp), the sampling density nsamp[ p,k ] is calculated by summing the number of times that the

SIMPLE BUCKET MODEL

interval was sampled for all SCE-UA simulations. A weight for a certain parameter set ui is then

Model description

estimated by 1. For each pth parameter, determine the interval kp (between 1 and K) of each parameter ui,p and consider the number of samples of the SCE-UA optimisation that were within that interval ¼ nsamp( p,kp). 2. The weight for the parameter set ui is then calculated as

Our first evaluation of the ParaSol method used a simple onestorage two-parameter model (Figure 1) which enables twodimensional visualization of parameter plots. In this model the precipitation values are added to the storage representing detention, interception and soil moisture losses. When the storage volume (parameter Smax in millimetres) is filled up, water flows over and is routed to runoff. When the storage is

Wðui Þ ¼ hQ

P p¼1

1 n sampðp; kp Þ

not filled or when it is overflowing the storage drains i1=P

ð16Þ

according to a fractional parameter (K) which is the fraction of water stored that drains to runoff in a single time step. This model is a one-dimensional simple representation of catch-

The “c” threshold is estimated by the following

ment hydrology and has an area of 0 since it is a onedimensional model. For a detailed description of the model

procedure: a. a. Sort all simulated parameter sets and GOC values according to decreasing probabilities.

see Sorooshian & Dracup (1980). For a 100 d long daily rain time series, daily flows were calculated using parameter values K ¼ 0.3 and Smax ¼ 100.

b. b. Multiply probabilities by weights. c. c. Normalize the weighted probabilities by division using PT with

PT ¼

S X

Wðui Þ* pðui jY obs Þ

ð17Þ

i¼1

with S the number of SCE-UA simulations, W(ui) the weight for parameter set ui and p(uijYobs) the probability of parameter set ui conditioned to the set of observations Yobs.d Sum normalized weighted probabilities starting from rank 1 until the sum gets higher than the cumulative probability limit (95% or 97.5%). The GOC corresponding to or just higher than the probability limit defines the “c” threshold.

Figure 1

|

Scheme of a simple bucket model with S the storage, Smax the maximum storage, and Qs and Qdrain partial flows contributing to the total flow Qtot.

283



A random heteroscedastic error (not auto-correlated) of

runoff (SCS curve number method (USDA Soil Conserva-

20% was added to the flow values for this ideal or “true” set

tion Service 1972)), soil percolation, lateral flow and

of parameters to get a synthesized observation series. These

groundwater flow and river routing (variable storage

100 d long rain and flow time series were then used to test

coefficient method (Williams 1969)) processes. The nutrient,

the optimization and uncertainty analysis methods.

erosion, crop and pesticide processes are based on the GLEAMS (Leonard et al. 1980), CREAMS (Knisel 1980) and EPIC (Williams et al. 1984) modelling tools. The catchment

Results

is sub-divided into sub-basins, river reaches and Hydro-

A total of 743 runs were performed by SCE-UA, of which 236

logical Response Units (HRUs). While the sub-basins can be

were selected for a 97.5 confidence region using the x 2 statistics.

delineated and located spatially, the further sub-division

The Bayesian statistics led to a smaller area consisting of only

into HRUs is performed in a statistical way by considering a

168 selections (Figure 2). Similar results are obtained for 10,000

certain percentage of sub-basin area, without any specified

Monte Carlo samples (Figure 2), showing that the importance

location in the sub-basin. ParaSol is programmed within the

sampling by SCE-UA does not falsify the results. It is evident

SWAT2003 version.

that the statistical methods and the sampling methods cover the true values of the model parameters, since the confidence

Parameter change options for SWAT

ranges bracket the true values (Table 1). In the ParaSol algorithm, as implemented with SWAT 2003, parameters affecting hydrology or pollution can be changed either in a lumped way (over the entire catchment), or in a

SWAT CASE STUDY

distributed way (for selected sub-basins or HRUs). The parameters can be modified by replacement, by addition of

SWAT

an absolute change or by a multiplication by a relative

The Soil and Water Assessment Tool (SWAT) (Arnold et al.

change. A relative change means that the parameters, or

1998) is a semi-distributed and semi-conceptual model that

several distributed parameters simultaneously, are changed

is able to calculate water, nutrient and pesticide transport at

by a certain percentage. However, a parameter is never

the catchment scale on a daily time step. It represents

allowed to go beyond predefined parameter ranges. For

hydrology by interception, evapo-transpiration, surface

instance, all soil conductivities for all HRUs can be changed

Figure 2

|

Confidence region for the x 2 statistics and the Bayesian statistics for the two-parameter test model.


284

Table 1

|


97.5% confidence ranges for the two parameters of the test model

ParaSol Min

k

Smax

Monte Carlo Max

Min

Max

x2

0.26

0.41

0.25

0.41

Bayes

0.29

0.36

0.28

0.37

x2

78

115

75

118

Bayes

85

105

83

107

simultaneously over a range of 250 to þ50% of their initial values, which are different for the HRUs according to their soil type. This mechanism allows for a lumped calibration of distributed parameters while they keep their relative physical meaning (soil conductivity of sand will be higher than soil conductivity of clay).

Case study on Honey Creek Honey Creek is a tributary of the Sandusky River (OH),

Figure 3

|

Location of the Honey Creek catchment within the Sandusky River basin (USA).

located within the Erie Watershed and Great Lakes basin (Figure 3). The SWAT model of Honey Creek covers an 2

this study (van Griensven & Bauwens 2003). They are listed

area of 338 km and consists of 1 sub-basin, represented by

in Table 2, as well as their sensitivity rank (rank 1 for the

five HRUs, a river reach and a point source. Average

most sensitive parameter). The distributed parameters (like

annual precipitation ranges from 881 mm at Fremont to

curve number) have been modified according to a relative

964 mm at Bucyrus. Historic precipitation data for the

change in the range 250% and þ 50%, whereby the HRUs

catchment show the highest monthly amount for July

having initially the highest (lowest) parameter values will

(99 mm) and the smallest amount for February (48 mm).

remain having the highest (lowest) values. Lumped par-

Annual mean discharge for the Honey Creek at Melmore

ameters (e.g. surface runoff lag coefficient) are modified

3 21

is 3.8 m s

.

across their suitable range via replacement.

The model was abstracted from the Sandusky model

Three different optimizations were carried out, for one,

that was provided by the University of Florida to the

two or three objective functions. One strategy (1OF) only

research group at the University of California, Riverside

focused on the SSQ for flows (Q-SSQ), requiring 6242 runs.

(van Griensven et al. 2006). Daily observations during the

Another strategy (2OF) was performed and joint objective

years 1998 – 1999 were used to calibrate the model. These

functions using SSQ and SSQR for the flow (Q-SSQR)

consisted of 661 flow observations and 518 sediment load

within 13,490 runs. The latter calibration puts more

estimates.

restrictions on the model, as it aims to improve model simulations of high flow events as well as middle and low

Application of ParaSol to Honey Creek

flow events. Automated calibrations based on SSQ alone often tend to force the model to underestimate the peak

The results of a global sensitivity analysis were used to select

flows in order to get lower SSQ values, apparent in many

ten parameters to be optimized for the ParaSol analysis of

auto-calibration studies such as Eckhardt & Arnold (2001)


285

Table 2

|


Parameters used in calibration, with sensitivity rank according to SSQ for the daily flows (Q) and the sediment concentrations (SS-conc) (van Griensven et al. 2004)

Parameter

Description1

Type

Range

CN2

SCS runoff curve number for moisture condition II (%)

Distributed

[250,50]

3

2

SMFMX

Maximum melt rate for snow during (mm/8C/day)

Lumped

[0,10]

2

17

ch_k

Channel conductivity (mm/hr)

Lumped

[0,150]

5

14

SMTMP

Snow melt base temperature (8C)

Lumped

[0,5]

7

5

ALPHA_BF

Baseflow alpha factor (days)

Lumped

[0,1]

8

1

USLE-P

Erosion management control factor (%)

Distributed

[250,50]

surlag

Surface runoff lag coefficient

Lumped

[0,10]

sol_awc

Available water capacity of the soil layer (mm/mm soil)

Distributed

[250,50]

SMFMN

Minimum melt rate for snow during the year (mm/8C/d)

Lumped

SFTMP

Snowfall temperature (8C)

Sol_z

Soil depth (%)

1

Rank (Q)

No effect

Rank (SS-conc)

4

1

7

10

3

[0,10]

7

6

Lumped

[0,5]

15

6

Distributed

[250,50]

9

10

Distributed parameters are varied according to a relative change that maintains their spatial relationship while lumped parameters are changed via replacement with a new value.

or Yapo et al. (1998). For many long-term studies, it is more

uncertainty, structural problems or system variability that

important to have a statistically correct representation of

may not fully be captured by the observed period, are not

the basin than a low SSQ (and hence a high Nash-Sutcliffe

considered. These sources of uncertainty may be very

efficiency (Nash & Sutcliffe 1970)). A third strategy “3OF”

important (Yapo et al. 1996; Gupta et al. 1998; Kavetski

corresponds to an optimizing strategy for a model to be

et al. 2003).

used in sediment load management. In addition to the two objective functions of the 2OF strategy, the calibration includes the SSQ for sediment loads (SS-SSQ) as well and

Bayesian versus x2-method

allows for investigating a calibration and parameter uncertainty analysis for both flow and water quality variables. The optimization algorithm SCE-UA ended after 9093 runs.

The Bayesian method had fewer selections and a narrower confidence region (Table 3 and Figure 2). This result is due to the fact that the Bayesian method does not account for the number of parameters in the calculation of the thresholds, while the x 2 method does. Indeed, when we

Results

apply the x 2 method with only one degree of freedom, a

The uncertainty analysis for both techniques is based on a

similar number of selections were obtained as in the

threshold for the global optimization criterion that would

Bayesian method (results not shown). Therefore, the x 2

yield a 97.5% confidence region. In general, the uncertainty

method was preferred to the Bayesian method as the

analysis resulted in narrow confidence bounds. It is

statistics are more advanced, as they account for the

important to keep in mind that the method only addresses

number of free parameters while giving more realistic

the parameter uncertainty as determined by data avail-

results. Thus, further discussion of the results is based on

ability. Other sources of uncertainty, such as input

the x 2 statistics.


286

Table 3

|

that the statistics (both x 2 and Bayesian) only consider

Number of simulations performed and in confidence region

Total


uncertainty in the parameter calibration process. Hence

x2

Bayes

1OF

6242

2512

1144

2OF

13490

4483

1820

3OF

9093

978

297

these statistics only account for unbiased random errors on observations, and these random errors can be averaged out if enough data is available. For that reason, it was not expected that all observations should fall within the confidence region. When models are to be used in decision-making, the outputs are generally converted to a single value or for a few

Parameter confidence regions

values, depending on the purpose of the policy. In water

None of the uncertainty regions for the parameters included

resources management, this value may be the yearly mass

the initial upper or lower parameter bounds. The size of the

balance and in that case it is important that the confidence

parameter regions for the x 2 confidence region varied

ranges are known. The main focus of uncertainty analysis

between the 1OF, 2OF and 3OF cases (Figure 4). It is very

for these decisions should be the uncertainty on these

clear that these ranges became narrower when more

output values. Therefore, uncertainty ranges were calcu-

objective functions, and thus more observations or restric-

lated for the mass balance of the outputs (Figure 6).

tions, were considered in the calibration. The graph also shows the limit of the procedure: it was not always possible to give a full range of non-sensitive parameters such as the parameter “USLE-P” on flow-only calibrations. This result means that it is possible that the parameter confidence regions may be underestimated (too narrow). This defect is a drawback of the sampling strategy used by SCE-UA that, in order to be efficient, focuses more on the sensitive parameters to find better parameter solutions. Therefore ParaSol should not be used to investigate parameter identifiability.

SSQ calibration (1OF) High errors can be observed for the 1OF case. The objective function for this case is equal to the sum of the squares of the errors of discharge, which is not an unbiased estimator of the errors and thus does not necessarily guarantee that the confidence region correctly estimates the mass balance (Figure 6). Additionally, this case resulted in relatively large confidence bounds for the daily outputs (Figure 5), whereby there were large underestimations present in the confidence region.

Confidence regions for the model outputs SSQ and SSQR joint calibration (2OFs) The daily simulations show very narrow uncertainty ranges (Figure 5). The reason for these narrow bounds is

SSQR gave a better representation of the global mass balance and the joint calibration puts more restrictions on the calibration and uncertainty calculations. Thus the 2OFs case clearly led to less biased results and smaller confidence bounds on the model outputs compared to the 1OF calibration (Figures 5 and 6). Water quantity and water quality joint calibration (3OFs) The joint water flow and sediment load calibration consisted of the two objective functions that were used for the 2OFs case with an additional objective function for the sum of the

Figure 4

|

Relative confidence regions for the parameters used in the ParaSol application to Honey Creek.

squares (SSQ) of the daily sediment loads. This strategy appeared to be effective, giving reasonable assessments for

287

Figure 5


|


Confidence intervals for simulated time series for the flows with strategy 1OF (a); with strategy 2OFs (b); with strategy 3OFs (c) and for the simulated daily sediment loads with strategy 3OFs (d). The confidence intervals are often small such that they look like a line instead of a bounded area. Note the slightly wider uncertainty bounds in panel (a) as opposed to the other panels of the figure.

the water balance and the sediment load balance but with a

necessary when compared to 2OFs (Figures 5(c) and 6).

not insignificant prediction bias in the results (Figure 6).

Such trade-offs are indications of structural problems in the

While the daily timing of peak sediment flux is well simulated,

model that prevent having the best results for both water

the quantification of the individual peaks was often not

flow and sediment loads.

correct. Such results were not surprising, since within SWAT there is a randomization in the calculation of the sediment loads through a storm peak assessment (Arnold et al. 1998). The daily rainfall data do not provide information on the subdaily rainfall intensity and therefore the sediment forecasts of SWAT are not fully deterministic but use some stochastic procedure to generate 30 min peak intensities, based on monthly statistical data of the climate. The joint calibration 3OFs had some impact on the water flow calculations due to trade-offs that appeared to be

Figure 6

|

Bias on the overall mass balance for flow (Q) and sediments (SS).


288


estimator of reality. This result does not mean that we

LIMITATIONS AND RECOMMENDATIONS

should simply revise or relax our statistical assumptions to

The ParaSol method is built on several assumptions:

give us wider bounds, as is often done with GLUE or other

a. normal and random distribution of the residuals

analyses. The revisions of uncertainty bounds must be done in a way that incorporates some information about the

(Equation (3)) b. independence of the objective functions (Equation (9))

nature of how the model is biased and then utilizes this

c. general assumption that the model is “true”

information to assess model predictive uncertainty. This

In real cases, these assumptions may not be correct and

topic should also be a fruitful and worthwhile pursuit.

may lead to underestimations of the parameter uncertainty. Therefore further investigation is needed to deal with these problems. For instance, other distributions could be used as the assumed distribution of the model residuals. One possible method would be to use the exponential power density E(s,g) with the use of the following equations to compute the likelihood of a parameter set u for describing the observed data Yobs (Box & Tiao 1973): pðujY obs Þ " / exp 2cðgÞ

The ParaSol method is related to many other optimization/uncertainty analysis methods that have been applied to catchment models. As in the Generalised Likelihood Uncertainty Estimation method (GLUE), it operates by selecting “good” parameter sets out of a sample population

2=ð1þgÞ # T X ðxi;measured 2 xi;simulated s

(Beven & Binley 1992). GLUE may also use Bayesian ð18Þ

i¼1

likelihood measures as is done here (Thiemann et al. 2001; Beven & Young 2003). An important difference is that GLUE uses Monte Carlo random samples and is thus not

where cðgÞ ¼

COMPARISON TO OTHER OPTIMIZATION/UNCERTAINTY ANALYSIS METHODS

G½3ð1 þ gÞ=2 G½ð1 þ gÞ=2

1=ð1þgÞ

able to locate the confidence region around a best solution :

ð19Þ

within a reasonable number of simulations. For comparison, the ParaSol results were compared to Monte Carlo sampling. A total of 500,000 Monte Carlo simulations were

Investigations should be pursued on the independence

generated and simulated with the SWAT model of Honey

of objective functions for the purposes of joint optimization

Creek. Subsets of the first generated 1000, 5000, 10,000,

and uncertainty analysis as was done here. Such studies will

50,000, 100,000, 250,000 and 500,000 are used to compare

be non-trivial since the interactions between objective

to Parasol and analyse the effectiveness of a random-

functions in water quality models would be expected to be

sampling based optimization and uncertainty analysis as

nonlinear due to the nature of the models. These inter-

opposed to the directed search methodology used here. The

actions may or may not be significant. We are not aware of

lowest value for each of these subsets is indicative of

any studies that have investigated this interaction within a

the performance of each of these Monte Carlo samples

Bayesian or even a statistical framework. Such studies

(Table 4). It is clear that the lowest objective function values

would prove valuable to the broader hydrologic and

for 500,000 simulations are still higher than what was found

environmental modelling communities. These studies

with ParaSol. This result means that ParaSol was a very

would no doubt shed light on how basin water quality

efficient and effective optimization method and that the

models represent and misrepresent coupled hydrologic and

number of Monte Carlo simulations that would be necess-

water quality processes.

ary to reach equally good results is many orders of

As for the assumption of model “trueness”, this

magnitude higher than that needed for ParaSol (, 14,000

assumption is not true of our analysis. The model not

simulations as opposed to more than half a million for

being correct indicates a risk that we are using a biased

Monte Carlo methods).


289

Table 4

|


The minimum value for the objective functions Q-SSQ, Q-SSQR and SS-SSQR as a function of the number of simulated Monte Carlo random samples for the parameters

Q-SSQ

Q-SSQR

SS-SSQR

1000

2.78E þ 04

1.18E þ 03

4.19E þ 07

5000

2.57E þ 04

8.02E þ 02

2.90E þ 07

10,000

2.53E þ 04

7.90E þ 02

2.90E þ 07

50,000

2.40E þ 04

4.42E þ 02

1.54E þ 07

100,000

2.37E þ 04

4.39E þ 02

1.54E þ 07

250,000

2.32E þ 04

4.21E þ 02

1.50E þ 07

within the parameter space. Locating it with the Monte Carlo

500,000

2.32E þ 04

3.22E þ 02

1.15E þ 07

simulations would require a very large number of simulations.

ParaSol

2.14E þ 04

1.75E þ 02

4.65E þ 06

|

Figure 7

The minimum and maximum value for the Q-SSQ objective function of strategy 1OF in the x 2 confidence regions according to ParaSol or the Monte Carlo simulations.

Another important difference between ParaSol and GLUE is how each deals with multi-objective problems. GLUE operates by sequential selections, which may lead to Comparing the uncertainty analysis results using Monte Carlo sampling and ParaSol reveals other differences between the approaches (Table 5 and Figure 7). For the 1OF case, the Monte Carlo samples gave a reasonable number of selections (Table 5), but when compared to the ParaSol results, none of the selections was located in the confidence region when ParaSol was used (Figure 7). Again, ParaSol showed its effectiveness and efficiency in locating the optimal solution and confidence region. The ParaSol results gave much smaller confidence regions for the parameters as well (Figure 8). This

no parameter solutions when there are high trade-offs between the objectives (Freer et al. 2003). With ParaSol, a high trade-off will lead to a wider uncertainty region and thus to more parameter sets. The x 2 method to define the threshold is related to firstorder approximations of parameter uncertainty such as in PEST (Doherty 2000). These first-order approximations, however, use extrapolation method to select the confidence limits using the following equation: 2 * ðu 2 u* ÞT V 21 u ðu 2 u Þ # xP;0:95

ð20Þ

result indicates that the confidence region was a very small area for P parameters being normally distributed according to Table 5

|

Number of selections for the x 2 confidence region as a function of the number of simulated Monte Carlo random samples for the parameters

No. of random samples

1OF

2OFs

3OFs

1000

9

2

1

5000

3

3

1

10,000

13

1

1

50,000

28

3

2

100,000

32

1

2

250,000

35

2

1

500,000

56

2

1

N(u p,Vu). In contrast, this paper presents a sampling-based method that does not require the assumption of a normal distribution for the parameters.

Figure 8

|

The x 2 confidence range for the curve number results according to ParaSol or the Monte Carlo simulations.

290



The Bayesian method has recently been used in many

For 2 years of daily observations, the confidence regions are

studies. These studies have either used a random sample to

small even when the x 2 statistics are used, because the

define high probability regions (Freer et al. 2001) or sampling

uncertainty analysis only covers the uncertainty in identification

methods are used that converge to sampling according to some

of the model parameters for a specific observation data series

probability distribution like SCEM-UA (Vrugt et al. 2003a,

and assumes that the model structure is perfect. These properties

Vrugt et al. 2003b). Using the SCE-UA simulations makes

limit the usefulness of the approach. However, the method

ParaSol much more efficient in calculation time yet still

allows consideration of questions pertaining to the applicability

relatively effective in finding optimal solutions and estimating

and informativeness of the available data for a basin.

uncertainty bounds, especially for nonlinear models. Given

A comparison of the ParaSol results to 500,000 Monte

the long run times of distributed water quality models (10,000

Carlo simulations showed that the SCE-UA sampling was

runs took approximately 2 weeks on a 2 GHz processor), the

very effective and efficient in delineating confidence regions

efficiency of the ParaSol method is necessary while it obviously

whereas Monte Carlo methods did not contain any

comes at the expense of some effectiveness and more

solutions within the ParaSol demarcated optimal region.

restrictive assumptions than methods such as GLUE. Still

Especially for water quality models, it is important that

the assumptions of ParaSol fall in between methods such as

the method deals with multi-objective problems and that it

GLUE and the more restrictive assumptions of PEST.

is efficient, as most of these models are demanding in

Another method that was typically designed for multi-

computation time while being effective in locating the

objective problems is Pareto optimization (Gupta et al. 1998;

confidence regions and providing uncertainty bounds for

Yapo et al. 1998). The latter is an interesting exercise that gives

model outputs. As ParaSol meets all these requirements, it

information on the degree of trade-offs that is needed between

fulfils an important need for water quality models.

the objectives. This information is related to uncertainty in model structure, but does not give a specific uncertainty estimate for the model parameters (Vrugt et al. 2003a).

ACKNOWLEDGEMENTS

Since the parameter uncertainty in water quality models can be large, ParaSol contributes to the reliability of water

The authors are grateful to Sabine Grunwald and Tom

quality models for decision-making by providing uncer-

Bishop of the University of Florida for sharing the Sandusky

tainty estimates for multiple model outputs. The optimiz-

catchment model and R. Srinivasan of Texas A&M

ation and uncertainty analysis were achieved with a high

University for supporting this research. Support for this

level of precision in an efficient way. The latter is definitely a

work was provided by the National Science Foundation

positive contribution since distributed water quality model-

through a CAREER award to T. Meixner (EAR-0094312).

ling is known to be computationally demanding.

REFERENCES CONCLUSION The ParaSol method performs a multi-objective optimization/uncertainty analysis in a single run by using a modified SCE-UA algorithm. Two approaches can be used to define a threshold for the selection of “good” simulations: one based on x 2 statistics and another based on Bayesian statistics. As the latter does not account for the number of free parameters in setting the threshold, it results in a smaller selection and in narrower confidence limits on parameters or outputs. Therefore, the x 2 statistical approach is preferred because the method uses more information.

Arnold, J. G., Srinivasan, R., Muttiah, R. S. & Williams, J. R. 1998 Large area hydrologic modeling and assessment part I: model development. J. Am. Wat. Res. Assoc. 34 (1), 73 – 89. Bard, Y. 1974 Non Linear Parameter Estimation. Academic Press, New York. Bayes, T. 1763 An essay towards solving a problem in the doctrine of chance. Phil. Trans. R. Soc. 53, 370– 418. Beven, K. 1993 Prophecy, reality and uncertainty in distributed hydrological modelling. Adv. Wat. Res. 16, 41 –51. Beven, K. & Binley, A. 1992 The future of distributed models: model calibration and uncertainty prediction. Hydrol. Process. 6, 279 –298. Beven, K. & Young, P. 2003 Comment on “Bayesian recursive parameter estimation for hydrologic models”. Wat. Res. Res.

291


39 (5), COM1-1 –COM1-4. by M. Thiemann, M. Torsset, H. Gupta, & S. Sorooshian. Box, G. E. P. & Tiao, G. C. 1973 Bayesian Inference in Statistical Analysis. Addison-Wesley-Longman, Reading, MA. Doherty, J. 2000 Model Independent Parameter Estimation (PEST). Manual. Watermark Computing, Australia. Duan, Q. 2003 Global optimization for watershed model calibration. In Calibration of Watershed Models (ed. Q. Duan, H. V. Gupta, S. Sorooshian, A. N. Rousseau & R. Turcotte), American Geophysical Union, Washington, DC. pp. 89 –104. Duan, Q., Gupta, V. K. & Sorooshian, S. 1992 Effective and efficient global optimization for conceptual rainfall-unoff models. Wat. Res. Res. 28, 1015 –1031. Duan, Q., Sorooshian, S., Gupta, H. V., Rousseau, A. N. & Turcotte, R. 2003 Advances in Calibration of Watershed Models. American Geophysical Union, Washington, DC. Eckhardt, K. & Arnold, J. G. 2001 Automatic calibration of a distributed catchment model. J. Hydrol. 251, 103– 109. Freer, J., Beven, K. & Ambroise, B. 2001 Bayesian estimation of uncertainty in runoff prediction and the value of data: an application of the GLUE approach. Wat. Res. Res. 37, 2161 –2173. Freer, J., Beven, K. & Peters, N. E. 2003 Multivariate seasonal period model rejection within the Generalised Likelihood Uncertainty Estimation procedure. In Calibration of Watershed Models (eds. Q. Duan, H. V. Gupta, S. Sorooshian, A. N. Rousseau & R. Turcotte), American Geophysical Union, Washington, DC. pp. 69 –88. Grayson, R. B. & Bloschl, G. 2001 Spatial modelling of catchment dynamics. In Spatial Patterns in Catchment Hydrology: Observations and Modelling (ed. R. B. Grayson & G. Bloschl), Cambridge University Press, Cambridge. pp. 51 –81. Gupta, H. V. & Sorooshian, S. 2003 Advances in automatic calibration methodologies of watershed models. In Advances in Calibration of Watershed Models (ed. Q. Duan, H. V. Gupta, S. Sorooshian, A. N. Rousseau & R. Turcotte), American Geophysical Union, Washington, DC. pp. 125 –132. Gupta, H. V., Sorooshian, S. & Yapo, P. O. 1998 Toward improved calibration of hydrologic models: multiple and noncommensurable measures of information. Wat. Res. Res. 34, 751–763. Hastings, W. K. 1970 Monte-Carlo sampling methods using Markov Chains and their applications. Biometrika 57, 97 – 109. Holland, J. H. 1995 Adaptation in Natural and Artificial Systems, University of Michigan Press, Ann Arbor, MI. p. 975. Kavetski, D., Franks, S. W. & Kuczera, G. 2003 Confronting input uncertainty in environemental modelling. In Calibration of Watershed Models (ed. Q. Duan, H. V. Gupta, S. Sorooshian, A. N. Rousseau & R. Turcotte), American Geophysical Union, Washington, DC. pp. 49 –68. Knisel, W. G. (ed.) 1980 CREAMS: A Field-scale Model for Chemicals, Runoff and Erosion from Agricultural Management Systems. USDA Conservation Research Rep. No. 26. US Department of Agriculture, Washington DC.


Legates, D. R. & McCabe, G. J. 1999 Evaluating the use of “goodness-of-fit” measures in hydrologic and hydroclimatic model validation. Wat. Res. Res. 35, 233 –241. Leonard, R. A. & Wauchope, R. D. 1980 The pesticide submodel. In CREAMS: A Field-scale Model for Chemicals, Runoff and Erosion from Agricultural Management Systems. USDA Conservation Research Report No. 26 (ed. W. G. Knisel), pp. 99–112. Madsen, H. 2003 Parameter estimation in distributed hydrological catchment modelling using automatic calibration with multiple objectives. Adv. Wat. Res. 26, 205 –216. Metropolis, N., Rosenbluth, A. W., Rosenbluth, M. N., Teller, A. H. & Teller, E. 1953 Equations of state calculations by fast computing machines. J. Chem. Phys. 21, 1087 –1091. Nash, J. E. & Sutcliffe, J. 1970 River flow forecasting through conceptual models. Part I A discussions of principles. J. Hydrol. 10, 282–290. Nelder, J. A. & Mead, R. A. 1965 Simplex method for function minimization. Comput. J. 7, 308– 313. Sorooshian, S. & Dracup, J. A. 1980 Stochastic parameter estimation procedures for hydrologic rainfall-runoff models: correlated and heteroscedastic error cases. Wat. Res. 16, 430–442. Thiemann, M., Trosset, M., Gupta, H. V. & Sorooshian, S. 2001 Bayesian recursive parameter estimation for hydrological models. Wat. Res. 37, 2521 –2535. USDA-Soil Conservation Service 1972 National Engineering Handbook Section 4 Hydrology. USDA Washington, DC, Chapters 4-10. van Griensven, A. & Bauwens, W. 2003 Multi-objective autocalibration for semi-distributed water quality models. Wat. Res. 39 (12), 1348. van Griensven, A., Meixner, T., Grunwald, S., Bishop, T. & Srinivasan, R. 2006 A global sensitivity analysis method for the parameters of multi-variable watershed models. J. Hydrol. 324 (1– 4), 10 –23. Vrugt, J.A., Gupta,, H.V., Bastidas, L.A., Bouten, W., and Sorooshian, S. (2003) Effective and efficient algorithm for multi-objective optimization of hydrologic models. Wat. Res. 39 (8), 1214. Vrugt, J. A., Gupta, H. V., Bouten, W. & Sorooshian, S. 2003b A shuffled complex evolution Metropolis algorithm for estimating posterior distribution of watershed model parameters. In Calibration of Watershed Models (ed. Q. Duan, H. V. Gupta, S. Sorooshian, A. N. Rousseau & R. Turcotte), American Geophysical Union, Washington, DC. pp. 105–112. Williams, J. R. 1969 Flood routing with variable travel time or variable storage coefficients. Trans. ASAE 12 (1), 100– 103. Williams, J. R., Jones, R. W. C. A. & Dyke, P. T. 1984 A modeling approach to determining the relationship between erosion and soil productivity. Trans. ASAE 27 (1), 129– 144. Yapo, P. O., Gupta, H. V. & Sorooshian, S. 1996 Automatic calibration of conceptual rainfall-runoff models: sensitivity to calibration data. J. Hydrol. 181, 23 – 48. Yapo, P. O., Gupta, H. V. & Sorooshian, S. 1998 Multi-objective global optimization for hydrological models. J. Hydrol. 204, 83 –97.

A global and efficient multi-objective auto-calibration

A global and efficient multi-objective auto-calibration

Suggest Documents

A Parallel Multiobjective Efficient Global Optimization - IEEE Xplore

Practical Autocalibration - 3Dflow

Autocalibration and Recurrent Adaptation: Towards a Plug and Play ...

Approximate quasi efficient solutions in multiobjective ... - SSMR

An Efficient Multiobjective Evolutionary Algorithm for Community ...

Efficient Multiobjective Optimization of Amphibious Aircraft Fuselage

Generation of efficient solutions in Multiobjective ...

Generation of efficient solutions in Multiobjective Mathematical ...

An Efficient Multiobjective Differential Evolution ... - Semantic Scholar

AN EFFICIENT METHOD FOR MULTIOBJECTIVE OPTIMAL

CAMERA AUTOCALIBRATION AND HOROPTER CURVES 1

Efficient Resource Allocation using a Multiobjective Utility Optimisation

Autocalibration: Finding Infinity in a Projective Reconstruction

Multiobjective global surrogate modeling, dealing ... - Semantic Scholar

Global convergence analysis of fast multiobjective

Effective and efficient algorithm for multiobjective ... - Semantic Scholar

An efficient method for multiobjective optimal control and optimal ...

Global Multiobjective Optimization via Estimation of ... - CiteSeerX

Global formulation for interactive multiobjective ... - Semantic Scholar

Autocalibration of a Projector-Screen-Camera System: Theory and ...

THE ABSOLUTE LINE QUADRIC AND CAMERA AUTOCALIBRATION

Multiobjective Order Assignment Optimization in a Global Multiple ...

and Multiobjective Bayesian Global Optimization for Mixed ... - liacs

Globally Convergent Autocalibration using Interval Analysis - CiteSeerX