Determining the optimal sample size in the Monte Carlo experiments1 Mustafa Y. ATA Department of Statistics, Faculty of Arts and Sciences, Gazi University, Ankara, Türkiye;e-mail:
[email protected] Abstract A convergence criterion for the Monte Carlo estimates will be proposed which can be used as a stopping rule for the Monte Carlo experiments. The proposed criterion searches a convergence band of a given width and length such that the probability of the Monte Carlo sample variance to fall outside of this band is practically null. After the convergence to the process variance realized according to the new rule, a confidence interval in the usual statistical sense can be determined for the steady-state mean of the process.
Key words: sequential confidence interval, stopping rule, Monte Carlo, convergence 1. Introduction In the Monte Carlo (MC) experiments, the aim of which is to estimate the expected value of some stochastic process, statistically valid confidence intervals (CI) can be constructed only if the steady-state mean and the variance of the process have the properties that 1 n ⎛ ⎞ w . p .1 lim ⎜ x n = ∑ x i ⎟ ⇒ µ ; n→∞ n i =1 ⎠ ⎝
(1.1)
. 1 n ⎛ ⎞ w . p .1 lim ⎜ s n2 = ( xi − x n ) 2 ⎟ ⇒ σ 2 ∑ n→∞ n − 1 i =1 ⎝ ⎠
where the MC sample {xi:i=1(1.1)n} is a realization of the stochastic process {X 1 , X 2 ,........}composed of independent and identically distributed (i.i.d.) random variables. Then by determining an acceptable CI half width (ACIHW) in terms of the absolute error, ε ≥ x n − µ and by appealing to the Central Limit Theorem(CLT) a CI can be constructed as (1.2)
⎛ σ ⎞ ⎟⎟ = 1 − α Pr ⎜⎜ x n − µ < z α 2 n⎠ ⎝
where z α 2 is the upper 100(1-α/2)th percentile of the standard normal distribution. Hence relying on the CI in (1.2), the deterministic sample sizes can be determined as, 1 Published in the Selçuk Journal of Applied Mathematics, 7(2): 103-108 ,2006 as being one of the best 21 papers presented in the International Conference on Modelling Simulation-AMSE’06 July 2006 in Konya-Selçuk University.
(1.3)
a CLT
n
2 ⎡⎛ σ⎞ ⎤ (ε, α, σ < ∞) = ⎢⎜ z α 2 ⎟ ⎥ ε⎠ ⎥ ⎢⎝ 2
according to the absolute error criterion which will be called here shortly as the Acceptable Confidence Interval Rule (ACIR). In practice, the variance of process is also unknown and it should be estimated simultaneously with the process mean. The moments of the process can be estimated by one of the two general approaches. In the fixedsample-size procedure which will be referred shortly as FCIR, a single simulation run of an arbitrary fixed length Nf is performed, and by using the computed variance of the MC sample of size Nf as the unbiased point estimate of the process variance, a deterministic sample size according to (1.3) is chosen for a given ACIHW. An experimenter using this procedure has to be content with either a default value of confidence level or with a default value of CI half width. Being have selected an acceptable CI half width, the experimenter usually prefers to perform the MC experiment in excessively long run lengths to avoid the default confidence level to be below than a reasonable level. Almost all of the published MC studies fix the MC sample size to an excessive number which is usually some multiple of number 1000, i.e. 1000m, m=1,2,… and being sure that the convergence is attained they do not need to state any confidence measure. Therefore in this procedure a saving in computing time due to not estimating process variance, may be overtaken by that required to generate the redundant MC sample points. In the sequential procedures, the length of a single simulation run is increased sequentially until an “acceptable” CI with a given width and confidence level can be constructed. The stopping rule proposed by Chow and Robbins[1] based on (1.3) is the standard sequential procedure in determining the optimal MC sample size and it will be referred here shortly as the Sequential Confidence Interval Rule (SCIR). If the computational time of generating a MC sample point is common to both procedure and the default MC sample size in sequential procedure is Ns which is much lower than Nf, then the SCIR is 100[(Nf- Ns)/Ns]% more efficient than the FCIR. If one assumes that the process is stationary and not autocorrelated, then there is not any theoretical difficulty in constructing statistically valid CIs, for the process mean and the variance respectively such as
(
)
(1.4)
Pr x n − µ < t1−α 2;n −1 s n2 n = 1 − α
(1.5)
⎡ (n − 1)s n2 (n − 1)s n2 ⎤ Pr ⎢ 2 < σ2 < 2 ⎥ =1− α χ α 2;n −1 ⎥⎦ ⎢⎣ χ1− α 2;n −1
where t1−α / 2;n −1 is the (1 − α / 2) th quantile of the t distribution with n-1 degrees of freedom and χ α2 / 2;n −1
and
χ α2 / 2;n −1 are the (α / 2) th and
(1 − α / 2) th quantile point, respectively, of the chi-square distribution with
n-1 degrees of freedom. The main argument in the present study is that it is more reasonable to have first the MC sample estimate of process variance via reducing the half width of sequentially constructed CIs for the process variance in (1.5) to a predetermined level of precision, i.e.
(1.6)
(
)
(
)
⎡ n * − 1 s 2* n * − 1 s n2* n δ=⎢ 2 − 2 χ1− α 2;n* −1 ⎢⎣ χ α 2;n* −1
⎤ ⎥ ⎥⎦
and to determine the MC sample size according to (1.3). However, since the chi-square distribution is not symmetric, constructing sequential confidence intervals with equal probability tails for the MC process variance is not a trivial job and requires considerable amount of computations.[2] Therefore, in estimating the process variance, the sequential confidence interval approach is not appropriate to be applied directly, from the point of computational efficiency. In this article, a convergence criterion will be introduced which can be used to devise a stopping rule for the MC experiments. Newly introduced convergence criterion will be applied in estimating the process variance and then the optimal MC sample size will be determined by (1.3). Therefore, it is a hybrid procedure consisting of the sequential and fixed-sample-size procedures. The general frame and the definition of the new criterion will be given in the next section. After then, the empirical distribution of the convergence band length will be explored via some Monte Carlo experiments, and finally, some concluding remarks will be made. 2. Empirical convergence Let X be a random variable with the distribution function F(x) and xi be a random sample from F(x) constituting the virtual observation in the ith trial of a MC experiment. A pooled sample consisting of all virtual observations from the 1st to the latest jth trial constitutes the MC sample {xi:i=1(1.1)j} from which the MC sample mean (2.1)
xj =
1 j ∑ xi j i =1
and the mean squares of the distances from the MC sample mean, i.e. the MC sample variance, (2.2)
s 2j =
1 j ( xi − x j ) 2 ∑ j − 1 i =1
for each j=1,2,........ are computed. So, the sequence of MC sample means and MC sample variances up to a large number N, i.e. (2.3)
{x
j
, s j : j = 1(1) N }
is a realization of a MC process. A MC process is strictly stationary in the sense that the MC sample mean will converge in probability to the expected value µ = X , if it exists, i.e. (2.4)
(
)
lim Pr x j − µ < ε → 1 . j →∞
For any given sufficiently large j, the MC sample mean in (2.1) and the MC sample variance in (2.2), a 100(1-α)% CI can be constructed with a half width of (2.5)
CI (ε ) = zα
s 2j 2
j
.
which will surely converge to zero as j → ∞ but slowly after some large j=N where s 2j gets sufficiently close to the true process variance σ 2 . In sequential stopping rule, for a given acceptable CI half width ACI(ε), MC experiment is ended where (2.6)
ACI (ε) = z α 2
s N2 . N
If (2.5) is to be achieved likely after the true variance of the process is converged by the MC sample variance, then a stopping rule can be based directly on the precision level of this convergence. Let ε be the precision level of this convergence. Then a sequential interval always covering the MC sample variances in (2.2) can be constructed with the upper and lower limits respectively as
(2.7)
⎧⎪U (s 2j −1 ), δ j = 0 ⎧⎪L(s 2j −1 ) , δ j = 0 2 ; L(s j ) = ⎨ 2 U (s ) = ⎨ 2 ⎪⎩s j + ε , δ j = 1 ⎪⎩s j − ε , δ j = 1
where
⎧⎪0 , δj =⎨ ⎪⎩1 ,
2 j
L( s 2j-1 ) < s 2j < U ( s 2j −1 )
{s
2 j
} {
}
≤ L( s 2j −1 ) ∨ s 2j ≥ U ( s 2j −1 )
.
The number of adjacent MC sample variances which shares the same upper and lower limits in (2.7), i.e. δ j = 0 for all of them, is a random variable Zj whose observed values can be defined as, with (2.8)
⎧⎪ z j −1 + 1 zj ←⎨ ⎪⎩0
,
δj =0
,
δ j =1
.
(2.7), and (2.8) constitute a sequence of shifting bands with fixed half width ε and variable length zj , in which the MC sample variance is trapped.
If the sample space of the random variable Zj is Ζ = {ζ i = i : i = 0,1,2,.........} and ζ is an index variable whose values corresponds to the points of Z, then replacing the convergence index j by ζ , (1.1) and (2.4) can be restated for the MC sample variance and the true variance of the process such as
(
)
lim(s 2j ) ⇒ σ 2 and lim Pr s 2j − σ 2 < ε → 1 , w. p .1
(2.9)
ζ →∞
ζ →∞
because as j → ∞ , also ζ → ∞ . For a given ε, the upper and lower limits in
( )
( )
(2.7) converge to some constants lim L s 2j → σ 2 − ε and limU s 2j → σ 2 + ε . ζ →∞
ζ →∞
Thus it can be asserted that an acceptable convergence band (ACB) with a width ε* and a lenght ζ* is achieved simultaneously with (2.6), i.e. j = N + ζ * −1 ⇔ z j = ζ * and a stopping rule can be proposed as (2.10)
{
}
N (ε*, ζ*) = j : z j = ζ * , j = 1,2,..... .
In this article, leaving the further asymptotic discussion of this proposition for a research in the near future, we will be content with a MC empirical verification of (2.10). The ACB half width ε* determines the floating-point precision of the MC estimate, and can be set to 10-d/2 where d is the desired number of significant digits after the decimal point. For a given ε*, the ACB length ζ* can be determined so as to be confident that the probability of the MC sample variance to fall out of the ACB(ε*,ζ*) is practically null for j ≥ N + ζ * . An optimal value for ζ* can be determined based on the information
(2.11)
(
)
Pr (ζ = ζ *) ≡ Pr Z j +1 = 0 ε*, Z j = ζ * ≈ 0.
3. Monte Carlo experiments The probability law in (2.11) can be approximated by some MC experiments. The MC experiments performed to obtain the empirical distribution in (2.11) showed that the random variable in question can be hypothesized well as a logarithmic series variate [3] whose density function is
(3.1)
Pr(ζ ) = c ζ {− ln(1 − c)ζ} , ζ = 1,2,....;0 < c < 1 −1
with the asymptotically unbiased estimator of the shape parameter c (3.2)
cˆ = 1 −
Prˆ(ζ = 1) . ζ
In the MC experiments, three stochastic processes were used to generate the MC sample points: (1) a discrete process of i.i.d. Bernoulli variate with a success probability of 0.5, denoted by B(0.5).
(2) a continuous process of i.i.d. Uniform variates with mean 0.5 and variance 0.083333, denoted by U(0,1). (3) a stationary autoregressive process defined as X i = 0.3 X i −1 + ai and denoted by AR(0,3), where the steady state mean and the variance of the process are 0 and 1.098901 respectively, and ai are i.i.d. Normal variates with mean 0 and variance 1. The design points were determined by selecting respectively the reasonable values for the levels of precision in convergence to the true value of process variance, i.e. for the ACB widths in each of the three experiments with the processes given above, as 0.01, 0.001, and 0.001. Then at each design point, 100 replications of a long MC simulation run performed until zj= 51 observed in each independent run . The frequencies of ζ = 1, 2, .....,50 were accumulated over all the replications and transformed to the relative frequencies f i (ζ = i ) , then an unbiased estimate of c in (3.2) was obtained as cˆ = 1 −
(3.3)
f 1 (ζ = 1) 50
∑ i. f (ζ = i) i =1
.
i
For a randomly selected case, the relative frequencies as the MC empirical probabilities and the theoretical probabilities computed from (3.2) by inserting the MC estimates of c are sketched together in Fig.1, to give an idea of how well the hypothesized density in (3.1) fits to the empirical one. 0,4
Empirical and Theoretical Probabilities
0,35
0,3
0,25 Empirical
Theoretical
0,2
0,15
0,1
0,05
0 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 CB Length
Fig.1. The empirical and the theoretical probabilities of the CB length ζ.
The global results of the MC experiments are summarized in Table1. which indicate that the reasonable value for the shape parameter c in (3.1) is about 0.9. Table 1. MC estimates of the shape parameter c. Model
µ
σ2
ε*
x(N )
B(0.5) U(0,1) AR(0.3)
0.5 0.5 0.0
0.2500 0.0833 1.0989
0.010 0.001 0.001
0.50147 0.2479 0.49980 0.0835 0.00400 1.1009
s 2 (N )
cˆ
0.9358 0.9499 0.9511
The figures under the columns x ( N ) and s 2 ( N ) are the grand averages of MC sample means and variances estimated in each replicate by the Shifting Convergence Band Rule (SCBR) with ACB(ε*, 50). According to the experimental results, the appropriate values for ζ* can be specified for a desired level of confidence 1-γ, as it is defined in the previous section, from the relation (3.4)
Pr(ζ = ζ*) ≅ 0.9 ζ* {− 2.3ζ*} ≤ γ . −1
An algorithm of the criterion to be used as an automated stopping rule for the standard MC experiments is presented in the appendix. The algorithm of the SCBR can be embedded in a typical Monte Carlo algorithm where the aim is to estimate the expected value of a process with a specified confidence level. After the convergence to the process variance is realized within a CB of a half-width ε*, and of a length ζ* , it returns the MC sample size N, the MC estimate of process mean and the MC estimate of process variance. In the algorithm of SCBR, u j and l j are the upper and the lower limits of the CB respectively at the jth trial. 4. Concluding remarks If a confidence interval for the steady-state mean of stochastic process is to be constructed via CLT, the MC estimate of the process variance is a prerequisite. It can be proposed that the convergence to the process variance can be determined by adopting the ACIR to the case of MC sample variance. But constructing the sequential confidence intervals for the process variance is inefficient since the distribution of the sample statistic is not symmetric one and therefore to determine the upper and lower points of the distribution with the tails having equal probabilities on both sides requires extra computational effort. However the presently proposed stopping rule bypasses this difficulty using the empirical convergence concept. It searches a convergence band of a given width and length such that the probability of the Monte Carlo sample variance to fall outside of this band is practically null which is certainly more efficient computationally than constructing sequentially valid confidence intervals for the process variance. After the convergence to the process variance realized according to the new rule, a confidence interval in the usual statistical sense can be determined for the steady-state mean of the process.
References 1. Chow, Y.S. and Robbins, H. (1965): On the asymptotic theory of fixed-width confidence
intervals for the mean, Annals of Mathematical Statistics, 36, 457-462. A.M. , Graybill,F. and Boes, D.C. (1974): Introduction to the Theory of Statistics, 3rd ed., McGraw-Hill, Tokyo.. 3. Evans, M. , Hastings, N. and Peacock, B. (1993): Statistical Distributions, 2nd ed., John Wiley and Sons, Inc., New York. 2. Mood,
Appendix: An algorithm for SCBR Initialize: u0 ← a;l0 ←b;ε ←ε*;ζ ←ζ*;T0 ←0;TK0 ←0 Do Steps: 1- 6, until z j = ζ*; Step: 1 j ← j +1, generate xj , Step: 2 Tj = Tj −1 + xj ,TKj = TKj −1 + x2j , TK T Step: 3 xj ← j , sj ← j − x2j , j j l j −1 ≤ sj ≤ uj −1 ⎧⎪0, Step: 4 δj = ⎨ {sj < lj−1}∨{sj > uj−1}, ⎪⎩1, δj = 0 ⎧⎪uj −1, ⎧⎪l j −1, Step: 5 uj = ⎨ , lj = ⎨ * * δj ≠ 0 ⎪⎩sj − ε , ⎪⎩sj + ε , ⎧⎪z j −1 +1, δj = 0 Step: 6 z j ← ⎨ . δj ≠ 0 ⎪⎩0, Return: N ← j, σˆ 2 ←uj − ε*; µˆ ← xj •
δj = 0 , δj ≠ 0
