Using Ranking and Selection to `Clean Up' After ... - CiteSeerX

Using Ranking and Selection to `Clean Up' After Simulation Optimization Justin Boesel The MITRE Corporation Barry L. Nelson Seong-Hee Kim Northwestern University August 17, 1999 Abstract

In this paper we address the problem of nding the simulated system with the best (maximum or minimum) expected performance when the number of systems is large and initial samples from each system have already been taken. This problem will be encountered when a heuristic search procedure|perhaps one originally designed for use in a deterministic environment|has been applied in a simulationoptimization context. Because of stochasticity, the system with the best sample mean at the end of the search procedure may not coincide with the true best system encountered during the search. This paper develops statistical procedures that return the best system encountered by the search (or one near the best) with a prespeci ed probability. We approach this problem using combinations of statistical subset-selection and indierence-zone ranking procedures. The subset procedures, which use only the data already collected, screen out the obviously inferior systems, while the indierence-zone procedures, which require additional simulation eort, distinguish the best from the less obviously inferior systems.

1 Introduction Many of the current approaches to simulation optimization can be placed into one of two categories: 1. Asymptotically convergent search algorithms: These algorithms, encountered frequently in the academic literature, guarantee that the best system is returned 1

as search eort goes to in nity. Restrictive conditions, however, can make them dicult to apply to a broad range of problems, and the amount of simulation eort required to converge can be prohibitive. 2. Deterministic search algorithms applied to stochastic problems: These algorithms, which are widely implemented in commercial software packages, may provide good solutions, but may also provide misleading results because they frequently ignore stochastic variation and they provide no statistical guarantees. While the rst approach guarantees to return the true best solution, the second approach does not. The aim of this paper is to \clean up" results generated by the second approach. Speci cally, we will develop methods that return the best system of all those systems visited by the search|or one within a user-speci ed distance of the best|with a pre-speci ed probability. Our goal is to deliver this statistical guarantee with as little additional simulation eort as possible beyond what has already been expended in the search. This paper will not discuss methods for improving the search or generating new and better solutions. Rather, it will deal only with the problem of nding the best solution among those already visited by the search algorithm. Implementation of this work in simulation-optimization software is described in Boesel, Nelson and Ishii (1999). The procedures developed here may also be used independently of a search when there are a xed (and perhaps large) number of alternatives to compare. The present paper extends the results in Nelson, Swann, Goldsman and Song (1998) to make them useful for the optimization context. The remainder of the paper is organized into four sections. Section 2 gives a brief overview of screening and selection procedures, and shows how they can work together to clean up after a search. The next two sections present distinct strategies for combining screening and selection. Section 3 describes a simple \restart" procedure that rst screens all systems visited by the search to remove the clearly inferior ones, then takes additional 2

data on the survivors to select the best. An algorithm to nd the best sample size when using this procedure is given. Section 4 describes a procedure that sorts the systems by rst-stage (search) sample mean prior to screening, then screens and selects sequentially. An empirical evaluation and comparison of these approaches is provided in Section 5. The Appendix contains proofs of the validity of the procedures developed in the paper, as well as supporting procedures and results.

2 Background We assume that a preliminary or rst-stage set of simulation output data generated by a search procedure is \dropped into our laps." Let k be the number of dierent systems in the data set, and let n0i be the number of replications already performed on system i. Notice that we do not require equal rst-stage sample sizes, since a search procedure may revisit solutions or take diering numbers of observations from them. Further, let Xij be the output from replication j of system i, which we assume are i.i.d. N(i; i2 ) random variables. Assuming normality is often reasonable when each replication output variable Xij is the average of a large number of more basic output variables; it may also be reasonable when Xij is a batch mean in a steady-state simulation, but we do not address that situation here. Systems are to be compared based on their true means, i, and we assume that larger i is better throughout this paper. The rst-stage sample mean of system i is

Xi(1) = n1

n0i X

0i j =1

Xij

while S02i is the rst-stage sample variance of system i; that is

S02i

1

= n0i ? 1

n0i  X j =1

2 Xij ? Xi(1) . 

Finally, let Sri2 be the sample variance of system i based on a second, independent sample of size nri. 3

2.1 Overview of Basic Methods 2.1.1 Screening

In many cases there will be systems visited by the search that are clearly inferior to others visited by the search. We will use a subset-selection procedure to screen out these clearly inferior systems. A subset-selection procedure returns a subset (whose size can be random or predetermined) that contains the best of the k systems with probability  1 ? . We propose the following procedure when the best system is the one with the largest true mean:

Screen-to-the-Best Procedure 1. Sample Xij ; i = 1; 2; : : : ; k; j = 1; 2; : : : ; n0i , where the Xij are i.i.d. N(i; i2) random variables. 2. Let

!1

2S 2 2 2S 2 t t i 0 i Wi` = n + `n 0` ; 8i 6= ` 0i 0` where ti = t(1?) ?1 1 ;n ?1 and t ; is the quantile of the t distribution with  0 degrees of freedom. k

i

n

o

3. Set I = i : 1  i  k and Xi(1)  X`(1) ? Wi`; 8` 6= i . 4. Return I as the subset of retained systems. In a single-stage subset selection procedure, such as this one, the number of systems included in the subset is random. If one is fortunate, the subset includes only a single system, the best. If one is unfortunate, the subset includes all k systems, and the procedure has not reduced the eld. Nelson et al. (1998) developed a single-stage subset selection procedure that permits unequal and unknown variances. Our Screen-to-the-Best Procedure extends their's to allow the unequal sample sizes that may be the result of a search. 4

A complete description of a slightly more general version of this procedure, and a proof of its validity, are included in the Appendix.

2.1.2 Selection To choose the best system from among those systems that are not obviously inferior, we will employ a two-stage indierence-zone ranking (IZ) procedure, which requires additional sampling of the competitive systems. Two-stage IZ procedures guarantee to select the best system with probability  1 ?  whenever the best is at least a user-speci ed amount, , better than the others. If there are some near-best solutions within  of the best, most two-stage IZ procedures will return the best or one of these near-best solutions. The user-speci ed quantity, , is called the indierence zone, and it represents the smallest dierence worth detecting. In a typical IZ procedure, such as Rinott's (1978) procedure, the total sample size required of system i is: 8
0 and Zj  N (0; 1), Pr fZj  Qj g = 1 + Pr f0  Zj  Qj g 2 n o = 1 + 1 Pr Zj2  Q2j 2 2 ( 0 1 2 2 n  S 1 1 = + Pr Zj2  t2[k] [2k] @ 2 [j] [k] 2 A 2 2 ] n[j]] + n[k]] 0 1) 2 2 S n  [ k ] + t2[j] [2j] @ 2 [j] 2 A ] n[j]] + n[k]]     12 + 21 1 ? 2 1 ? (1 ? 0)1=(k?1) = (1 ? 0 )1=(k?1) where the inequality in (8) follows from Lemma 1 with

n[j]] !

1 = 1 ? 2 = n 2 + n 2 : [j ] [k] [k] [j ] Substituting this result into (7) shows that PrfCS g 

kY ?1 j ?1

(1 ? 0 )1=(k?1) = 1 ? 0:

30

(8)

7.2 Extended Rinott Procedure In his 1978 paper, Rinott showed that if rst-stage samples are all of size n0 , then kY ?1

8 > > >
> = C> C 1=2 C> A> > ;

0

h 1? (9) n ? 1 n ? 1 i=1 > > 0 0 : Yi(n0 ? 1) + Yk (n0 ? 1) where Yi(df ) are independent 2df random variables for i = 1; 2; : : : ; k, and h = h(2; (1 ? )1=(k?1) ; n0). Following steps analogous to Rinott, we can show that under our modi ed procedure, where each system i may have a dierent initial sample size n0i  2, PrfCS g >

PrfCS g >

B

B E > B @

kY ?1

8 > > >
> = C> C C 1=2 > A> > ;

(10) n0i ? 1 + n0k ? 1 Yi(n0i ? 1) Yk (n0k ? 1) where n0k is the initial number of replications performed on the best system. Recall that in our procedure h = h(2; (1 ? )1=(k?1) ; nmin) where nmin = mini fn0i g. To prove the validity of our procedure, we need to show that (10)  1 ? . Before we can show this, however, we need to prove a lemma that depends upon the following de nitions and theorems from Shaked and Shanthikumar (1994). Let X and Y be random variables. i=1

B E > B @ > > :

De nition: If E[(X )]  E[(Y )] for all convex [concave] functions  : < ! > =

> > Zy >
?1  r?1  dx: > > 22? 2 > > 2 ?( ?2 1 ) > : ; r

r

r

r

2

Since the function outside of the brackets is a chi-squared density, it is always positive. The term inside the brackets is positive at x = 0, is decreasing for x > 0, and must eventually fall below zero. As a result, F1 ? G1 has just one crossover point and the direction is [+; ?]. 33

Let F2 be the c.d.f. of (r ? 1)=X and G2 be the c.d.f. of r=Y . The expression F2 ? G2 has the same number of crossover points as F1 ? G1 (one), but in the opposite direction, [?; +]. Finally, let F3 be the c.d.f. of a + (r ? 1)=X and G3 be the c.d.f. of a + r=Y . The expression F3 ? G3 has the same number of crossover points as F2 ? G2 (one), and in the same direction, [?; +]. Combining this fact with (15) satis es the conditions of Theorem 3, so we get ? 1: (16) a + Yr icv a + r X Thus, since the function ?(b)?1=2 is concave increasing for all b > 0, !

!

?1=2  ?1=2 r ? 1 r E ? a+ X E ? a+ Y by (16) and the de nition of increasing concave order. Therefore  ?1=2 !  ?1=2 ! r r ? 1 E a+ E a+ X Y and for any constant h > 0  ?1=2 !  ?1=2 ! r r ? 1 hE a + Y  hE a + X which can be rewritten as 

0

0

1

1

C B C B h h C B C : (17) B EB 1=2 C  E B  1=2 C A @ A @ r r ? 1 a+ Y a+ X We now provide a sequence of arguments that result in establishing that Theorem 3 applies to the random variables who expected values are compared in (17): Recall that if F2 is the c.d.f. of (r ?1)=X and G2 is the c.d.f. of r=Y , then the expression F2 ? G2 has one crossover point with direction, [?; +]. Therefore, the expression G2 ? F2 has one crossover point with direction, [+; ?]. Let F4 be the c.d.f. of (a + (r ? 1)=X )1=2 , and G4 be the c.d.f. of (a + r=Y )1=2 . The expression G4 ? F4 has the same number of crossover points as G2 ? F2 (one), and has the same direction, [+; ?].

34

Next let F5 be the c.d.f. of (a + (r ? 1)=X )?1=2 , and G5 be the c.d.f. of (a + r=Y )?1=2. The expression G5 ? F5 has the same number of crossover points as G4 ? F4 (one), but has opposite direction, [?; +]. Finally, let F6 be the c.d.f. of h= (a + (r ? 1)=X )1=2 , and G6 be the c.d.f. of h= (a + r=Y )1=2 . The expression G6 ? F6 has the same number of crossover points as G5 ? F5 (one), and has the same direction, [?; +]. Thus, because of the number and direction of the crossover points, and because of (17), Theorem 3 yields h h 1=2 icv   1=2 : r r ? 1 a+ Y a+ X Now since , the c.d.f. of the standard normal distribution, is concave increasing on the interval (0; 1), the de nition of increasing concave ordering gives the desired result 0

0

B

B

0

11

B B B  EB @ @

11

0

CC B B CC h h CC B B CC C: B B  C  E C 1=2 1=2 C AA @ @ AA r r ? 1 a+ Y a+ X

Now we are in a position to show (10)  1 ? . Let 0

h

B

Ri =  B B @

(18)

1 C C 1=2 C A

: n0i ? 1 + n0k ? 1 Yi(n0i ? 1) Yk (n0k ? 1) Notice that each Ri is a non-negative, real-valued function of Y1; Y2; : : : ; Yk , and is nondecreasing in each of its arguments. Consider an individual E[Ri ] term in the product (10) and condition on Yk . Then 8 > > >
> >
> :

Lemma 3 tells us that (19) is 8 > > >
> >
> :

> :

i

> :

n0i ? 1 n0k ? 1 Yi(n0i ? 1) + Yk (n0k ? 1)

0

h

B

 EY >>EY >> BB@  k

h

B

B E[Ri ] = EY >EY > B @

n0k ? 1 n0i ? 2 Yi(n0i ? 2) + Yk (n0k ? 1) 35

199 > > > > >> C= = C C 1=2 A> > >> > ; ;>

199 > >> > C> => = C 1=2 C>> A>> > ;> ;

:

(19)

(20)

We can continue this process of reducing the degrees of freedom of Yi to show that (20) is 8 > > >
> >
>EY >> BB@  k

> :

i

> :

h nmin ? 1 n0k ? 1 Yi(nmin ? 1) + Yk (n0k ? 1)

199 > > >> C> = => C 1=2 C>> A>> > ; ;>

Applying this process for each i 6= k we obtain 8 > > >
> >
> >
> > > >> C= = C C 1=2 A> > >> > ; ;>

h : (22) > > n ? 1 n ? 1 > > i6=k i6=k 0k min : : Yi(nmin ? 1) + Yk (n0k ? 1) To complete the proof, we condition instead on Yi and iteratively reduce the degrees of freedom of Yk down to nmin ? 1 in exactly the same fashion, giving Y

E[Ri] 

Y

i6=k

Y

EY

k

E[Ri ] 



B

B E B > Y > @ i

Y

i6=k Y

i6=k

B

B E > B @ > > :

h nmin ? 1 nmin ? 1 Yi(nmin ? 1) + Yk (nmin ? 1)

19 > > C> = C C 1=2 > A> > ;

(1 ? )1=(k?1) = 1 ? 

where the last inequality follows because of the way we choose h.

Remark: Rinott also proved (9) for h0 = h(k; (1 ? ); n0), which is slightly smaller than our value, h = h(2; (1 ? )1=(k?1) ; n0). While we conjecture that (10) is true for this smaller h0, we were unable to prove it. As Figure 3 illustrates, however, the dierence between these values is small, about 2%{3% for k  100.

7.3 Validity of Screen, Restart and Select Theorem 4 For the procedure described in Section 3, PrfCS g  1 ?  whenever [k] ? [k?1]  . Proof: The Screen, Restart and Select procedure will attain a correct selection if the true best system survives the screen and has the largest overall sample mean of the survivors. Let X[(in] ) be the nth -stage sample mean of the system whose true mean is the ith smallest 36

h(2)/h(k) 1.0001.0051.0101.0151.0201.025

1-alpha = 0.90

0

20

40

60

80

100

60

80

100

60

80

100

k

h(2)/h(k) 1.000 1.005 1.010 1.015 1.020

1-alpha = 0.95

0

20

40 k

1.000

1.005

h(2)/h(k) 1.010 1.015

1.020

1-alpha = 0.99

0

20

40 k

Figure 3: The ratio h(2; (1 ? )1=(k?1) ; n0)=h(k; (1 ? ); n0) vs. k for n0 = 10 and selected values of 1 ? .

37

(thus, [k] is the index of the best system). Based on the properties of the screening procedure, we know that the true best system will survive the screen with a probability p greater than or equal to 1 ? . That is,

p  (1) Prfk survives screeningg = Pr X[(1) k]  X[i] ? W[i][k]; 8i 6= k  1 ? : n

o

Because the procedure takes new rst-stage samples between the screening and selection stages, we know that if the true best system survives the screen, it will have the p largest overall sample mean with a probability greater than or equal to 1 ?  (assuming [k] ? [k?1]  ). That is,

p  (2) Pr X[(2) k]  X[j ] ; 8j 6= k k survives screening  1 ? :





Therefore, n

o

 (1) PrfCS g = Pr X[(1) k]  X[i] ? W[i][k]; 8i 6= k    (2) (2)   Pr X[k]  X[j] ; 8j 6= k k survives screening



p

p

1? 1?

= 1 ? :

7.4 Selecting [nlow, nhigh] to start the Golden-Section method Recall that in the single-system case, Ni is minimized at the point where 2 h (n ) Sri : nr =  !

r

2 = min S 2 and S 2 = max S 2 . If we assume that S 2 = S 2 , meaning that the Let Slow i 0i i 0i high ri 0i sample variance of the initial sample and the restart sample will be the same, then no system will have a minimizing nr larger than (h(n ) Shigh=)2, and no system will have a r

38

minimizing nr lower than (h(n ) Slow =)2. To set the endpoints of the starting interval, we nd the nr 's that satisfy r

nlow = h(nlow)Slow

!2

2 h S and nhigh = (nhigh) high : !

The minimum will fall within this interval because for any value of nr outside of the interval, moving a small distance  toward the interval will reduce each Ni, thus reducing the total.

7.5 Screening by Non-Competitive (`Dead') Systems In the original Group-Screening procedure of Nelson et al. (1998), the authors argue that if the true best system, [k], survives screening with probability  1 ? 0, then the entire procedure has PrfCS g  1 ? (0 + 1). Our Modi ed Group-Screening Procedure (Section 4) diers from the original in that each system that is screened must face screening by all systems in all previous groups, not just the surviving systems from previous groups; this provides a slightly tighter screen. We also allow unequal initial sample sizes. If we can prove that the true best system survives screening with probability  1 ? 0 despite these changes, then we can claim that the entire procedure has PrfCS g  1 ? (0 + 1 ) using the same reasoning as Nelson et al. (1998).

Proof: Let G1; G2 ; : : : ; Gp be groups of systems such that G1 [G2 [


[Gp = f1; 2; : : : ; kg, Gi \ Gj = ; for i 6= j , and jGij  1 for all i. At the `th step in the experiment, the systems in G` will be screened with respect to each other and all systems from previous groups. Let ` be the index such that [k] 2 G` , and de ne F = G1 [ G2 [


[ G`?1 (with F = ; if ` = 1). Furthermore, de ne I as all systems from F that survived screening, and J = F n I , the systems in F that did not survive screening. Let E denote the event \system [k] survives screening." Formally, 

 (1)  (2)  (1) E = X[(1) k]  X[j ] ? W[k]j ; 8j 2 J [ G` and X[k]  X[j ] ? W[k]j ; 8j 2 I 39



where each j 6= [k] that is encountered before [k] is either in J [ G` or in I , but not both. Under our procedure, [k] has a better chance of surviving if its group, G` , is encountered earlier, rather than later, in the search procedure. That is,



PrfE ` < pg  PrfE ` = pg: The reason is that when ` = p, [k] faces exactly the same screens it would face if ` < p, plus some more, as long as the allocation of systems to groups and the relative ordering of non-G` groups remains unchanged. Thus, if we can prove that PrfE ` = pg  1 ? 0 over all allocations of systems to groups, then PrfEg  1 ? 0 .

PrfE ` = pg ( (1) X[k] ? Xj(1) ?W[k]j  (1;1) ; 8j 2 J [ G` and  Pr vj(1;1) vj ) (1) (2) X[k] ? Xj ? W [k]j  (1;2) ; 8j 2 I vj(1;2) vj (  = Pr Zj(1;1)  Q(1j ;1) ? (1[k;]1)j ; 8j 2 J [ G` and vj )  [k]j (1;2) (1;2) Zj  Qj ? (1;2) ; 8j 2 I vj

(23)

where 1

1

] j2 ! 2 ] j2 ! 2 (1;2) (1;1) vj = n + N vj = n + n 0[k] 0j 0[k] j X[(ka]) ? Xj(b) ? [k]j [k]j ( a;b ) Zj = and Qj(a;b) = W(a;b : (a;b) vj vj ) Now, by symmetry of the normal distribution, and the fact that [k]j  0, we can rewrite (23) as n

o

Pr fEj` = pg  Pr Zj(1;1)  Q(1j ;1) ; 8j 2 J [ G` and Zj(1;2)  Q(1j ;2) ; 8j 2 I : The proof proceeds by doing the following steps: 40

(24)

Step 1. Condition on S102 ; : : : ; Sk20 and use Slepian's inequality to break (24) into the product of two probabilities.

Step 2. Use Slepian's inequality and Lemma 2 on each piece to break any joint proba-

bilities into products of marginal probabilities for the Zj(a;b) . Altogether there will be k ? 1 terms in the product.

Step 3. Show that each marginal probability is  (1 ? 0) ? using Banerjee's lemma. 1

k 1

Step 1. In order to apply Slepian's inequality, we must show that, conditional on S102 ; : : : ; Sk20 and j = 6 ` 6= [k]:  (1)  (1)  (2) Cov[X[(1) k] ? Xj ; X[k] ? X[`] ]  0: This is trivially true since Cov = Var[X[(1) k] ]  0. Therefore, by Slepian's inequality, still conditional on S102 ; : : : ; Sk20, (24) is greater than or equal to h

n

o

oi

n

E Pr Zj(1;1)  Q(1j ;1) ; 8j 2 J [ G` Pr Zj(1;2)  Q(1j ;2) ; 8j 2 I :

(25)

Step 2. Within each of the two probability statements above, it is easy to show that the Cov[Zj(a;b) ; Z`(a;b) ] = Var[X[(ka]) ]=(vj(a;b) v`(a;b) )  0, so we can do a further application of Slepian's inequality to show that (25) is greater than or equal to 2

E4

n

Y

j 2J [G`

(1;1) o Y

Pr Zj(1;1)  Qj

j 2I

3

o

n

Pr Zj(1;2)  Q(1j ;2) 5 :

(26)

Since each of these probabilities is an increasing function of Qj(a;b) , which is in turn an increasing function of the Sj20, Lemma 2 shows that (26) is greater than or equal to Y

j 2J [G`

h

n

E Pr Zj(1;1)  Q(1j ;1)

oi Y

j 2I

h

oi

n

E Pr Zj(1;2)  Q(1j ;2) :

Step 3. All that remains is to show that h

oi

n

E Pr Zj(a;b)  Qj(a;b)  (1 ? 0) 41

1

?

k 1

:

This was done for the case (a; b) = (1; 1) in the proof of the Screen-to-the-Best Procedure with unequal sample sizes. The proof for the (1; 2) case follows exactly the same steps. Critical to the proof is that Wij is based only on the rst-stage sample variances, Si20, because Si20 is independent of both the rst- and second-stage sample means.

7.6 Modi ed Group-Screening with Sorting Below we will show that under the Modi ed Group-Screening Procedure, sorting by rststage sample mean prior to screening does not alter the PrfCS g guarantee, provided the group size is one.

Proof: Let set J include only those systems encountered before [k] that fail screening, and let set I include only those systems encountered before [k] that survive screening. Let E denote the event \system [k] survives screening when no sorting is done." More formally, 



 (1)  (1)  (2) E = X[(1) k]  Xj ? W[k]j ; 8j 2 J and X[k]  Xj ? W[k]j ; 8j 2 I : For a xed value of X[(1) k] , system [k] has a better chance of survival if it is encountered earlier, rather than later, in the screening process. That is, n

o

Pr E X[(1) k] = x and [k] is not encountered last n o  Pr E X[(1) k] = x and [k] is encountered last

(27)

The assertion above is obvious when group size is equal to one, because if [k] is encountered last, it faces exactly the same screens it would face if it were not last, plus some more. o n We know that without sorting Pr E [k] is encountered last  1 ? . However, it is unlikely that this statement is true when sorting is employed, because [k] being encountered last under sorting implies that X[(1) k] has a relatively low value. Of course, under sorting, [k] will not always be encountered last. To prove that sorting is valid, we 42

will take an expectation over all possible values of X[(1) k] , combining the monotonicity of n o position showed in (27) with the fact that Pr E [k] is encountered last  1 ? . Let f (
) be the p.d.f. of the rst-stage sample mean from system [k]. For any outcome n o  (1) of X[1](1) ; X[2](1) ; : : : ; X [(1) k?1] , X[k] = x implies a position (rank) for [k] under sorting. Let E 0 be the same as E when sorting is employed. Then Pr fE 0g

=

Z1 X k 

?1 i=1

n o Pr E 0 X[(1) = x and [ k ] has rank i k]

n

o



 Pr [k] has rank i X[(1) k] = x f (x)dx



Z1

n

o

Pr E X[(1) k] = x and [k] is encountered last f (x)dx

?1 n



= Pr E [k] is encountered last when sorting is not employed

o

 1 ? : The inequality holds because for any rank of [k], the probability of surviving screening is not increased by forcing [k] to be encountered last, and if [k] is always encountered last then this is equivalent to the event E without sorting.

7.7 Lower Bound on PCS Under Choice Procedure Suppose that, after screening, we use the rst-stage data in conjunction with some decision rule to determine which sub-procedure, Restart or Screen-and-Continue, to pursue. Below, we will prove that under this choice procedure PrfCS g  (1 ? 0)  (1 ? 21), where 1 ? 0 is the con dence level used in the screening phase, and 1 ? 1 is the con dence level used in the selection phase. For notation, let B be the event that the best system survives screening, while DR is the event that the decision rule|whatever it is|favors restart. A \bar" over an event indicates its complement. Let the subscript C indicate probabilities computed under the assumption that we always continue, while R indicates probabilities computed under the assumption that we always restart. 43

Using this notation, we can write the probability of a correct selection under the choice procedure, given that the best system survives screening, as PrfCS jB g = Pr fCS jB; D Rg PrfD R g + Pr fCS jB; DRg PrfDR g: C R

(28)

First, we will nd a lower bound on PrC fCS jB; D Rg PrfD Rg. We know from Nelson et al. (1998) that the probability of selecting the best in the Screen-and-Continue procedure, given that the true best survives screening, is greater than 1 ? 1. In our notation, Pr fCS jB g  1 ? 1: C Conditioning on the outcome of the decision rule yields Pr fCS jB g = Pr fCS jB; DRg PrfDR g + Pr fCS jB; D Rg PrfD R g  1 ? 1: C C C

(29)

Therefore, Pr fCS jB; D Rg PrfD Rg  1 ? 1 ? Pr fCS jB; DRg PrfDR g C C

 1 ? 1 ? PrfDRg:

(30)

We know that under restart, if the best survives screening, the probability of success is greater than or equal to 1 ? 1, regardless of the outcome of the decision rule. As a result, PrR fCS jB; DRg  1 ? 1. Combining this result with (30) yields, PrfCS jB g = Pr fCS jB; D Rg PrfD Rg + Pr fCS jB; DRg PrfDR g C R

 1 ? 1 ? PrfDR g + (1 ? 1 ) PrfDR g = 1 ? 1 ? 1 PrfDR g  1 ? 21:

Consequently, the overall choice procedure (screening and selection phases) yields PrfCS g  (1 ? 0)  (1 ? 21 ) 44

which is equal to (1 ? =2)  (1 ? ) if 0 = 1 = =2. As a result, PrfCS g  1 ? 3=2 + 2=2  1 ? 3=2:

References Banerjee, S. 1961. On con dence interval for two-means problem based on separate estimates of variances and tabulated values of t-table. Sankhya, A23:359{378. Bazaraa, M.S., H.D. Sherali, and C.M. Shetty. 1993. Nonlinear programming: theory and algorithms. New York: John Wiley & Sons. Boesel, J., B.L. Nelson, and N. Ishii. 1999. A Framework for Simulation-Optimization Software. Technical Report, Department of Industrial Engineering and Management Sciences, Northwestern University. Nelson, B.L., J. Swann, D. Goldsman, and W. Song. 1998. Simple procedures for selecting the best simulated system when the number of alternatives is large. Technical Report, Department of Industrial Engineering and Management Sciences, Northwestern University. Rinott, Y. 1978. On two-stage selection procedures and related probability-inequalities. Communications in Statistics|Theory and Methods A7:799{811. Shaked, M. and J.G. Shanthikumar. 1994. Stochastic orders and their applications. San Diego: Academic Press. Tamhane, A. C. 1977. Multiple comparisons in model I one-way anova with unequal variances. Communications in Statistics, A6:15{32. Tong, Y. L. 1980. Probability inequalities in multivariate distributions. New York: Academic Press.

45