Discrete-Event Simulation Optimization Using ...

Discrete-Event Simulation Optimization Using Ranking, Selection, and Multiple Comparison Procedures: A Survey JAMES R. SWISHER Mary Washington Hospital SHELDON H. JACOBSON University of Illinois and ¨ ENVER YUCESAN INSEAD

An important use for discrete-event simulation models lies in comparing and contrasting competing design alternatives without incurring any physical costs. This article presents a survey of the literature for two widely used classes of statistical methods for selecting the best design from among a finite set of k alternatives: ranking and selection (R&S) and multiple comparison procedures (MCPs). A comprehensive survey of each topic is presented along with a summary of recent unified R&S-MCP approaches. Procedures are recommended based on their statistical efficiency and ease of application; guidelines for procedure application are offered. Categories and Subject Descriptors: A.1 [Introductory and Survey]; I.6.6 [Simulation and Modeling]: Simulation Output Analysis General Terms: Algorithms; Experimentation Additional Key Words and Phrases: Simulation optimization, ranking and selection, multiple comparisons

1. INTRODUCTION The most common goal of discrete-event simulation studies is the selection of the best system design from among a set of competing alternatives. Simulation S. H. Jacobson was supported in part by the Air Force Office of Scientific Research under grant F49620-01-0-0007 and by the National Science Foundation under grant DMI-9907980. Authors’ addresses: J. R. Swisher, Mary Washington Hospital, 1001 Sam Perry Boulevard, Fredericksburg, VA 22401; email: [email protected]; S. H. Jacobson, Department of Mechanical and Industrial Engineering, University of Illinois, Urbana, IL 61801; email: [email protected]; ¨ email: E. Yucesan, Technology Management Area, INSEAD, 77305 Fountainbleau, France; email: [email protected]. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or direct commercial advantage and that copies show this notice on the first page or initial screen of a display along with the full citation. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, to redistribute to lists, or to use any component of this work in other works requires prior specific permission and/or a fee. Permissions may be requested from Publications Dept., ACM, Inc., 1515 Broadway, New York, NY 10036 USA, fax: +1 (212) 869-0481, or [email protected].

C 2003 ACM 1049-3301/03/0400-0134 $5.00 ACM Transactions on Modeling and Computer Simulation, Vol. 13, No. 2, April 2003, Pages 134–154.

Discrete-Event Simulation Optimization

•

135

optimization is a structured approach to determine optimal settings for input parameters (i.e., the best system design), where optimality is measured by a (steady-state or transient) function of output variables associated with a simulation model [Swisher et al. 2004]. In the 1970s and 1980s, most simulation optimization techniques were path search based, involving some type of gradient estimation technique (e.g., response surface methodology, perturbation analysis) imbedded in a stochastic approximation algorithm [Jacobson and Schruben 1989]. Over the last decade, however, two statistical techniques, ranking and selection (R&S) and multiple comparison procedures (MCPs), have gained popularity in simulation optimization. These techniques are applicable when the input parameters are discrete and the number of designs to be compared is both discrete and relatively small (traditional techniques limit selection from among a maximum of approximately 20 alternatives, while some recent techniques allow up to several thousand alternatives). The particular method that is applicable is dependent upon the type of comparison desired by the analyst and the properties of the simulation output data. Jacobson and Schruben [1989] and Fu [1994a, 1994b] contain extensive reviews of simulation optimization techniques, with a focus on continuous parameter optimization. Ho et al. [1992] discuss ordinal optimization and how it can improve the efficiency of simulation optimization. Swisher et al. [2004] present a general review of discrete simulation optimization techniques that includes, but is not limited to, R&S and MCPs. R&S procedures are statistical methods specifically developed to select the best system or a subset that contains the best system design from a set of k competing alternatives [Goldsman and Nelson 1994]. In general, these methods assure the probability of a correct selection at or above some user-specified level. MCPs specify the use of certain pairwise comparisons to make inferences in the form of confidence intervals about relationships among all designs [Fu 1994a]. In short, R&S provides the experimenter with the best system design while MCPs provide information about the relationships among the designs (e.g., how much better the best design is in comparison to the alternatives). R&S and MCPs are particularly well suited for computer simulation. In particular, these procedures are statistically valid in the context of simulation as the underlying assumptions of normality and independence of observations can be easily achieved through appropriately batched output data or sample averages of independent replications, and through adequate assignment of the pseudo-random number generator seeds, respectively. Furthermore, common random numbers, a variance reduction technique widely used in simulation, can be used to enhance the efficiency of some of the procedures. The following notation will be used throughout this discussion: Let Y ij represent the j th simulation output (such as a replication or a batch mean, after accounting for initialization effects, where a batch mean is the average taken over a set (batch) of simulation output data values) from the ith design alternative, for i = 1, 2, . . . , k and j = 1, 2, . . . , n. Let µi = E[Y ij ] and σi2 = Var[Y ij ] denote the expected value and variance, respectively, of an output from the ith design alternative. Let µ[1] ≤ µ[2] ≤ · · · ≤ µ[k] denote the ordered but unknown expected values for the outputs of the k alternatives. Let Y j = (Y 1 j , Y 2 j , . . . , Y k j )’ be the k × 1 vector of outputs across all design alternatives for output j , where ACM Transactions on Modeling and Computer Simulation, Vol. 13, No. 2, April 2003.

136

•

J. R. Swisher et al. Table I. Key Dates and Contributions in R&S and MCPs

Date 1953 1954 1955 1956 1975

Author(s) Tukey Bechhofer Dunnett Gupta Dudewicz & Dalal

1978 1984 1984 1985 1989 1991

Dudewicz & Taneja Gupta & Hsu Hsu Koenig & Law Sullivan & Wilson Yang & Nelson

1993

Matejcik & Nelson

1994 1995

Goldsman & Nelson Nelson & Matejcik

1996

Chen et al.

2001b 2001

Chick and Inoue Nelson et al.

Contribution Origin of All-pairwise Multiple Comparisons Origin of Indifference-zone R&S Procedures Origin of Multiple Comparisons with a Control Origin of Subset Selection R&S Procedures Elimination of variance constraints for Indifference-Zone R&S Multivariate R&S formulation First reference to R&S, MCP unification Origin of Multiple Comparisons with the Best Extension of Indifference-zone R&S Procedures Elimination of variance constraints for Subset Selection R&S Control Variates and Common Random Numbers for Allpairwise Multiple Comparisons, Multiple Comparisons with the Best, and Multiple Comparisons with a Control Establishes connection between Indifference-zone R&S and Multiple Comparisons with the Best Unification of R&S and MCPs perspectives Procedures for simultaneous R&S and Multiple Comparisons with the Best Heuristic for fully sequential ranking and selection with an optimal computing budget allocation scheme Decision-theoretic approach to ranking and selection Combination of screening (subset selection) and indifferencezone ranking approaches

Y1 , Y2 , . . . are assumed to be independent and identically distributed (i.i.d.) with multivariate normal distribution Y j ∼ N (µ, 6) where µ = (µ1 , µ2 , . . . , µk )’ is the unknown mean vector and 6 is the unknown variance-covariance matrix. The subscript “•” denotes averaging with respect to that subscript. For example, the average design alternative output Pperformance measure value across all replications (or batch means) is Y¯ i· = nj=1 Y i j /n. Two popular variance reduction techniques, common random numbers (CRN) and control variates, are also referred to extensively during this discussion. CRN or correlated sampling [Banks et al. 1996] takes advantage of the same set of random numbers across all design alternatives for a given replication or batch mean. CRN is typically designed to induce positive correlation among the outputs of each respective design alternative for a given replication, thereby reducing the variance of the difference between the mean design alternative point estimators. Likewise, control variates attempt to reduce the variance in simulation outputs of the different design alternatives for a particular replication or batch mean. Control variates depend on knowledge of a second random variable, X , with known expectation, that also is correlated with the output of interest, Y [Law and Kelton 2000]. An experimenter is then able to define a new estimator for E[Y ] that involves X - E[X ] and has smaller variance than a direct estimator for E[Y ] that only involves Y. Table I displays a historical overview of the most significant contributions in the development of R&S and MCPs and their application to discrete-event ACM Transactions on Modeling and Computer Simulation, Vol. 13, No. 2, April 2003.

Discrete-Event Simulation Optimization

•

137

simulation. Details of the articles listed in Table I can be found in the body of this article in the following sections: Section 2 addresses R&S procedures (indifference-zone, subset selection, and other approaches). Section 3 covers MCPs (combined paired-t, Bonferroni, all-pairwise comparisons; all-pairwise multiple comparisons; multiple comparisons with a control; and multiple comparisons with the best). Section 4 discusses insights into R&S-MCP unification, including combined R&S-MCP techniques. Conclusions and future directions for R&S and MCPs are reported in Section 5. 2. RANKING AND SELECTION (R&S) Ranking and selection is a commonly prescribed method for selecting the best system from among a set of competing alternatives. Bechhofer [1954] proposed the fundamental concepts employed in R&S procedures. The majority of work in R&S can be classified into two general approaches: indifference-zone ranking and subset selection. Indifference-zone ranking is treated in Section 2.1, while Section 2.2 treats subset selection approaches. Section 2.3 treats those R&S procedures that fall outside of the two general approaches. A general overview of R&S can be found in many places. Law and Kelton [2000] contains an excellent introduction to the topic with corresponding references to more mathematically intense treatments. Likewise, Sanchez [1997] gives an overview of R&S with several sample scenarios and an extensive list of references. Goldsman [1983] provides a good perspective of R&S as it stood in relation to simulation in the early 1980s. Goldsman and Nelson [1994, 1998] contain comprehensive state-of-the-art reviews of R&S in simulation. Wherever possible, they attempt to unify R&S and MCP perspectives. Goldsman et al. [2002] describe insights into when R&S procedures can be most effectively adapted for use with steady-state simulations (i.e., simulation models that can be run for a sufficiently long time such that the initial conditions no longer affect the simulation output and the output distribution is stationary). They also give guidelines on how to evaluate such procedures. Fu [1994a, 1994b] provides an overview of R&S as it relates to the larger domain of simulation optimization. Kleijnen [1977] discusses R&S in the context of other useful statistical tools in simulation. Bechhofer et al. [1995] is a thorough text on R&S that includes practical hints for practitioners. Carson [1996] describes how AutoStat, an add-on package for the simulation software AutoMod, allows users to easily implement R&S procedures. 2.1 Indifference-Zone Procedures The concept of R&S was first introduced by describing a problem where the goal is to select the best population (i.e., the population with the largest mean for some population statistic from a set of k normal populations). Typically, an experimenter will collect a certain number of observations from each population (Y ij ) and select the best population using statistics based on these observations. Since the observations are realizations of random variables, it is possible that the experimenter will not select the best population. However, if the best ACM Transactions on Modeling and Computer Simulation, Vol. 13, No. 2, April 2003.

138

•

J. R. Swisher et al.

population is in fact selected, the experimenter is then said to have made the correct selection (CS). In addition, an experimenter may be indifferent (at some level) in the selection of a population when the populations are nearly the same. That is, if µ[k] – µ[k−1] is very small, then the experimenter may view the two best populations as essentially the same and not have a preference between the two. To quantify this, define δas the indifference-zone parameter. If µ[k] − µ[k−1] < δ, the experimenter is said to be indifferent to choosing the system associated with µ[k] or µ[k−1] . Define the probability of correct selection as P{CS} = P{µ[k] > µ[i] , for all i 6= k|µ[k] − µ[i] ≥ δ} ≥ P∗ where {δ, P∗ } are pre-specified by the experimenter. Note that 1/k < P∗ < 1 is required since P{CS} = 1/k could be achieved by simply choosing a population at random. The original indifference-zone R&S procedure proposed by Bechhofer [1954] assumes unknown means, µ1 , µ2 , . . . , µk , and known common variance, σ 2 , for all k populations (i = 1, 2, . . . , k). This procedure is a single-stage procedure; hence, the total number of observations required (N ) is determined a priori by the experimenter’s choice of {δ, P∗ }. Specifically, N = d(ck, P ∗ σ/δ)2 e, where ck, P ∗ is a constant obtained from a table (e.g., see Bechhofer [1954]) and dxe is the ceiling function of x. After taking N independent observations from each of the k populations, the population with the largest sample mean is selected as the best. Indifference-zone R&S procedures need not be single-stage. Indeed, following Bechhofer [1954], several articles approach the problem as a multistage problem, where the experimenter collects a user-specified number of observations, checks certain stopping criteria, and then either continues sampling or stops and selects the best system. Paulson [1964] and Bechhofer et al. [1968] describe such methodologies. The major disadvantage of these approaches is the continued requirement for common, known variance among populations. This assumption is hard to justify for discrete-event simulations. When a simulation analyst is modeling a system that does not physically exist, it is often impossible to know the system output’s variance. In addition, modeling an existing system may still not allow the analyst to know its output’s variance because of the potentially high cost or practical infeasibility of data collection. Moreover, even when the variance is known, ensuring common system output variance across different system designs may be difficult. For these reasons, modern indifference-zone R&S procedures typically require neither equal nor known variances. Although a small number of early articles (e.g., Zinger and St. Pierre [1958] and Zinger [1961]) describe procedures for unequal, but known variances, Dudewicz and Dalal [1975] were the first to present an indifference-zone methodology requiring neither equal nor known variances. Their research represented a major step forward in R&S methodology, making the application of indifference-zone techniques particularly suitable to discrete-event simulation. Dudewicz and Dalal describe a two-stage procedure in which the experimenter chooses δ, P∗ , and n0 , where n0 is the number of observations (i.e., batched or replicated means) to be collected during the first stage of the procedure. The first-stage variances are then used to determine the number of observations required in the second stage. A weighted average of the first and second-stage ACM Transactions on Modeling and Computer Simulation, Vol. 13, No. 2, April 2003.

Discrete-Event Simulation Optimization

•

139

sample means is then used to select the best system (i.e., the system with the largest weighted average). Dudewicz [1976] presents the same procedure, with applications to simulation. Rinott [1978] describes a modification to the second stage of Dudewicz and Dalal [1975] that in some cases yields a greater P{CS}, but may require a larger total number of observations. Goldsman et al. [2002] report an extension of this procedure to steady-state simulation. Indifference-zone procedures are constructed based on the least-favorable configuration assumption that the true mean of the best design is exactly δ better than all other alternative designs and that all other designs are tied for the second best. While this assumption facilitates the determination of the required number of observations from each competing design, Ni , by making the computation independent of the true or sample means, it also makes it as hard as possible to distinguish the best design from the competing alternatives. As a consequence, indifference zone procedures may prescribe more observations than necessary to achieve the desired probability of correct selection. This is particularly inefficient in situations where the performance of alternative designs is drastically different. Since indifference zone procedures do not use sample mean information, the opportunity of identifying clearly inferior designs and excluding them from further consideration is lost. There have been various attempts to enhance the efficiency of these procedures. Koenig and Law [1985] extend the indifference-zone approach for use as a screening procedure. They outline a method for selecting a subset of size m (user-specified) of the k systems so that with probability at least P∗ , the selected subset will contain a system whose mean is within an indifference zone, δ, from the best system. This procedure requires only the use of a different tabulated constant when computing the second stage sample size as compared to Dudewicz and Dalal [1975]. Koenig and Law [1985] also present a slightly modified screening procedure with the goal of selecting the m (user-specified) best of k systems without regard to rank. This procedure, however, requires minor modifications to the definition of δ and P∗ for implementation. This procedure also differs from those discussed in Section 2.2 since it allows the indifference zone, δ, to be specified in the subset selection process. Goldsman [1985] explores the use of standardized time series theory to determine variance estimators for R&S methodologies. An application of indifference-zone R&S procedures to choosing the best airspace configuration for a major European airport is described in Gray and Goldsman [1988]. Goldsman [1986] also gives an easy-to-read tutorial for those interested in the basics of the indifference-zone approach. The indifference-zone procedures discussed thus far select the best system as a univariate problem (i.e., with a single measure of system performance). Obviously, an experimenter could weight several parameters of interest to form a single measure of effectiveness (e.g., see Butler et al. [2001], Morrice et al. [1998], and Swisher and Jacobson [2002]). However, Dudewicz and Taneja [1978] describe a multivariate procedure that does not require reduction to a univariate model. They address the problem by defining a multivariate normal vector composed of p > 1 component variates (population output statistics of interest) having an unknown and unequal variance-covariance matrix. They then redefine δ as the Euclidian distance from a mean vector to the best mean ACM Transactions on Modeling and Computer Simulation, Vol. 13, No. 2, April 2003.

140

•

J. R. Swisher et al.

vector and follow the indifference zone approach (i.e., as originally introduced by Dudewicz and Dalal [1975]). Goldsman [1987] presents extensions of Bechhofer’s original work to multivariate (two-factor) models. Further extensions of the multivariate procedure for R&S remain an open research area. 2.2 Subset Selection Procedures In contrast to indifference-zone procedures, Gupta [1956] introduces a procedure for obtaining a subset of random size that contains the best system, with user-specified probability P∗ —without the specification of an indifference-zone (i.e., δ = 0). This procedure and others like it are known as subset selection R&S procedures. Gupta and Santner [1973] and Santner [1975] extend the original methodology to allow for user specification of a maximum size for the subset. Gupta and Hsu [1977] illustrate an application of these procedures to motor vehicle fatality data. Like the original indifference-zone R&S procedures, the original subset selection procedures required equal and known variances among system alternatives. For this reason, subset selection R&S procedures have rarely been applied to discrete-event simulation. However, Sullivan and Wilson [1989] describe a procedure that allows for unknown and unequal variances, as well as the specification of an indifference-zone. Although their R&S procedure makes subset selection more attractive for discrete-event simulation, indifference-zone procedures remain the more popular of the two. In most cases, an analyst wishes to determine the best system, not identify a subset containing the best. However, for those situations in which the analyst wishes to identify a subset containing the best, specialized indifference-zone procedures allow for a priori specification of the subset’s size (Koenig and Law [1985]). Subset selection procedures can also be coupled with indifference zone approaches to achieve better statistical efficiency. Recently, Nelson et al. [2001] introduced a decomposition lemma, which allows for the application of an indifference zone selection procedure to those designs that were first screened by a subset selection procedure and still guarantee an overall probability of correct selection. Wilson [2001] presents an enhanced lower bound on the probability of correct selection by introducing a multiplicative decomposition lemma. These lemmas furnish the theoretical basis for combining independently developed screening and ranking procedures to deal with situations where the number of alternatives is large enough to render conventional R&S procedures impractical. 2.3 Other R&S Approaches As R&S procedures have become more popular, the selection problem has also been modeled in a variety of ways. Chen [1988] and Goldsman [1984a, 1984b] both model the problem as that of selecting the multinomial cell that has the largest underlying probability. Chen focuses on subset selection problems using this model while Goldsman focuses on indifference-zone problems. Miller et al. [1998] also consider efficient multinomial selection procedures. ACM Transactions on Modeling and Computer Simulation, Vol. 13, No. 2, April 2003.

Discrete-Event Simulation Optimization

•

141

A natural extension of the combined screening-and-selection procedures is a fully sequential procedure, where a single observation is taken from each alternative design that is still in the running at the current stage of sampling and inferior designs are eliminated from further consideration until a best design is identified. In general, fully sequential procedures entail a great deal of overhead, though Kim and Nelson [2001] introduce a clever approach in managing this overhead. Goldsman et al. [2002] describe two interesting extensions of their approach. Alternatively, Chen [1995] and Chen et al. [1996] have formulated the R&S problem as a multistage optimization problem in which clearly inferior designs are identified in earlier stages of sampling and discarded from further consideration. They introduce a Bayesian heuristic to allocate further incremental computing budget to those alternatives that may increase the probability of correct selection. Chen et al. [1997] extend this work by presenting a different method for estimating gradient information. Chen et al. [1998] report a further extension of this work that accounts for simulation experiments with different sampling costs. Through numerical experimentation, they observe this approach to be more efficient than the method discussed in Chen et al. [1997]. Chen et al. [2000a, 2000b] offer further approaches to enhance the efficiency ¨ of their allocation scheme. Yucesan et al. [2001] look at the efficiency of simulation experiments for R&S procedures using the optimal computing budget allocation (OCBA) algorithm. Tong and Wetzell [1984] and Futschik and Pflug [1997] propose other sequential R&S methods. Miller et al. [2002] use R&S to estimate the probability that a simulated system is the best system. Chick [1997] presents a decision-theoretic analysis procedure for selecting the best simulated system. Inoue and Chick [1998] compare Bayesian and frequentist approaches for selecting the best system. A more extensive empirical comparison appears in Inoue et al. [1999]. Chick and Inoue [1998, 2001a, 2001b] extend Chick’s [1997] work to the study of sampling costs and value of information arguments to improve the computational efficiency of identifying the best system. Finally, Ahmed and Alkhamis [2002] demonstrate how an R&S procedure can be effectively combined with simulated annealing. 3. MULTIPLE COMPARISON PROCEDURES (MCPS) In contrast to R&S procedures, in which the goal is to make a decision, the goal of MCPs is to quantify the differences between systems’ performance (not guarantee a decision). Three general classes of MCPs have been developed: all-pairwise comparisons approaches are covered in Section 3.1, multiple comparisons with a control (MCC) are discussed in Section 3.2, and multiple comparisons with the best (MCB) are covered in Section 3.3. Several books and survey papers have been presented that give a general overview of MCPs. Hochberg and Tamhane [1987] contains a comprehensive review of MCPs. Goldsman and Nelson [1994, 1998] present comprehensive stateof-the-art reviews of MCPs in simulation. Wherever possible, they attempt to unify R&S and MCP perspectives. Fu [1994a, 1994b] contain an overview of MCPs as they relate to the larger field of simulation optimization. Kleijnen ACM Transactions on Modeling and Computer Simulation, Vol. 13, No. 2, April 2003.

142

•

J. R. Swisher et al.

[1977] discusses MCPs in the context of other useful statistical tools in simulation. Wen and Chen [1994] present single-stage sampling procedures for different MCPs. Hsu [1996] is a thorough text on the theory and application of MCPs. 3.1 All Pairwise Comparisons Methodologies All pairwise comparison methodologies can be grouped into two general categories: (1) the brute force approach (combined paired-t, Bonferroni, all-pairwise comparisons), as described by Fu (1994a, 1994b), and (2) all pairwise multiple comparisons (MCA). In the brute force approach, all possible pairwise confidence intervals for system designs are examined (i.e., (µi − µ j ) ∈ (Y i· − √ Y j · ) ± tn−1,α/2 / n for all i, j pairs), resulting in a total of k(k − 1)/2 confidence intervals. Due to the Bonferroni inequality, each confidence interval must be constructed at level 1- α/[k(k − 1)/2] in order to have an overall confidence level of at least (1 − α) for all the intervals jointly. Clearly, for more than ten alternatives, the width of the individual confidence intervals becomes quite large. Unfortunately, unless there is a clear winner among the systems (i.e., a system with the confidence interval for the difference with all other pairs that is strictly positive), one gains little inference from this approach. MCA has its origins in Tukey [1953] and is similar to the brute force method, except that instead of constructing separate confidence intervals and using the Bonferroni inequality to determine an overall confidence bound, a simultaneous set of confidence intervals at an overall (1 − α) level is formed. Tukey’s √ α s/ n simultaneous confidence intervals are (µi − µ j ) ∈ (Y i· − Y j · ) ± rk,k(n−1) α for all i 6= j where s is the pooled sample standard deviation and rk,k(n−1) is the upper α-quantile of the Studentized range distribution with k systems and k(n − 1) degrees of freedom (found in tables in Hochberg and Tamhane [1987]). Like the previous method, MCA requires k(k −1)/2 confidence intervals to be constructed. However, MCA obtains an overall simultaneous confidence level with the same confidence half-widths for each pairwise comparison, while the brute-force method obtains a different confidence half-width for each pairwise comparison and uses the Bonferroni inequality to establish a bound on the overall confidence. Yang and Nelson [1991] describe a revision for MCA that allows CRN and control variates to be used. Wu and Chen [2000] outline a two-stage method for multiple comparisons with the average for normal distributions under heteroscedasticity. They also provide both tables and software programs that can be used to apply their method. 3.2 Multiple Comparisons with a Control (MCC) There are situations when an experimenter may wish to compare a set of alternatives to a predefined control, which may represent an existing system. The construction of (k − 1) simultaneous confidence intervals in comparison to a fixed control (e.g., the current design) is attributed to Dunnett [1955] and is known as MCC. Yang and Nelson [1991] present a revision for MCC that allows the use of variance reduction techniques such as common random numbers (CRN) and control variates. Dudewicz and Ramberg [1972], Dudewicz and ACM Transactions on Modeling and Computer Simulation, Vol. 13, No. 2, April 2003.

Discrete-Event Simulation Optimization

•

143

Dalal [1983], and later Bofinger and Lewis [1992] expand the traditional MCC procedure by describing two-stage MCC procedures. Damerdji and Nakayama [1996] describe a two-stage MCC procedure that allows different systems to have different probability distributions. Nelson and Goldsman [2001] present two-stage experimental design and analysis procedures for comparing several designs against a single (standard) design. 3.3 Multiple Comparisons with the Best (MCB) MCB procedures have their origin in Hsu [1984] and Hsu and Nelson [1988]. MCB’s intent is similar to that of R&S procedures: determine the best system from a set of alternatives. MCB addresses this problem by forming simultaneous confidence intervals on the parameters µi − max j 6=i µ j for i = 1, 2, . . . , k. These (k − 1) simultaneous confidence intervals bound the difference between the expected performance of each system and the best of the other systems. By their very nature, MCB procedures include subset selection (much like the R&S procedures described in Section 2.2) by identifying the set of design alternatives whose MCB confidence interval includes zero. Yang and Nelson [1989, 1991] and Nelson and Hsu [1993] describe modifications to the MCB procedure that incorporate two variance reduction techniques (control variates and CRN). Their results suggest that using variance reduction can shorten the length of the confidence intervals for a specified level of confidence. Nelson’s [1993] robust MCB procedure also allows CRN under the assumption that the variance-covariance matrix exhibits sphericity (see Section 4). Nakayama [1997b, 2000] presents a two-stage MCB procedure that also uses CRN for steady-state simulations, and shows that it is asymptotically valid. Goldsman and Nelson [1990] outline an MCB procedure for steady-state simulation experiments. They also discuss results on how the batch size can impact the probability of correct selection when using the simulation technique of batch means. Nelson and Banerjee [2001] present a two-stage MCB procedure that simultaneously achieves several objectives for a given probability of correct selection. Nakayama [1995] presents general results on applying MCB using batch means in steady-state simulations. Yuan and Nelson [1993] discuss MCB procedures for steady-state simulations, where the simulation outputs are assumed to follow an autoregressive process with normally distributed residuals. This procedure does not however allow the use of CRN. Damerdji et al. [1996] describe a procedure for selecting the best system for transient simulations with known variances. Nakayama [1996, 1997a] introduces a single-stage MCB procedure that is asymptotically valid for steady-state simulations, hence extending the domain of applicability of previous MCB procedures. Kolonja et al. [1993] describe an application of MCB using CRN to a truck dispatching system simulation model. 4. UNIFIED PROCEDURES There has been a recent effort to unify the fields of R&S and MCPs. Gupta and Hsu [1984] are the first to propose a methodology for simultaneously executing R&S and MCB. Matejcik and Nelson [1993, 1995] establish a fundamental ACM Transactions on Modeling and Computer Simulation, Vol. 13, No. 2, April 2003.

144

•

J. R. Swisher et al.

connection between indifference-zone procedures and MCB. The idea of combining indifference-zone approaches with MCB is appealing to the simulation analyst. Such an approach not only selects the best system with prespecified confidence, but it allows one to draw inferences about the relationships between systems that may facilitate decision-making based on secondary criteria that are not reflected in the performance measure selected. Nelson and Matejcik [1995] show that most indifference-zone procedures can simultaneously compute MCB confidence intervals with the width of the intervals corresponding to the indifference-zone. Therefore, both indifferencezone ranking and MCB inference can be derived from the same experiment with a prespecified MCB interval width δ. They describe four R&S—MCB procedures that depend on having normally distributed data, but do not require known or equal variance: (1) (2) (3) (4)

Rinott’s Procedure (Procedure R), Dudewicz and Dalal’s Procedure (Procedure DD), Clark and Yang’s Procedure (Procedure CY), and Nelson and Matejcik’s Procedure (Procedure NM).

Procedure R is an extension of Rinott’s [1978] two-stage indifference-zone R&S procedure as described in Section 2. Procedure R requires n0 (the firststage sample size) independent and identically distributed samples from each of the k independently-simulated systems. The marginal sample variance for each system is then computed and used to determine the final sample size for each system, Ni (for i = 1, 2, . . . , k). After Ni − n0 additional independent and identically distributed observations are taken from each of the k systems, independent of the first-stage samples and independent of the other secondstage samples, the system with the largest overall sample mean is selected as the best. In addition, MCB confidence intervals on µi − max j 6=i µ j are formed. Likewise, Procedure DD (based on Dudewicz and Dalal [1975]) is performed in the same manner with the only difference being in the calculation of the sample means. While Procedures R and DD allow for both R&S selection and MCB inference, their requirement for independence among all observations precludes the use of CRN. The total sample size required to obtain the desired confidence level is dependent on the sample variances of the systems. In particular, the larger the sample variance, the more replications (or batch means) are required. For this reason, simultaneous R&S-MCB procedures that exploit CRN should require fewer total observations to obtain the same confidence level. Procedure CY is based upon Clark and Yang’s [1986] indifference-zone R&S procedure. As one of the few R&S procedures that allow CRN, Clark and Yang [1986] use the Bonferroni inequality to account for the dependence induced by CRN. It is therefore a conservative procedure that typically prescribes a larger total number of observations than are actually necessary to make a correct selection. Like Procedure R, Procedure CY is performed in two stages. In the first stage, independent and identically distributed samples from each of the k systems are taken using CRN across systems. The sample variances of the differences are then used to compute the final sample size, N (where N does ACM Transactions on Modeling and Computer Simulation, Vol. 13, No. 2, April 2003.

Discrete-Event Simulation Optimization

•

145

not vary across systems for Procedure CY). After taking the remaining N − n0 independent and identically distributed observations, again using CRN across systems, the system with the largest sample mean is selected as the best and the MCB confidence intervals are constructed. Nelson and Matejcik [1995] find that Procedure CY can be effective in reducing the total number of samples required to make a correct selection in comparison with both Procedures R and DD. However, they also observe that the benefit gained from using Procedure CY is diminished when the number of systems to be compared, k, is large, since the conservatism of Procedure CY from the Bonferroni inequality increases as k increases and, at some point, overwhelms the benefit induced by CRN. To overcome this problem, Procedure NM (motivated by Nelson’s [1993] robust MCB procedure) is presented. Since this procedure is used to illustrate the application of R&S and MCPs in Section 5, it is described in detail here. Procedure NM assumes that the unknown variancecovariance matrix, 6, exhibits a structure known as sphericity. More specifically, the sphericity structure takes the form  X

2ψ1 + τ 2 ψ1 + ψ2 · · · ψ1 + ψk

 ψ + ψ 2ψ + τ 2 · · · ψ + ψ 1 2 2 k  2 =  ..  .

     

ψk + ψ1 ψk + ψ2 · · · 2ψk + τ 2 q P Pk k ψi2 − i=1 ψi such that 6 is guaranteed to be positive where τ 2 > k i=1 definite [Nelson and Matejcik 1995]. Sphericity implies that Var[Y ij −Y lj ] = 2τ 2 for all i 6= l . This means that the variances of all pairwise differences across systems are equal, even though the marginal variances and covariances may be unequal. Sphericity generalizes compound symmetry [Nelson and Matejcik 1995], which takes the form:   1 ρ ··· ρ  ρ 1 ··· ρ  X   2 . =σ   .. .   ρ ρ ··· 1 Several researchers have suggested that compound symmetry accounts for the variance reduction effects of CRN (see Tew and Wilson [1994], Nozari et al. [1987], and Schruben and Margolin [1978] for more details). Procedure NM is valid when 6 satisfies sphericity; however, Nelson and Matejcik [1995] show it to be extremely robust to departures from sphericity. Goldsman and Nelson [1998] give the following description of the procedure: (1−α) (1−α) , where Tk−1,(k−1)(n is the (1) Specify δ, α, n0 . Let g = Tk−1,(k−1)(n 0 −1),0.50 0 −1),0.50 (1 − α)-quantile of the maximum of a multivariate t random variable with k −1 dimensions, (k −1)(n0 −1) degrees of freedom, and common correlation 0.50. ACM Transactions on Modeling and Computer Simulation, Vol. 13, No. 2, April 2003.

146

•

J. R. Swisher et al.

(2) Take independent and identically distributed samples Y i1 , Y i2 , . . . , Y in0 from each of the k competing systems using CRN across systems. (3) Compute the sample variance of the difference under the condition of sphericity as Pk Pn0 ¯ ¯ ¯ 2 2 i=1 j =1 (Y i j − Y i . − Y . j + Y .. ) 2 . S = (k − 1)(n0 − 1) (4) Compute the final required sample size (constant for all k alternatives) as N = max {no , d(gS /δ)2 e}. (5) Take N − n0 additional independent and identically distributed observations from each system, using CRN across systems. (6) Compute the overall sample means for each system as 1 XN Y¯ i . = Y ij for i = 1, 2, . . . , k. j =1 N (7) Select the system with the largest Y¯ i· as the best alternative. (8) Simultaneously, form the MCB confidence intervals as µi − max µ j ∈ [−(Y¯ i . − max Y¯ j . − δ)− , (Y¯ i . − max Y¯ j . + δ)+ ] j 6=i

j 6=i

j 6=i

for i = 1, 2, . . . , k where − x − = min {0, x} and x + = max {0, x}. (1−α) in Step (1) of Procedure NM can be obtained The value of Tk−1,(k−1)(n 0 −1),0.50 from Table 4 in Hochberg and Tamhane [1987] or Table B.3 in Bechhofer et al. [1995]. Values that fall outside of the tables can be computed using a FORTRAN computer program (see Dunnett [1989]). Nelson and Matejcik [1995] report results that suggest that Procedure NM is superior to Procedures R, DD, and CY in terms of the total number of observations required to obtain the desired confidence level. Procedure NM’s only potential drawback is that the assumption of sphericity may not be exactly or even approximately satisfied in many situations [Nelson and Matejcik 1995]; however, they show it to be sufficiently robust to departures from sphericity for use in practice. The authors also suggest that the simulation analyst consider slightly inflating the nominal coverage probability to assure adequate coverage.

5. APPLICATION OF PROCEDURE NM Combined R&S–MCB procedures are more attractive than using either R&S or MCB individually, since combined procedures allow for both R&S selection and MCB inference with little or no additional computational overhead. The power of these procedures is that they provide both the optimal system design with prespecified confidence (R&S) and inferences about that design’s superiority (MCB). Procedure NM [Nelson and Matejcik 1995] is the most efficient of the existing combined procedures. For this reason, Swisher and Jacobson [2002] selected it as the simulation optimization technique to apply to the determination of the optimal clinic design for a two-physician family practice clinic from among a group of seventeen competing alternative designs [Swisher et al. 2001]. The results reported here are taken from Swisher and Jacobson [2002] to illustrate procedure NM. Optimality is defined in terms of a multiattribute ACM Transactions on Modeling and Computer Simulation, Vol. 13, No. 2, April 2003.

Discrete-Event Simulation Optimization

•

147

Table II. Comparison of Tests for Normality Normality Test Anderson-Darling Kolmogorov-Smirnov Shapiro-Wilk W

p-Value 0.6829 > 0.15 0.8100

performance measure referred to as the clinic effectiveness (CE) measure that in essence penalizes clinic profit for poor patient service (e.g., excessive waiting). The first step in applying Procedure NM is to verify that the output data used is normally distributed. Therefore, the sample means of 30 batches of 10 observations each, from 30 baseline model replications were tested for normality, where an observation is the CE value for a given simulated day. Three tests for normality were applied (using Analyse-It for Microsoft Excel v1.32): Anderson–Darling modified for use with unknown population mean and variance [D’Agostino and Stephens 1986], Kolmogorov–Smirnov modified for use with unknown population mean and variance [D’Agostino and Stephens 1986], and Shapiro–Wilk W [Royston 1992]. The results of these tests (see Table II) show large p-values, suggesting that the null hypothesis of normally distributed data would not be rejected at any reasonable confidence level. Procedure NM allows the specification of an indifference zone. A value of δ = 10 was used to select the optimal clinic design from among the seventeen competing alternatives. Since a P{CS} = (1 − α) = 0.95 is desired, a more conservative value of α = 0.03 will be used, as suggested by Nelson and Matejcik (0.97) = 2.7910 (as derived [1995]. Given n0 = 30, δ = 10, and α = 0.03, g = T16,464,0.50 from Dunnett’s [1989] procedure). The sample variance under the condition of sphericity, S 2 , is 882.80 and N = max{30, d(gS/w)2 e} = 69. Therefore, N −n0 = 69 − 30 = 39 observations were generated for each of the seventeen clinic designs by simulating 417 clinic days (27 deleted transient observations plus 390 usable steady-state observations) using CRN across designs and forming 39 batches of size b = 10. Using the formulas guven in Section 4, the overall sample means and MCB intervals were then formed (see Table III). Procedure NM selects clinic design 4, with Y 4· , = 406.17, as the best clinic design. From a R&S perspective, this means that with probability of at least 0.97, clinic design 4 has mean µ4 within δ = 10 of the design with the true largest mean, µ[k] . Examination of the MCB intervals allows for inferences on the (assumed) superiority of design 4. Note that, four other designs (7, 8, 9, and 12) have MCB intervals that also contain zero. This means, from an MCB perspective, there is no one clearly superior design. The resulting subset selection inference concludes that designs 4, 7, 8, 9 and 12 are all clearly superior to the remaining systems (whose upper MCB bounds are zero); there is however no clear winner among these five designs, though had one of these designs possessed a lower MCB bound of zero, while the rest were upper-bounded by zero, then that design would have been selected as the best by MCB. One of the benefits of using a combined R&S–MCB procedure is that the analyst gains inferences on systems other than the best, which may lead to the selection of an inferior system (if it is not inferior by much) based on some secondary ACM Transactions on Modeling and Computer Simulation, Vol. 13, No. 2, April 2003.

•

148

J. R. Swisher et al. Table III. Overall Sample Means and MCB Results for Procedure NM

Design

Y i·

Lower MCB Limit

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17

388.17 392.41 394.18 406.17 380.70 386.07 405.93 398.74 405.32 382.72 381.11 401.98 392.53 392.07 393.78 378.67 378.28

−28.00 −23.76 −21.99 −9.76 −35.47 −30.10 −10.24 −17.43 −10.85 −33.45 −35.06 −14.19 −23.65 −24.10 −22.39 −37.50 −37.90

Y i· − max Y j 6=i

j·

−18.00 −13.76 −11.99 0.24 −25.47 −20.10 −0.24 −7.43 −0.85 −23.45 −25.06 −4.19 −13.65 −14.10 −12.39 −27.50 −27.90

Upper MCB Limit 0 0 0 10.24 0 0 9.76 2.57 9.15 0 0 5.81 0 0 0 0 0

Table IV. Mean Daily Clinic Profit for the Five Best Clinic Designs Clinic Design 4 7 8 9 12

Mean Daily Profit ($) 851.92 990.78 931.89 833.88 972.08

criterion not reflected in the performance measure of interest [Matejcik and Nelson 1993]. Although profit is a component of the (CE) performance measure used, no real inference on clinic profit can be made from examining this measure. Therefore, a decision-maker would also likely be interested in examining clinic profit for the five clinic designs whose MCB intervals cover zero. Table IV contains the mean daily clinic profit (without service penalties) for each of the five best clinic designs. Clinic design 7 produces approximately $140 per day more clinic profit than clinic design 4 (the design selected as the best). In addition, the overall sample mean for clinic design 7 is $0.24 less than the overall sample mean of design 4 (see Table III). In essence, the MCB inference provided by Procedure NM leads the clinical decision-maker to choose clinic design 7, despite the fact that clinic design 4 was selected as the best by Procedure NM’s R&S result. If only a R&S approach had been used to evaluate the clinic designs, the clinical decisionmaker would have selected an excellent design in terms of CE. However, that choice may cost the clinic $140 per day in profit (if the difference between the profit measures between the two designs is statistically significant) compared to an equally good (from an MCB perspective) choice. In this case, the value of the application of a combined R&S–MCB procedure is obvious. ACM Transactions on Modeling and Computer Simulation, Vol. 13, No. 2, April 2003.

Discrete-Event Simulation Optimization

•

149

Table V. Recommended Techniques Given Experimentation Goals and Situations Situation Select the best system given a predetermined indifference-zone; variances unequal and unknown; CRN allowed. Screen for a subset containing the best system using an indifference zone; variances unequal and unknown. Select a subset of random size containing the best system; variances unequal and unknown. Combine screening and selection approaches to enhance efficiency of traditional R&S; variances unequal and unknown. Select the best system using a decision-theoretic approach for optimal sampling in a fully sequential manner; CRN allowed. Examine all pairwise confidence intervals of differences between systems; CRN and control variates allowed. Compare confidence intervals of differences between competing systems and a control; CRN and control variates allowed. Form simultaneous confidence intervals of differences between competing systems to select the best; CRN and control variates allowed. Form simultaneous confidence intervals of differences between competing (steady-state) systems to select the best; different random number streams used across systems for a given replication. Select the best system and provide inferences about the relationships between systems; variances unequal and unknown; CRN allowed. Select the best system and provide inferences about the relationships between systems; CRN allowed; assumption that variance-covariance matrix exhibits sphericity.

Technique IZ R&S

Reference(s) Clark and Yang [1986]

IZ R&S

Koenig & Law [1985]

SS R&S

Sullivan & Wilson [1989]

Screen and R&S

Nelson et al. [2001]

Sequential R&S

Chick and Inoue [2001]

MCA

Yang & Nelson [1991]

MCC

Yang & Nelson [1991]

MCB

Nelson & Hsu [1993]

MCB

Yuan and Nelson [1993]

Unified R&S-MCB

Clark and Yang [1986], Nelson and Matejcik [1995]

Unified R&S-MCB

Nelson and Matejcik [1995]

6. CONCLUSIONS Ranking and selection and multiple comparison procedures are excellent tools for selecting the best of k competing alternatives in discrete-event simulation. A summary listing of recommended (most widely used and/or well-established) R&S and MCP procedures given particular experimental goals and situations is given in Table V. R&S approaches allow the simulation analyst to screen alternatives so that a subset of size m may be fully studied. They also allow ACM Transactions on Modeling and Computer Simulation, Vol. 13, No. 2, April 2003.

150

•

J. R. Swisher et al.

the analyst to choose the best of k alternatives, given an indifference-zone, at or above a user-specified probability. MCPs allow one to make inferences about the relationships among k alternatives. The typical simulation analyst would likely benefit most from applying the unified R&S–MCB procedures proposed by Matejcik and Nelson [1993, 1995] and Nelson and Matejcik [1995]. These procedures allow the analyst to select the best system and gain insight into how much better the best is in comparison to the remaining alternatives. Unified methodologies such as these should dominate the R&S and MCP literature in the future. In addition, further research in the area of multivariate R&S procedures would be beneficial for those situations in which the analyst cannot easily reduce the performance measures to a univariate measure. ACKNOWLEDGMENTS

The authors wish to thank the special issue editor, Dr. Barry L. Nelson, and two anonymous referees, for their thoughtful comments and suggestions that have led to a significantly improved manuscript. REFERENCES AHMED, M. A. AND ALKHAMIS., T. M. 2002. Simulation-based optimization using simulated annealing with ranking and selection. Comput. Oper. Res. 29, 4, 387–402. BANKS, J., CARSON, J. S., AND NELSON, B. L. 1996. Discrete-Event System Simulation, 2nd ed. Prentice-Hall, Upper Saddle River, N.J. BECHHOFER, R. E. 1954. A single-sample multiple decision procedure for ranking means of normal populations with known variances. Ann. Math. Stat. 25, 16–39. BECHHOFER, R. E., KIEFER, J., AND SOBEL, M. 1968. Sequential Identification and Ranking Procedures. The University of Chicago Press, Chicago, Ill. BECHHOFER, R. E., SANTNER, T. J., AND GOLDSMAN, D. M. 1995. Design and Analysis of Experiments for Statistical Selection, Screening, and Multiple Comparisons. Wiley, New York. BOFINGER, E. AND LEWIS, G. J. 1992. Two-stage procedures for multiple comparisons with a control. Amer. J. Math. Manage. Sci. 12, 4, 253–275. BUTLER J., MORRICE, D. J., AND MULLARKEY, P. W. 2001. A multiple attribute utility theory approach to ranking and selection. Manage. Sci. 47, 6, 800–816. CARSON, J. S. 1996. AutoStatTM : Output statistical analysis for AutoModTM users. In Proceedings of the 1996 Winter Simulation Conference, J. M. Charnes, D. J. Morrice, D. T. Brunner, and J. J. Swain, Eds. IEEE, Piscataway, N.J., 492–499. CHEN, C. H. 1995. An effective approach to smartly allocate computing budget for discreteevent simulation. In Proceedings of the 34th IEEE Conference on Decision and Control. IEEE, Piscataway, N.J., 2598–2605. CHEN, C. H., CHEN, H. C., AND DAI, L. 1996. A gradient approach for smartly allocating computing budget for discrete event simulation. In Proceedings of the 1996 Winter Simulation Conference. J. M. Charnes, D. J. Morrice, D. T. Brunner, and J. J. Swain, Eds. IEEE, Piscataway, N.J., 398–405. ¨ CHEN, C. H., YUAN, Y. CHEN, H. C., YUCESAN , E., AND DAI., L. 1998. Computing budget allocation for simulation experiments with different system structures. In Proceedings of the 1998 Winter Simulation Conference. M. Abrams, P. Haigh, and J. Comfort, Eds. IEEE, Piscataway, N.J., 735– 741. ¨ CHEN, C. H., LIN, J., YUCESAN , E., AND CHICK, S. E. 2000a. Simulation budget allocation for further enhancing the efficiency of ordinal optimization. Disc. Event Dyn. Syst.: Theory Appl. 10, 251–270. ¨ CHEN, H. C., CHEN, C. H., DAI, L., AND YUCESAN , E. 1997. New development of optimal computing budget allocation for discrete event simulation. In Proceedings of the 1997 Winter Simulation Conference. S. Andradottir, K. J. Healy, D. H. Withers, and B. L. Nelson, Eds. IEEE, Piscataway, N.J., 334–341. ACM Transactions on Modeling and Computer Simulation, Vol. 13, No. 2, April 2003.

Discrete-Event Simulation Optimization

•

151

¨ CHEN, H. C., CHEN, C. H., AND YUCESAN , E. 2000b. Computing effort allocation for ordinal optimization and discrete event simulation. IEEE Trans. Automat. Cont. 45, 5, 960–964. CHEN, P. 1988. On selecting the best of k systems: An expository survey of subset-selection multinomial procedures. In Proceedings of the 1988 Winter Simulation Conference. M. Abrams, P. Haigh, and J. Comfort, Eds. IEEE, Piscataway, N.J., 440–444. CHICK, S. E. 1997. Selecting the best system: A decision-theoretic approach. In Proceedings of the 1997 Winter Simulation Conference. S. Andradottir, K. J. Healy, D. H. Withers, and B. L. Nelson, Eds. IEEE, Piscataway, N.J., 326–333. CHICK, S. E. AND INOUE, K. 1998. Sequential allocations that reduce risk for multiple comparisons. In Proceedings of the 1987 Winter Simulation Conference. A. Thesen, H. Grant, and W. D. Kelton, Eds. IEEE, Piscataway, N.J., 669–676. CHICK, S. E. AND INOUE, K 2001a. New procedures to select the best simulated system using common random numbers. Manage. Sci. 47, 8, 1133–1149. CHICK, S. E. AND INOUE, K. 2001b. New two-stage and sequential procedures for selecting the best simulated system. Oper. Res. 49, 5, 732–743. CLARK, G. M. AND YANG, W. 1986. A Bonferroni selection procedure when using common random numbers with unknown variances. In Proceedings of the 1986 Winter Simulation Conference. J. R. Wilson, J. O. Henriksen, and S. D. Roberts, Eds. IEEE, Piscataway, N.J., 313–315. D’AGOSTINO, R. B. AND STEPHENS, M. A. 1986. Goodness of Fit Techniques. Marcel Dekker, New York. DAMERDJI, H. AND NAKAYAMA, M. K. 1996. Two-stage procedures for multiple comparisons with a control in steady-state simulations. In Proceedings of the 1996 Winter Simulation Conference. J. M. Charnes, D. J. Morrice, D. T. Brunner, and J. J. Swain, Eds. IEEE, Piscataway, N.J., 372–375. DAMERDJI, H., GLYNN, P. W., NAKAYAMA, M. K., AND WILSON, J. R. 1996. Selecting the best system In transient simulations with known variances. In Proceedings of the 1996 Winter Simulation Conference. J. M. Charnes, D. J. Morrice, D. T. Brunner, and J. J. Swain, Eds. IEEE, Piscataway, N.J., 281–286. DUDEWICZ, E. J. 1976. Statistics in simulation: How to design for selecting the best alternative. In Proceedings of the 1976 Winter Simulation Conference. T. J. Schriber, R. G. Sargent, H. J. Highland, Eds. IEEE, Piscataway, N.J., 67–71. DUDEWICZ, E. J. AND DALAL, S. R. 1975. Allocation of observations in ranking and selection with unequal variances. Indi. J. Stat. 37B, 1, 28–78. DUDEWICZ, E. J. AND DALAL, S. R. 1983. Multiple comparisons with a control when variances are unknown and unequal. Amer. J. Math. Manage. Sci. 3, 275–295. DUDEWICZ, E. J. AND RAMBERG, J. S. 1972. Multiple comparisons with a control: Unknown variances. In The Annual Technical Conference Transactions of the American Society of Quality Control, vol. 26, 483–488. DUDEWICZ, E. J. AND TANEJA, V. S. 1978. Multivariate ranking and selection without reduction to a univariate problem. In Proceedings of the 1978 Winter Simulation Conference. H. J. Highland, L. G. Hull, N. R. Neilsen, Eds. IEEE, Piscataway, N.J., 207–210. DUNNETT, C. W. 1955. A multiple comparisons procedure for comparing several treatments with a control. J. Ameri. Stat. Assoc. 78, 965–971. DUNNETT, C. W. 1989. Multivariate normal probability integrals with product correlation structure. Appl. Stat. 38, 564–579. Correction: 42, 709. FU, M. 1994a. Optimization via simulation: A review. Ann. Oper. Res. 53, 199–247. FU, M. 1994b. A tutorial review of techniques for simulation optimization. In Proceedings of the 1994 Winter Simulation Conference. J. D. Tew, S. Manivannan, D. A. Sadowski, and A. F. Seila, Eds. IEEE, Piscataway, N.J., 149–156. FUTSCHIK, A. AND PFLUG, G. C. 1997. Optimal allocation of simulation experiments in discrete stochastic optimization and approximative algorithms. Europ. J. Oper. Res. 101, 2, 245–260. GOLDSMAN, D. 1983. Ranking and selection in simulation. In Proceedings of the 1983 Winter Simulation Conference. S. Roberts, J. Banks, and B. Schmeiser, Eds. IEEE, Piscataway, N.J., 387–393. GOLDSMAN, D. 1984a. A multinomial ranking and selection procedure: Simulation and applications. In Proceedings of the 1984 Winter Simulation Conference. S. Sheppard, U. Pooch, and D. Pegden, Eds. IEEE, Piscataway, N.J., 259–263. ACM Transactions on Modeling and Computer Simulation, Vol. 13, No. 2, April 2003.

152

•

J. R. Swisher et al.

GOLDSMAN, D. 1984b. On selecting the best of k systems: An expository survey of indifference-zone multinomial procedures. In Proceedings of the 1984 Winter Simulation Conference. S. Sheppard, U. Pooch, and D. Pegden, Eds. IEEE, Piscataway, N.J., 107–112. GOLDSMAN, D. 1985. Ranking and selection procedures using standardized time series. In Proceedings of the 1985 Winter Simulation Conference. D. Gantz, G. Blais, and S. Solomon, Eds. IEEE, Piscataway, N.J., 120–123. GOLDSMAN, D. 1986. Tutorial on indifference-zone normal means ranking and selection procedures. In Proceedings of the 1986 Winter Simulation Conference. J. Wilson, J. Henriksen, and S. Roberts, Eds. IEEE, Piscataway, N.J., 370–375. GOLDSMAN, D. 1987. Ranking and selection tutorial: 2-factor normal means procedures. In Proceedings of the 1987 Winter Simulation Conference. A. Thesen, H. Grant, and W. D. Kelton, Eds. IEEE, Piscataway, N.J., 52–57. GOLDSMAN, D., KIM, S. H., MARSHALL, W. S., AND NELSON, B. L. 2002. Ranking and selection for steady-state simulation: Procedures and perspectives. INFORMS J. Comput. 14, 1, 2– 19. GOLDSMAN, L. AND NELSON, B. L. 1990. Batch-size effects on simulation optimization using multiple comparisons. In Proceedings of the 1990 Winter Simulation Conference. O. Balci, R. P. Sadowski, R. E. Nance, Eds. IEEE, Piscataway, N.J., 288–293. GOLDSMAN, D. AND NELSON, B. L. 1994. Ranking, selection and multiple comparisons in computer simulation. In Proceedings of the 1994 Winter Simulation Conference J. D. Tew, S. Manivannan, D. A. Sadowski, and A. F. Seila, Eds. IEEE, Piscataway, N.J., 192–199. GOLDSMAN, D. AND NELSON, B. L. 1998. Statistical screening, selection, and multiple comparison procedures in computer simulation. In Proceedings of the 1998 Winter Simulation Conference. D. J. Medeiros, E. F. Watson, J. S. Carson, and M. S. Manivannan, Eds. IEEE, Piscataway, N.J., 159–166. GRAY, D. AND GOLDSMAN, D. 1988. Indifference-zone selection procedures for choosing the best airspace configuration. In Proceedings of the 1988 Winter Simulation Conference. M. Abrams, P. Haigh, and J. Comfort, Eds. IEEE, Piscataway, N.J., 445–450. GUPTA, S. S. 1956. On a decision rule for a problem in ranking means. Mimeograph Series No. 150, Institute of Statistics. University of North Carolina, Chapel Hill, Chapel Hill, N.C. GUPTA, S. S. AND HSU, J. C. 1977. Subset selection procedures with special reference to the analysis of two-way layout: Application to motor-vehicle fatality data. In Proceedings of the 1977 Winter Simulation Conference. H. J. Highland, R. G. Sargent, J. W. Schmidt, Eds. IEEE, Piscataway, N.J., 81–85. GUPTA, S. S. AND HSU, J. C. 1984. A computer package for ranking, selection, and multiple comparisons with the best. In Proceedings of the 1984 Winter Simulation Conference. S. Sheppard, U. Pooch, and D. Pegden, Eds. IEEE, Piscataway, N.J., 251–257. GUPTA, S. S. AND SANTNER, T. J. 1973. On selection and ranking procedures—a restricted subset selection rule. In Proceedings of the 39th Session of the International Statistical Institute, vol. 1. HO, Y. C., SREENIVAS, R., AND VAKILI, P. 1992. Ordinal optimization of discrete event dynamic systems. Disc. Event Dynam. Syst. 2, 2, 61–88. HOCHBERG, Y. AND TAMHANE, A. C. 1987. Multiple Comparison Procedures. Wiley, New York. HSU, J. C. 1984. Constrained simultaneous confidence intervals for multiple comparisons with the best. Ann. Stat. 12, 1136–1144. HSU, J. C. 1996. Multiple Comparisons: Theory and Methods. Chapman & Hall, London, England. HSU, J. C. AND NELSON, B. L. 1988. Optimization over a finite number of system designs with one-stage sampling and multiple comparisons with the best. In Proceedings of the 1988 Winter Simulation Conference. M. Abrams, P. Haigh, and J. Comfort, Eds. IEEE, Piscataway, N.J., 451– 457. INOUE, K. AND CHICK, S. E. 1998. Comparison of Bayesian and frequentist assessments of uncertainty for selecting the best system. In Proceedings of the 1998 Winter Simulation Conference. M. Abrams, P. Haigh, and J. Comfort, Eds. IEEE, Piscataway, N.J., 727–734. INOUE, K., CHICK, S. E., AND CHEN, C. H. 1999. An empirical evaluation of several methods to select the best system. ACM Trans. Model. Comput. Simul. 9, 4, 381–407. JACOBSON, S. H. AND SCHRUBEN, L. W. 1989. A review of techniques for simulation optimization. Oper. Res. Lett. 8, 1–9. ACM Transactions on Modeling and Computer Simulation, Vol. 13, No. 2, April 2003.

Discrete-Event Simulation Optimization

•

153

KIM, S. H. AND NELSON, B. L. 2001. A fully sequential procedure for indifference-zone selection in simulation. ACM Trans. Model. Comput. Simul. 11, 3, 251–273. KLEIJNEN, J. P. C. 1977. Design and analysis of simulations: practical statistical techniques. Simulation., 28, 3, 81–90. KOENIG, L. W. AND LAW, A. M. 1985. A procedure for selecting a subset of size m containing the one best of k independent normal populations, with applications to simulation. Communi. Stat. B14, 3, 719–734. KOLOJNA, B., KALASKY, D. R., AND MUTMANSKY, J. M. 1993. Optimization of dispatching criteria for open-pit truck haulage system design using multiple comparisons with the best and common random numbers. In Proceedings of the 1993 Winter Simulation Conference. G. W. Evans, M. Mollaghasemi, E. C. Russell, and W. E. Biles, Eds. IEEE, Piscataway, N.J., 393–401. LAW, A. M. AND KELTON, W. D. 2000. Simulation Modeling and Analysis, 3rd Edn. McGraw-Hill, New York. MATEJCIK, F. J. AND NELSON, B. L. 1993. Simultaneous ranking, selection and multiple comparisons for simulation. In Proceedings of the 1993 Winter Simulation Conference. G. W. Evans, M. Mollaghasemi, E. C. Russell, and W. E. Biles, Eds. IEEE, Piscataway, N.J., 386–392. MATEJCIK, F. J. AND NELSON, B. L. 1995. Two-stage multiple comparisons with the best for computer simulation. Oper. Res. 43, 4, 633–640. MILLER J. O., NELSON, B. L., AND REILLY, C. H. 1998. Efficient multinomial selection in simulation. Naval Res. Logi. 45, 5, 459–482. MILLER J. O., NELSON, B. L., AND REILLY, C. H. 2002. Estimating the probability that a simulated system will be the best. Naval Res. Logi. 49, 4, 341–358. MORRICE, D. J., BUTLER, J., AND MULLARKEY, P. W. 1998. An approach to ranking and selection for multiple performance measures. In Proceedings of the 1998 Winter Simulation Conference. D. J. Medeiros, E. F. Watson, J. S. Carson, and M. S. Manivannan, Eds. IEEE, Piscataway, N.J., 719–725. NAKAYAMA, M. K. 1995. Selecting the best system in steady-state simulations using batch means. In Proceedings of the 1995 Winter Simulation Conference. C. Alexopoulos, K. Kang, W. R. Lilegdon, and D. Goldsman, Eds. IEEE. Piscataway, N.J., 362–366. NAKAYAMA, M. K. 1996. Multiple comparisons with the best in steady-state simulations, In Proceedings of the 2nd International Workshop on Mathematical Methods in Stochastic Simulation and Experimental Design. S. M. Ermakov and V. B. Melas, Eds. 230–235. NAKAYAMA, M. K. 1997a. Multiple-comparison procedures for steady-state simulations. Ann. Stat. 25, 6, 2433–2450. NAKAYAMA, M. K. 1997b. Using common random numbers in two-stage procedures for multiple comparisons with the best for steady-state simulations. In Proceedings of the 11th European Simulation Multiconference. A. R. Kaylan and A. Lehmann, Eds. 155–159. NAKAYAMA, M. K. 2000. Multiple comparisons with the best using common random numbers in steady-state simulations. J. Stat. Plan. Inf. 85, 1–2, 37–48. NELSON, B. L. 1993. Robust multiple comparison procedures under common random numbers. ACM Trans. Model. Comput. Sim. 3, 3, 225–243. NELSON, B. L. AND BANERJEE, S. 2001. Selecting a good system: procedures and inference. IIE Trans. 33, 3, 149–166. NELSON, B. L. AND GOLDSMAN, D. 2001. Comparisons with a standard in simulation experiments. Manage. Sci. 47, 3, 449–463. NELSON, B. L. AND HSU, J. C. 1993. Control-variate models of common random numbers for multiple comparisons with the best. Manage. Sci. 39, 8, 989–1001. NELSON, B. L. AND MATEJCIK, F. J. 1995. Using common random numbers for indifference-zone selection and multiple comparisons in simulation. Manage. Sci. 41, 12, 1935–1945. NELSON, B. L., SWANN, J., GOLDSMAN, D., AND SONG, W. M. 2001. Simple procedures for selecting the best simulated system when the number of alternatives is large. Oper. Res. 49, 6, 950–963. NOZARI, A., ARNOLD, S., AND PEGDEN, C. 1987. Statistical analysis for use with Schruben and Margolin correlation induction strategy. Oper. Res. 35, 127–139. PAULSON, E. 1964. A sequential procedure for selecting the population with the largest mean from k normal populations. Ann. Math. Stat. 35, 174–180.

ACM Transactions on Modeling and Computer Simulation, Vol. 13, No. 2, April 2003.

154

•

J. R. Swisher et al.

RINOTT, Y. 1978. On two-stage selection procedures and related probability-inequalities. Communi. Stat. A7, 8, 799–811. ROYSTON, P. 1992. Approximating the Shapiro-Wilk W test for non-normality. Stat. Comput. 2, 117–119. SANCHEZ, S. M. 1997. It is a far, far better mean I find. . . . In Proceedings of the 1997 Winter Simulation Conference. S. Andradottir, K. J. Healy, D. H. Withers, and B. L. Nelson, Eds. IEEE, Piscataway, N.J., 31–38. SANTNER, T. J. 1975. A restricted subset selection approach to ranking and selection problems. Ann. Stat. 3, 334–349. SCHRUBEN, L. W. AND MARGOLIN, B. 1978. Pseudorandom number assignment in statistically designed simulation and distribution sampling experiments. J. Ameri. Stat. Assoc. 73, 504–525. SULLIVAN, D. W. AND WILSON, J. R. 1989. Restricted subset selection procedures for simulation. Oper. Res. 37, 52–71. SWISHER, J. R., HYDEN, P., JACOBSON, S. H., AND SCHRUBEN, L. W. 2004. A survey of recent advances in discrete input parameter discrete-event simulation optimization. IIE Trans., to appear. SWISHER, J. R. AND JACOBSON, S. H. 2002. Evaluating the design of a family practice healthcare clinic using discrete-event simulation. Health Care Manage. Sci. 5, 2, 75–88. SWISHER, J. R., JACOBSON, S. H., JUN, J. B., AND BALCI, O. 2001. Modeling and analyzing a physician clinic environment using discrete-event (visual) simulation. Comput. Oper. Res. 28, 2, 105–125. TEW, J. D. AND WILSON, J. R. 1994. Estimating simulation metamodels using combined correlationbased variance reduction techniques. IIE Trans. 26, 2–16. TONG, Y. L. AND WETZELL, D. E. 1984. Allocation of observations for selecting the best normal population. In Design of Experiments: Ranking & Selection, 213–224. TUKEY, J. W. 1953. The problem of multiple comparisons. Unpublished manuscript. WEN, M. J. AND CHEN, H. J. 1994. Single-stage multiple comparison procedures under heteroscedasticity. Ameri. J. Math. Manage. Sci. 14, 1, 2, 1–48. WILSON, J. R. 2001. A multiplicative decomposition property of the screening-and-selection procedures of Nelson et al. Oper. Res. 49, 6, 964–966. WU, S. F. AND CHEN, H. J. 2000. Two-stage multiple comparisons with the average for normal distributions under heteroscedasticity. Comput. Stat. Data Anal. 33, 2, 201–213. YANG, W. N. AND NELSON, B. L. 1989. Optimization using common random numbers, control variates, and multiple comparisons with the best. In Proceedings of the 1989 Winter Simulation Conference. E. A. MacNair, K. J. Musselman, and P. Heidelberger, Eds. IEEE, Piscataway, N.J., 1112–1120. YANG, W. N. AND NELSON, B. L. 1991. Using common random numbers and control variates in multiple-comparison procedures. Oper. Res. 39, 4, 583–591. YUAN, M. AND NELSON, B. L. 1991. Multiple comparisons with the best for steady-state simulation. ACM Trans. Model. Comput. Sim. 3, 1, 66–79. ¨ YUCESAN , E., LUO, Y. C., CHEN, C. H., AND LEE, I. 2001. Distributed web-based simulation experiments for optimization. Sim. Prac. Theory 9, 1–2, 73–90. ZINGER, A. 1961. Detection of the best and outlying normal populations with known variances. Biometrika 48, 457–461. ZINGER, A. AND ST. PIERRE, J. 1958. On the choice of the best amongst three normal populations with known variances. Biometrika 45, 436–446. Received September 2001; revised July 2002 and December 2002; accepted December 2002

ACM Transactions on Modeling and Computer Simulation, Vol. 13, No. 2, April 2003.