This tab simulates the null hypothesis of one design that performs exactly at its baseline on every one of K responses, and it estimates how often the design will be falsely reported as differing on at least one of them. One design is simulated R times and judged on K responses (a mean wait, a utilization, a fill rate, and so on), each against its own baseline, and the report is one yes or no: does the design differ from the baseline on any response? Those K tests are a family because the report is wrong whenever any one of them is wrong. Every response's true mean sits exactly at its baseline, and so every flag is a false alarm; the fraction of runs in which the design is reported as differing is the family's Type-I error rate.
Judge K responses at level α each, and the chance that at least one verdict is wrong, called the family-wise error rate, climbs with K: the more responses you test, the more often one of them differs by chance alone, and looking through all of them for any difference is fishing. This tab counts that rate directly, and the Bonferroni correction control shows what holding it at α does to every interval.
Had the question been which of K separate responses differ from their baselines, each judged and acted on alone, each verdict would carry its own α and no family correction would be wanted: the expected number of wrong verdicts would be αK, and shrinking every test to α/K would only turn false alarms into missed detections. A family matters only when one decision rests on all K tests, as it does here, where the report is a single yes or no, and wherever designs are compared with each other.
The K intervals here ask K questions, and each carries its own α, which is why they need a correction. A multivariate test (Hotelling’s T² for one design, the multivariate form of the one-sample t test) asks one question instead: do the design’s K true means all sit at their baselines? One test holds α on its own, with no correction, and so it is the one-shot answer to this page’s problem. Like the F test of one-way ANOVA, it says only whether something moved, not which response, and a rejection still has to be followed by a look at the responses one at a time, where the same family problem waits.
With several designs and several responses, the same idea is MANOVA, one-way ANOVA for several response variables at once: one test of whether any design differs on any response, at one α, in place of K separate ANOVAs, one on each response, which would each carry an α of their own.
Comparing designs means intervals on differences: a benchmark against every other design gives K − 1 of them, and every pair gives K(K − 1)/2. The intervals share data, and so the independent-comparisons formula is no longer exact, but the Bonferroni bound needs no independence at all, which is why it is the textbook's choice when designs are run with common random numbers. Turn common random numbers on, and every difference interval narrows to about half its width, and so a planted difference is detected far more often at the same family-wise error rate: the answer to a conservative correction is variance reduction, not a looser α.
After simulating K designs, the natural report is the design with the best sample mean and its confidence interval. This tab asks how often that interval contains the winning design's true mean. An ordinary 95% interval contains the true mean 95% of the time when the design was named before the data were seen. Here, it was named after: it won because its sample mean came out best, and a sample mean comes out best partly through luck, because the best of K sample means is the extreme of K draws. That luck is part of the winner's reported mean, which therefore overstates how good the design is, and its interval, centered on that mean, misses the true mean more often than α.
The tab counts the misses directly, and the Bonferroni level puts the coverage back. The gap between the winner's sample mean and its true mean, the selection bias, is reported in its own card.
A correction that makes false alarms rarer also makes true detections rarer. Plant a real difference and watch how often each rule finds it. Holm's step-down keeps the same family-wise guarantee as Bonferroni and gives some of the power back, and because it works on the sorted p-values rather than on the intervals, this is the tab where p-values get a column of their own. Holm's first threshold is Bonferroni's, and so it helps only once more than one comparison is real. At every pair among ten designs, 45 comparisons, each Bonferroni interval is about 1.8 times as wide as the uncorrected one, and only large differences are still detected; that loss is what motivates a one-shot test such as tab ⑤, ANOVA and Post Hoc Tests.
One-way ANOVA's logic runs in two steps: the F test asks whether any design differs, and a post hoc test asks which. The F test holds α without a correction, but it does not say which designs differ, and each post hoc test here is a single critical difference on the sorted means: Tukey's honestly significant difference from the studentized range, Fisher's least significant difference run only after a significant F, and Bonferroni's t at α/C. Bonferroni-corrected pairwise intervals (the ones tab ② builds) are themselves a legitimate post hoc test, which is why they sit beside Tukey's and Fisher's here. With several response variables in place of one, one-way ANOVA becomes MANOVA, one test of whether any design differs on any response.
Tukey's honestly significant difference and Bonferroni's t both judge every pair at once, and the two critical-difference cards above show the gain is real but modest: Tukey's is the smaller of the two, never the larger, but rarely by much.
Fisher's protected LSD holds the family-wise error rate at α only when there are exactly three designs (Hayter, 1986); with five, and with the last two designs planted δ above the rest by default, F almost always flags a difference, and so the protection does nothing and the four null pairs (three among the unmoved designs and one between the two moved ones) are each tested at a bare α. Watch its family-wise error rate above settle well past α, whereas Tukey's and Bonferroni's do not.
ANOVA and all three post-hoc rules here assume every design is sampled independently of every other, an assumption common random numbers break by design: each rule pools a within-design variance across designs that no longer vary independently, Bonferroni's rule included. Under common random numbers, a comparison instead falls back to the paired interval at α/C that tab ② builds, where the Bonferroni inequality still holds because it needs no independence at all. When every comparison is against one benchmark design rather than among all of them, Dunnett's test is the exact many-versus-one counterpart of Tukey's rule here.
One-way ANOVA is a linear model: each output is the grand mean, plus a shift for its design, plus noise. The K design shifts are free parameters tied only by their average, which is why the model can ask only whether the designs differ and then, pair by pair, which, and why one residual mean square feeds every post hoc rule above.
The differences among the K designs often come from choosing different settings of a common set of parameters, not from K unrelated alternatives. When the designs are the levels of one factor, such as one to five servers, a model with a term for that factor asks one question, whether the response rises with the factor, in place of K(K − 1)/2 pairwise ones; when they are the cells of a factorial, such as two queue disciplines crossed with three staffing levels, it asks for each factor’s effect and their interaction, three questions in place of fifteen. Most of the multiple-comparisons burden disappears because the question changed, not because the correction improved, and the fitted model also predicts designs that were never simulated. Fitting such a model to simulation output is called metamodeling.
Independence across designs is an assumption of the one-way model, not of ANOVA as a whole. When every design’s r-th replication shares one random-number stream, the replication is a block, and a model with a term for each replication makes every comparison within a replication, as tab ②’s paired intervals do. The correlation that common random numbers create is then part of the model rather than a violation of it.
A fixed-sample test cannot add replications where they turn out to be needed. This two-stage procedure can. A pilot stage of R₀ replications per design screens out designs that are clearly worse, a second stage adds replications only to the survivors, as many as each one's variance and the practically significant difference ε call for, and the survivor with the best mean is selected. With probability at least 1 − α the selection is the best design or one within ε of it.
The columns in the chart above differ in height because the number of replications the second stage draws varies from run to run. A fixed-sample test such as one-way ANOVA draws the same number of replications from every design no matter how the data turn out; it cannot add replications where they turn out to be needed. This procedure can: its second stage adds replications only to the survivors of screening and only as many as each survivor's own variance and ε call for.
With probability at least 1 − α/2, the half of the error budget the screen takes, the best design is among the survivors of step 2. Some tools, such as Arena's Process Analyzer, stop there: a best-scenario chart draws one box plot per design, highlights every design the screen cannot rule out as a possible best, often several, and takes an error tolerance as ε. The procedure here goes on to size those survivors and pick one.
A screening-only variant of the procedure stops after step 2 and selects directly among the survivors, using a t quantile at the upper-tail probability 1 − (1 − α)^(1/(K−1)) instead of splitting the guarantee between two stages. A selection-only variant skips screening altogether and takes every design to the same second-stage size using Rinott's constant h(R₀, K, 1 − α) because the whole guarantee then falls on that one stage.
The 1 − α guarantee is proved for normal data sampled independently across designs, and the procedure is also used with common random numbers in practice.
Ranking and selection decides among a handful of named designs. When a design is a point in a parameter space, such as a staffing level, a reorder point, or a batch size, the candidates cannot be enumerated: the search itself decides where to simulate next and how much, and every estimate it compares carries noise.
Metaheuristics search by trial and improvement rather than by gradient. Scatter search paired with tabu memory is the engine inside OptQuest, which Arena, AnyLogic, and other simulation packages license. The other common families take their moves from nature, each keeping a population or a search history that steers the next round of candidates away from ground already covered:
Driven by a simulation, every one of these evaluates a noisy objective, and so each has a version that re-evaluates its elites, averages replications before comparing, or hands its finalists to ranking and selection. A wider survey of metaheuristics lists many more.
The methods above treat the simulation model as a black box that can only be probed. An alternative approach is to build a meta-model of the simulation model that helps to identify generalizable structure across the parameter space. For example, response-surface and stochastic-approximation methods fit a local model of the objective and follow that model toward an optimum.
Ranking and selection fits into the loop in two places. Page ⑥'s two-stage procedure is the clean-up step, deciding among the search's finalists with a stated guarantee once the search itself has narrowed the field; a lighter screening rule can also sit inside the loop, stopping replications on a candidate that is already clearly worse than the others.
The search never sees a candidate's true expected performance. Every value it compares is a sample mean from a finite number of replications. The candidate with the largest sample mean is partly the candidate that happened to draw favorably, and so its sample mean overstates its true performance, as page ③ shows for a fixed set of designs. This is why the loop's last step re-estimates the chosen candidate on fresh replications before reporting it: a sample mean and interval computed from new replications are not biased by the selection that picked the candidate.
Every comparison the search makes is one more hypothesis test, and a search makes thousands of them. If the rule were simply to move to any candidate whose interval excludes the current best's at α = 0.05, then one comparison in twenty between two equally good candidates would move the search to the wrong one. Pages ①, ②, and ④ measure exactly this accumulation of false alarms. The search rules above therefore do not decide each comparison once: they keep replicating candidates that are close and drop only candidates that are clearly worse.
Pages ①–⑤ compare a fixed set of designs against fixed samples from each. Page ⑥ adds replications adaptively but still to a fixed set of designs chosen in advance. This page searches an open set: the design space itself is not enumerated, and the statistical tools built on the earlier pages (confidence intervals, corrections, and a screening rule) appear here again, now as components inside a search loop rather than as the whole procedure.