A confidence interval is a visual representation of many hypothesis tests all at once. Here, because we know the population mean ahead of time, we can confirm that a 95% confidence interval captures the mean 95% of the time.
—true density and μ▮sample histogram and rug⊢●⊣interval and its mean (solid teal captures μ; dashed red, hollow dot, misses)
Sample mean x
—
True μ
5
Sample std. dev. S
—
True σ (σ²)
0.1 (0.01)
tα/2,n−1 at 95%
—
Half-width H = t·S/√n
—
Captures μ?
—
Interval x ± H, 95%
—
Each run draws n values from the normal population by inverse-transform sampling, computes x and S, and forms the interval x ± t·S/√n. The population is known here on purpose: it is the only way to check whether an interval did its job. Run many experiments, and watch the tally settle near α, the fraction that the confidence level promises will miss. Drag the dashed μ line to move the mean, or drag the curve itself to change σ; the fields above follow as you drag, and the last sample stays put so you can see the population move beneath it. The x-axis stays fixed so a change of σ shows as a change of width; μ can move only within it, and Clear history re-centers the axes on the current population.
Experiment history – confidence intervals for μ
Change a parameter between runs to compare interval widths across the divider, or drag the dashed line to move μ.
Where the interval comes from and what changes its width
A one-sample t test at level α fails to reject the hypothesized mean μ whenever |x − μ| ≤ tα/2,n−1·S/√n. Solving that inequality for μ gives the set of means the data cannot rule out, which is the confidence interval. An interval that excludes a hypothesized value is the same thing as a rejection at level α.
Half-width:H = tα/2,n−1 · S / √n
Interval:[ x − H , x + H ]
Guarantee:Pr( μ ∈ [x − H, x + H] ) = 1 − α
Lower α: wider, misses lessRaising the confidence level raises t. The interval widens, and fewer of them miss. At 99.9% almost none miss; at 40% most do. This is the only control that changes the miss rate.
Larger n: narrower, same miss rateH shrinks like 1/√n, and t itself falls toward 1.96 as the degrees of freedom grow. Four times the samples halves the width. The fraction that miss stays at α.
Larger σ: wider, same miss rateH grows in proportion to the spread of the population because S estimates σ. Doubling σ doubles the width, and again the fraction that miss stays at α.
Common Random Numbers (CRN) uses statistical blocking to apply the same input combination to every system under test so that the differences between systems can be isolated from the differences across inputs.
●system 1 / input 1○system 2 / input 2—joins a pair (same replication, bottom row only)●paired differences dᵢ = y₂ᵢ − y₁ᵢ┄true means—zero⊢●⊣interval for δ (teal captures, red dashed misses)*excludes zero
System 1: y₁ (S₁)
—
True δ = E[h₂] + shift − E[h₁]
—
System 2, independent inputs: y₂ (S₂)
—
System 2, common inputs: y₂ (S₂)
—
Welch half-width for δ (df —)
—
Paired half-width for δ (df —)
—
Correlation of paired outputs r
—
Does CRN reduce variance?
—
CRN variance Sd²
—
Indep. input variance S₁² + S₂²
—
Welch p-value against δ = 0
—
Paired p-value against δ = 0
—
Each replication draws a uniform random number for system 1 and one for system 2; the two arms differ only in whether system 2's number is its own (independent inputs) or the same one system 1 got (common random numbers). Both arms use the same 2n runs. When the two systems respond to their input in the same direction, sharing it makes their outputs rise and fall together, the shared part cancels in the difference, and the paired interval is narrower and more often excludes zero, which is the evidence that the systems differ. A star above an interval marks one that leaves out zero, which is the same claim as a p-value below α: the systems differ.
Common random numbers (paired):d ± tα/2,n−1 · Sd/√n (dᵢ = y₂ᵢ − y₁ᵢ)
Both intervals are for the same δ; they differ in what they treat as the sample: two independent sets of outputs or one set of n differences.
Why sharing inputs narrows the interval and when it widens it
Variance of the difference of means:Var(ȳ2 − ȳ1) = (S1² + S2² − 2 r S1 S2) / n (r = 0 for independent inputs)
Paired:Var(d) = Sd² / n
And identically:Sd² = S1² + S2² − 2 r S1 S2
The color of an interval answers "did it capture the true δ?"; the star answers "does it rule out δ = 0?". The two are different questions, and a red interval can carry a star. With the shift set to 0, the systems do not differ, and about α of the runs still earn a star: those are the false positives.
Positive correlation shrinks the difference's variance: two systems reading the same input tend to rise and fall together, and so their difference is more stable than two systems reading independent inputs. Once the arms share inputs, the paired analysis is mandatory because the two samples are no longer independent; the right tool is the one-sample interval on the n independent differences, which is the paired t test in its interval form. A pilot run can check whether pairing helps. Run both systems on common inputs and compare Sd² against S1² + S2²: if Sd² comes out smaller, pairing is reducing variance, and the ratio of the two says by how much.
Same model, shiftedWith both systems built from the same model, sharing the input drives the correlation of paired outputs close to one: the outputs rise and fall together almost perfectly. Nearly all of the variance cancels in the difference, and the paired interval collapses toward the shift itself.
Opposite responsesexp(u) rises with u, whereas 1 / (1 + u) falls. Sharing u pairs one system's high outputs with the other's low ones, the correlation of paired outputs turns negative, and the paired interval comes out wider than Welch's. Common random numbers assume the two systems respond to their input in the same direction; pick models that disagree, and pairing backfires.
The bowlThe bowl model, (u − ½)², has no consistent direction: it falls for u below one half and rises above it. Paired with a monotone system, the bowl's own response has no matching direction to correlate with, and so the correlation of paired outputs stays near zero and pairing does almost nothing beyond independent inputs.
Antithetic variates are designed to create negative correlation between pairs of simulation outputs to accelerate the natural averaging that comes from the Law of Large Numbers. Even runs use PRNG draws (1 − u) that are the complement of the draws (u) from the prior odd run.
Output-type shown:
True mean E[h(U)]
—
Independent mean (n pair means)
—
Antithetic mean (n pair means)
—
Independent half-width (df —)
—
Antithetic half-width (df —)
—
Sample corr. of the pair outputs y, y′
—
Does AV reduce variance?
—
AV variance Sz²
—
Indep. pair variance Sw²
—
One run, in draw order: the 2n outputs and their pair means
Both arms evaluate the model 2n times and share the first n draws, each pairing them with a second draw and averaging the pair: the independent arm's second draw is fresh, the antithetic arm's is the mirror 1 − u, and so the two differ only in where that second draw comes from. In practice, all 2n values would be used, with 2n − 1 degrees of freedom; the pairing only makes the comparison one-for-one, and the antithetic arm's sample is its n pair means. When h is monotone, a high draw pairs with a low one, the pair means cluster around the true mean, and the interval narrows. Each output can also carry noise, zero by default, which the complement cannot cancel, and so the antithetic interval narrows only by the share of variance that comes through u; turn the noise up, and the two arms converge.
Experiment history – confidence intervals for E[h(U)]
Independent draws (n pair means)
Antithetic pairs (n pair means)
Why negatively correlated pairs narrow the interval
Pair mean:Z = (Ya + Yb) / 2, E[Z] = E[Y]
Its variance:Var(Z) = ( Var Ya + Var Yb + 2 Cov(Ya, Yb) ) / 4
With antithetic inputs:Ya = h(U), Yb = h(1 − U)
Averaging two values adds no bias, and the variance of the average falls below half the variance of one value exactly when the two are negatively correlated. Two independent draws never are. But U and 1 − U are perfectly negatively correlated, and if h is monotone (rising or falling throughout), a large h(U) comes with a small h(1 − U), and so the outputs inherit a negative correlation from the inputs. Each pair is one independent observation, and so the n pair means go through the ordinary interval formula, and the generator is only consulted n times for 2n model evaluations.
Pick the bowlFor h(u) = (u − ½)², the mirror 1 − u gives exactly the same value, and so the antithetic pair mean is the value itself, with none of the variance-canceling that a true mirror provides. The independent arm still gets the ordinary variance-halving of averaging two unrelated draws, and so its interval comes out narrower here. Monotonicity is the whole trick.
The limiting caseA model that is a linear function of a normal inverse transform, h(u) = μ + σΦ⁻¹(u), has h(1 − u) = 2μ − h(u): every pair mean is exactly μ, and the interval has zero width. Pick u itself: every pair mean is exactly 0.5, and the interval is a single point. Real models sit between that and the bowl.
Antithetic and common random numbersBoth manipulate inputs to induce correlation. Common random numbers want positive correlation between two systems' outputs so their difference is stable; antithetic variates want negative correlation within one system so an average is stable.
Control variates use knowledge about how extreme draws from an input model are relative to the mean of that model in order to isolate explained variance (explained by inputs) from model variance.
c = —
True mean E[h(U)]
—
Plain mean y
—
Control-variate mean z
—
Plain half-width (df —)
—
Control-variate half-width (df —)
—
Estimated c* = Suy / Su²
—
R²: share of y var explained by u
—
The figure above shows one run of n draws as three panels in draw order: how far each input lands from its known mean 0.5, each output with the part that tracks that deviation marked on the pin, and what is left after subtracting that part, which keeps the same mean as the output but has less spread because the part u predicts is gone. What remains is the part u cannot predict, including any output noise; a run whose u values come out high still has a high y average for that reason, but the adjusted values do not. Drag the slider to move c away from its estimated value and watch the bottom panel and its interval respond; every c leaves the mean alone, and so the interval stays valid at any setting.
Experiment history – confidence intervals for E[h(U)]
Plain average of h(u)
Control-variate estimate z = h(u) − c·(u − 0.5)
Why a correlated quantity with a known mean reduces variance
Adjusted variable:Z = Y − c·(X − E[X]), E[Z] = E[Y] for every c
Its variance:Var(Z) = Var Y − 2c·Cov(X, Y) + c²·Var X
Best c:c* = Cov(X, Y) / Var X, Var(Z) = Var Y · (1 − ρ²XY)
Any c leaves the mean alone because the term being subtracted has mean zero. The variance is a parabola in c with its minimum at the regression slope of Y on X, and at that slope the fraction of Var Y that survives is 1 − ρ². Here, ρ² is the true share explained, and the R² a run reports estimates it from that run's own data. The more strongly the control co-varies with the output, the larger the reduction. Here the control is the model's own input, but in a simulation it can be anything with a known mean that the output responds to: the average of the sampled service times, say, whose mean is a parameter of the input model.
Pick the bowlFor h(u) = (u − ½)², the output is symmetric in u, and Cov(U, h(U)) is exactly zero. The fitted slope wobbles around zero from run to run, and the interval does not shrink. A control has to co-vary with the output to help.
Estimating c undershoots a littleThe slope is fitted from the same run it corrects, which makes the interval slightly optimistic. At n = 50, the capture rate is about two points under nominal; at n = 10, it runs about ten points under, near 83% for a nominal 95%. Larger runs, or a c fixed from a pilot, remove the shortfall.
The same idea as regressionThe control-variate estimate is the fitted value of the regression of Y on X at X = E[X]. Two or more controls generalize to a multiple regression in the same way.
What the control cannot removeThe control subtracts only the part of y that a straight line in u predicts. With no noise, that line explains all of u, about 98% of exp(u), 97% of 1/(1 + u), and 96% of √u, but none of the bowl; what remains is the curvature of h. Noise adds a second part no function of u can predict, and so R² falls as σe rises and the two intervals converge. A model closer to linear, or a quieter one, gives the control more to remove.
Importance sampling allows for estimating rare events by concentrating inputs only on “important” cases that are likely to generate those rare events. To recover the right proportions, likelihood ratios reweight the biased outcomes.
Sample shown:
True p = P(h(X) ≥ T)
—
Direct: events in n
—
Weighted: draws in the event
—
Direct estimate ± H
—
Weighted estimate ± H
—
Effective sample size ESS
—
Likelihood weighting
hist[h(X̃)] · W(X̃) → hist[h(X)]
Both approaches draw n samples; direct sampling runs the model on standard normals and counts the outputs beyond T. Importance sampling draws inputs from a proposal distribution N(θ, 1) (also called the biasing or importance distribution), runs the same model, and multiplies each output beyond T by the likelihood ratio W(y) = φ(y)/φ(y − θ). The pair of plots above counts the proposal's outputs once and by W(y), turning its histogram into the target's; the tail past T comes from the draws the proposal put there. Move the pointer across those three plots to read a bin's count, its weight, and its weighted count together. For a monotone model, the output event is the input event x ≥ h⁻¹(T), and so the shift aims there. The threshold line, its pre-image, and the proposal curve above can all be dragged: the T line vertically, the others horizontally.
Experiment history – confidence intervals for p = P(h(X) ≥ T)
Direct sampling (mean of the indicator)
Importance sampling (mean of the weighted indicator)
Why sampling from the wrong distribution, corrected by a weight, gives the right answer
Effective sample size:ESS = (Σ W)² / Σ W² over the draws in the event
The identity holds for any proposal density that is positive wherever the target is, and so the sampler is free to put its draws where the interesting outputs come from. The weight lives on the input and undoes exactly the bias introduced there, regardless of what the model h does afterward. The variance, though, depends on the choice of θ. Weights near one across the input threshold give a tight interval, whereas a shift far past it makes the few draws nearest the edge carry weights thousands of times larger than the rest, the effective sample size collapses, and the interval widens again; pushed far enough, that same shift makes the interval unreliable, and it can even run below zero for a quantity that is a probability. Slide θ past the input threshold, and watch the effective sample size in the stats fall.
Where the interval failsDirect sampling fails when the sample is all zeros: the interval collapses to the single point [0, 0], wrong every time. Importance sampling fails when θ sits too far past the input threshold, and so the weights grow uneven and the interval narrows while it misses; ESS is the warning sign for that failure.
The same interval formulaNothing changes downstream: the weighted values are independent, their mean is p, and the one-sample t interval captures p about 100(1 − α)% of the time. Importance sampling is a change of what is averaged, not of how the average is analyzed.
Bias on purposeThe other three techniques leave the sampling distribution alone and reduce variance by correlation or by a correction term. The target distribution almost never produces the event of interest; this one deliberately samples from a proposal distribution instead and corrects for it with the weight.
Aim on the input scaleθ lives on the input, not the output, and so it has to aim at the input threshold, not at T itself. For exp(x), the input threshold is ln T, not T: at the default T = 20 that is about 3.00, whereas aiming θ at 20 directly would center the sampler on inputs where exp(x) is astronomically larger than the threshold, wasting nearly every draw and leaving the few that do land near the input threshold carrying enormous, uneven weights.
Two input thresholdsx² reaches T from two input thresholds, ±√T; a one-sided shift covers one and never visits the other, and so the weighted estimate settles at half the truth, with a narrow interval that misses every time, and nothing in the sample warns: ESS looks as healthy as on the identity because the uncovered side is simply absent, not represented by a few huge weights. The estimator is unbiased only in the sense that an astronomically rare left-side draw would carry an astronomically large weight. The fix is a sampler that covers both input thresholds, such as a two-component mixture.