Power Explorer—statistical power by simulation and by formula

1 · Test & hypotheses pick a test, and then say what the truth might be and how sure you want to be

Test

Significance level α Alternative

Direction Sample size n

2 · Null vs. alternative the whole story of one test, on one axis—drag n and watch β shrink

Why a two-sided test at α reads the α/2 column of a printed table

Most textbook t and chi-square tables are one-tail tables: the column header is the probability in a single upper tail. A two-sided test at level α splits its rejection probability across two tails, α/2 in each, and so the boundary you need is the one-tail critical value at α/2—for α = 0.05, you read the 0.025 column, not the 0.05 column. The panel above does this translation for the current settings and shows both values so you can see which one the shaded boundary actually is.

Flipping the toggle to one-sided visibly slides the boundary inward (from the α/2 value to the α value) and power goes up when the truth lies in the favored direction. That gain is exactly why the direction must be chosen before the data are seen: choosing it afterward, to match where the data fell, quietly doubles the real α.

Why the t case needs a “noncentral” distribution

Under the null, the t statistic is the familiar Student t—that is what the table tabulates. Under the alternative, the numerator no longer has mean zero, and so the statistic follows a different law: the noncentral t, indexed by a noncentrality parameter δ = (μ₁ − μ₀)√n/σ that measures how many standard errors the truth sits from the null. Power is the area of that noncentral curve beyond the critical values.

It is not simply a shifted t: dividing by an estimated s skews the curve, which you can check here—the orange curve is visibly asymmetric at small n and settles toward a shifted normal as n grows. Tables never printed the noncentral t (one page per δ), which is why textbooks shipped OC charts instead—see tab ③.

3 · Monte Carlo the histograms overlaid on the plot above are outcomes drawn at random under both the null and the alternative; α sets the detection threshold, and the resulting empirical type-I error, type-II error, and power are summarized here

Replications m Seed Auto-run on change

Convergence of the estimate

Why power is not 1 − α, and what the two shaded errors actually are

α and β live under different curves. α is set by you and spent under the null: it is the chance the test cries wolf when nothing is going on. You do not get to choose β: it is the area of the alternative curve that falls inside the acceptance region, and it depends on everything at once—the effect size, σ, n, α, and the sidedness. Power = 1 − β. Because α and β are areas under different curves, no arithmetic ties power to 1 − α; a test can have α = 0.05 and power 0.10, and the plot above will happily show you one. In the language of machine-learning classifiers, power is the test's sensitivity (its recall, or true-positive rate), β is its false-negative rate, and α its false-positive rate.

The simulation makes the definition concrete: each replication draws a full dataset from the alternative, runs the test exactly as you would on real data, and records reject or not. The rejection fraction estimates power. A second, independent run under the null estimates the attained α—for a well-calibrated test it should hover near the nominal level, and for the discrete binomial test it sits visibly below.

What the convergence trace demonstrates

The orange line is the running rejection fraction after each batch of replications; the band is its 95% (Wilson) confidence interval, which narrows like 1/√m. Where a teal line is drawn, it is the exact answer from distribution theory, and the estimate settles onto it because the formula and the simulation are two routes to the same number. The formula is faster and exact, but it exists only for tests whose statistic has known distribution theory under the alternative. The simulation needs nothing but the ability to generate data, which is why it still works on tab ⑤, where no formula exists at all.

5 · Take it with you the same analysis, in the tools you will actually use

Every snippet above leans on a closed form. The fully empirical route—simulating both hypotheses from a model of your own and counting rejections—needs no formula at all; tab ⑤ ends with that pattern as a take-home template.

Power vs. n and power vs. effect at the current settings, with the seed and every parameter recorded in the header, and so any number here can be regenerated.

Operating-characteristic chart the printed chart your textbook uses, drawn live for the current test

An OC chart plots β—the probability of failing to reject—against the standardized effect, one curve per sample size. Before software, this was how a power analysis was done: pick the curve for your n, run a finger up from your effect size, read β off the axis. Do that lookup here, and then notice it is the same point as on the power curve of tab ② flipped upside down: power = 1 − β. In classifier terms, an OC chart plots a false-negative rate, and reading power off it is reading sensitivity.

Look up: d = β = n =

The effect and β are the two coordinates of the orange dot; n is the answer the chart gives back, the smallest sample size whose curve passes at or below the dot. That curve is drawn in orange. Typing an n instead runs the lookup the other way, putting the dot on that curve to read its β. On the chart itself, drag the dot to move both coordinates at once, or drag either dashed projection line to move one coordinate with the other held. Reading the chart never moves the lookup: with a mouse, hovering anywhere reads off the β, effect, and implied n at that point, and with a finger, touching anywhere does the same. The reading follows the pointer across the chart, and it clears when the cursor leaves or the finger lifts.

What a printed OC chart is a slice through

Power is a function of four things at once—effect size, n, α, and sidedness—and a sheet of paper can hold only two axes. A printed OC chart freezes α and the sidedness (that is why textbooks print separate charts for α = 0.05 and α = 0.01, one-sided and two-sided), spends its axes on effect and β, and handles n by printing a family of curves. This page draws the same slice live—change α or the sidedness in tab ① and the whole chart redraws, which is the manipulation no printed chart can offer.

Reading β at your effect and n here, and reading power on tab ②, are the same lookup: every OC chart is a power curve drawn upside down.

Plan from pilot data a few real observations replace guessed planning values

A power analysis is typically done to determine the number of samples that an experiment will need. However, the analysis requires information about the variance in each population being sampled, and those variances are often unknown before the experiment has been run. Consequently, a power analysis often depends on a pilot study, a small sample collected first to estimate exactly those variances.

An example for parametric testing of means. Imagine that your goal is to test whether a population's mean deviates significantly from a null hypothesis (a one-sample t-test). For a power analysis, you may have to run a pilot study to sample enough individuals from that population to estimate its standard deviation. Similarly, if you paired individuals from two populations to estimate the mean difference in a response to a shared stimulus, a power analysis would require pilot data on the standard deviation of the paired differences. Pairs like these can come from: matched individuals in a real experiment; a real system's outputs paired against a simulation driven by the same inputs, as in validation; or—the special case explored at the bottom of this page—two configurations of a simulated system sharing common random numbers.

The textbox below gathers data for both cases: a single list of numbers, written down a column or across one line, or two columns (comma, tab, or space separated) of paired data that differences will be taken over. Either summary can be sent to tab ① for the power analysis of a one-sample t-test (i.e., for either raw values or the differences between each pair). A two-column summary can also be sent to the bottom of this page for the power analysis of a paired t-test and an investigation of how correlation within pairs can be leveraged for more efficient simulation studies—fewer runs for the same power.

Summary cards appear here once a pilot is pasted, each with a button that sends its planning value where it belongs.

Comparing two configurations with paired runs why correlated experiments need fewer runs—the statistics behind common random numbers

To compare two configurations of a system, run both under the same input conditions and test the paired differences D = X₁ − X₂ with the one-sample t-test of tab ①. What the pairing buys is variance: the differences have

σ_d = √(σ₁² + σ₂² − 2ρσ₁σ₂).

So positive correlation ρ between the paired observations shrinks σ_d, and every bit of shrinkage is pure power—or, run the other direction, fewer runs for the same power. In contrast, the method of antithetic variates induces negative correlation between paired runs to reduce the variance of their average. Whereas common random numbers aim to reduce the runs needed to detect a small difference between two systems, antithetic variates aim to reduce the runs needed for a high-precision estimate of a single system's performance—the σ-of-a-pair-mean readout below tracks that side of the story.

σ₁ σ₂ difference to detect Δ pairs n target power

correlation ρ Uses the α set on tab ①, two-sided.

Power of n pairs, as ρ runs from −1 toward 1

Pairs needed for the target, against ρ

ρ > 0 is common random numbers. Driving both configurations with the same random-number streams makes their outputs rise and fall together, and so the noise cancels in the difference—that is the right half of both plots, where σ_d collapses and the required n falls. ρ < 0 is the antithetic idea, pointed at a different target. Antithetic variates negatively correlate paired runs of one configuration so their average is stable: Var(½(X + X′)) = ¼(σ² + σ² + 2ρσ²) shrinks as ρ goes negative—the same formula with the sign of the cross-term flipped because a sum wants negative correlation exactly where a difference wants positive. Both are variance-reduction techniques in stochastic simulation; this panel is the power analysis that says what they are worth.

Power with no formula: a gamma-regression slope the simulate-and-count definition is the only route here—and it is enough

Power for generalized linear models—logistic, Poisson, or gamma regression—generally has no clean closed form, and simulation is the standard professional route. This panel runs the Wald t-test of H₀: β₁ = 0 in a gamma regression, the GLM for a positive, right-skewed response such as a service time or a repair cost, with mean exp(β₀ + β₁x) and shape k. It runs through exactly the same Monte Carlo engine as tab ① (same seed field, same counting); the only difference is that no teal line will appear on the trace because there is no exact value to draw. The estimate and its interval are the answer.

β₀ true slope β₁ shape k n replications m Unlike the tests of tab ①, there is no null value to type: a slope test's null is β₁ = 0 by convention, and “true slope” is the truth the null misses. x is a fixed, standardized design.

Convergence—with no exact line to converge onto

Two things the simple cases hide

Multiple regression. The power to detect one coefficient depends not only on that coefficient but on how correlated its predictor is with the other predictors: correlation inflates the standard error of the estimate (the variance-inflation factor), and so the same slope can be easy to detect in one design and hopeless in another. Sample-size recipes that take only “the effect” are silently assuming uncorrelated predictors. R handles the general case through pwr::pwr.f2.test (Cohen's f² formulation) and Python through statsmodels.stats.power.FTestPower; both leave translating your design into f² to you, and simulation sidesteps that translation entirely.

Multiple comparisons. When a study tests many hypotheses at once, the per-comparison α must shrink for the family-wise error rate to stay at the nominal level—Bonferroni divides α by the number of tests; Tukey and Dunnett are the sharper procedures for all-pairs and against-a-control comparisons—and power falls accordingly, and so the n from any single-test analysis on this page is an underestimate for a multi-comparison study. This page does not compute family-wise power.

Where to go next pick the tool by what the problem needs, not by what is installed

Tool	Covers	GLM power (logistic, Poisson, gamma)	Multiple regression	Multiple comparisons
MATLAB `sampsizepwr` Statistics and Machine Learning Toolbox	the z, t, two-sample t, variance, and proportion tests of tab ①—exactly the five this page labels with a MATLAB id—solving in either direction depending on which argument is left `[]`	no—simulate (the toolbox's `fitglm` in a loop)	no	no
R: `stats` + `pwr`	`power.t.test`, `power.prop.test`, `power.anova.test` in base; the `pwr` package adds chi-square, correlation, and general f² cases	no standard function—simulate with `glm` in a loop	yes—`pwr::pwr.f2.test`	no family-wise power; adjusted-α power via the same functions with α/k
Python: `statsmodels.stats.power`	t, two-sample t, z, ANOVA, chi-square GOF classes, each with one `solve_power` that solves for whichever argument is `None`—the two directions of this page in one call	no standard function—simulate with `statsmodels` GLM in a loop	yes—`FTestPower` with f²	no family-wise power
*GPower** standalone, point-and-click	the standard test families (t, F, χ², z, exact), a priori and post-hoc, with plots—the tool students meet in psychology and biology courses	logistic and Poisson regression under its “z tests” family (approximations, not simulation)	yes—fixed-model R² cases	per-comparison α only

When none of these fit—a hierarchical model, a simulation model with no likelihood, a custom decision rule—the method of tab ① always does: write the data generator, run the test, count rejections. Past the textbook cases, that is how power analysis is done. The section below hands you that loop as a template.

Take it with you the simulate-and-count pattern, for a model of your own

Everything on this page reduces to one pattern: a test is a classifier, and a power analysis measures that classifier on labeled data. Generate datasets from the null model, generate more from the alternative, run the same decision rule on both, and count—the rejection rate on the null draws estimates the attained α, and the rejection rate on the alternative draws estimates power. The template arrives filled in with tab ①'s default t test so its output can be checked against this page (exact power 0.5645); replace the two generators and the classifier with your own model—a queueing simulation, a custom statistic—and nothing else changes.

Language

To solve for n instead, wrap the count in a loop that raises n until the power estimate clears the target—that is all any solver does. Each interval is that language's own exact binomial 95% CI; quadrupling m halves its width.