Lesson 3: Hypothesis testing fundamentals

Lesson 2 showed that a sample statistic almost never lands exactly on the true population parameter, purely because of sampling error. Hypothesis testing is the formal procedure Green Belts use to decide whether an observed difference, such as a change in average work-order cycle time after a process change, is likely a real effect or just that ordinary sampling noise. Every hypothesis test in this course, from the t-tests in the next lesson through the chi-square tests three lessons later, follows the same underlying logic this lesson introduces.

Null and alternative hypotheses

Every hypothesis test starts by stating two competing claims. The null hypothesis (H0) is the default, "no difference" or "no effect" claim, such as "the average work-order cycle time is unchanged." The alternative hypothesis (Ha) is the claim a team suspects might actually be true, such as "the average cycle time has changed." A test never proves H0 true; it only evaluates whether the sample data provides strong enough evidence against it to reject it in favor of Ha. If the evidence is not strong enough, the correct conclusion is to fail to reject H0, never to say H0 has been "accepted" or "proven," because absence of strong evidence against a claim is not the same as evidence that the claim is true.

The p-value decision rule

A hypothesis test produces a p-value: the probability of observing data at least as extreme as what was actually collected, if H0 were true. A small p-value means the observed data would be unlikely under H0, which counts as evidence against it. The decision rule compares the p-value to a pre-chosen significance level, alpha (α), commonly set at 0.05: if p is less than alpha, reject the null hypothesis; otherwise, fail to reject it.

Worked example. A team compares average work-order cycle time before a process change (5.4 days) against after the change (4.9 days), runs a t-test, and gets p = 0.021, with alpha set at 0.05 before the test was run. Since 0.021 is less than 0.05, the team rejects H0 and concludes the change in average cycle time is statistically significant.

Practical vs. statistical significance

A statistically significant result only says the observed difference is unlikely to be pure sampling noise; it says nothing about whether that difference is large enough to matter operationally. With a large enough sample, even a tiny, practically meaningless difference can produce a very small p-value. In the cycle-time example above, whether a 0.5-day improvement is worth the cost of rolling the process change out program-wide is a practical significance question, answered with program knowledge and cost data, not with the p-value alone. The reverse problem also happens: a real, practically important difference can fail to reach statistical significance if the sample size is too small to detect it reliably.

Type I error, Type II error, and power

Because a hypothesis test is a decision made from incomplete information, two kinds of mistakes are possible, shown in the figure above. A Type I error is rejecting a null hypothesis that is actually true, a false positive, with probability equal to alpha. A Type II error is failing to reject a null hypothesis that is actually false, a false negative, with probability called beta (β). Power is the probability of correctly detecting a real effect when one exists, defined as 1 - β; higher power means a lower chance of missing a real improvement or a real problem.

Sample size is the main lever a Green Belt controls before running a test. Increasing sample size reduces standard error, which increases power (and so lowers β) for a given effect size and alpha, without changing alpha itself. This is the practical reason pilot studies with very small sample sizes are risky: even a real, meaningful improvement can be missed simply because the study was underpowered to detect it.

One-tailed vs. two-tailed tests

A two-tailed test checks for a difference in either direction (Ha: the mean is not equal to some value), splitting alpha between both tails of the distribution; at alpha = 0.05, each tail gets 0.025, which for a z-test corresponds to a critical value of about ±1.96. A one-tailed test checks for a difference in only one specified direction (Ha: the mean is greater than, or less than, some value), placing all of alpha in a single tail; at alpha = 0.05 that gives a critical value of about 1.645 on that one side. A one-tailed test is only appropriate when the question genuinely only cares about one direction, decided before looking at the data, such as "did the defect rate decrease" when an increase would never change the decision either way; choosing one-tailed after seeing the data to make a marginal result significant is not a legitimate use of the technique.

Choosing the right kind of test

Which specific hypothesis test applies depends on the data type and what is being compared. The next three lessons cover each row below in depth; this table previews the overall map so each specific test in this course has a place to fit.