Quick & Dirty Hypothesis Testing

Hypothesis testing answers the more focused question "Based on sample data, is the hypothetical number Uo a reasonable guess for the mean of the population from which the sample came?"

If the sample mean is so high that if forces the 95% one-sided lower bound (found by subtracting the 90% two-sided margin of error from the sample mean) to be above U0, we can be 95% confidence that the mean of the sample's parent population is also above Uo.  Statisticians call this "rejecting the hypothesis U<Uo at the 5% significance level." Significance is just 1 minus confidence; the lower the significance, the higher the confidence.

If thesample mean is so low that it forces the 95% one-sided upper bound (found by adding the 90% two-sided margin of error from the sample mean) to be below Uo, we can be 95% confidence that the mean of the sample's parent population is also below Uo.  Statisticians call this "rejecting the hypothesis U>Uo at the 5% significance level." Significance is just 1 minus confidence; the lower the significance, the higher the confidence.

If Uo is outside the 90% "between" confidence interval (either above the 90%
two-sided upper bound or below the 90% two-sided lower bound), we can be 90% confidence that the mean of the sample's parent population is different from Uo.  Statisticians call this "rejecting the hypothesis U=Uo at the 10% significance level."




A rough test of whether two populations are the same

Often we need to know whether two populations have the same population mean or whether they each have different population means. 

If we take a sample from each population, the sample means will almost certainly be at least a little different whether the population means are the same or not.  To do a rough test of whether or not we can be 95% confident that the population means are different, set up three columns of data in Excel: one for each sample, and a longer column for the two samples combined.  Then use the Descriptive Statistics tool with a confidence level of .95.  If the absolute value of the difference between the two sample means is much greater than the margin of error (miscalled "confidence level" by Excel) for the combined sample, there is good evidence that the two population means are different.  If the absolute value of the difference between the two sample means is much smaller than the margin of error for the combined sample, there is no significant evidence that the two population means are different. (Note this is not the same as evidence that they are the same!)  If the absolute value of the difference between the two sample means is fairly close to the margin of error for the combined sample, collect more data or use an exact method from a statistics textbook.

(The two populations can be naturally occurring populations, but very often one population is the future values of some large collection of people or things if none of them receive a new treatment and the other population is the future values of the same collection if they all receive the new treatment.  This is the hypothesis we are testing when we use a control group to represent the first "population" and an experimental group to represent the second.)