Statistics · Topic 9 of 13
Confidence Intervals & P Values
Theory
When we take a sample, the mean or proportion we calculate is only an estimate of the true population. Statistical software provides specific values to help us understand how reliable this estimate is, and whether we can make firm conclusions.
1. Understanding the p-value
- A p-value is the probability of getting data as extreme as the data you actually observed, assuming that the null hypothesis is true.
- A small p-value (typically less than 0.05) indicates that the result is statistically significant, providing evidence to reject the null hypothesis.
- If the p-value is 0.05 or greater, there is not enough evidence to support a difference or relationship, meaning you must fail to reject the null hypothesis.
2. Confidence Intervals (CIs)
- Because it is usually impossible to test an entire population, we cannot know the exact population mean or proportion. Instead, software generates a confidence interval—a range of plausible values for the true population parameter. If the study were repeated 100 times, about 95 of the intervals produced this way would contain the true value. That is what the "95%" refers to: how reliable the method is, not the probability that this particular interval is right.
- Wider confidence intervals indicate more uncertainty in the data, whereas narrower confidence intervals indicate less uncertainty.
You may see some books say there is a 95% probability that the true value lies in the interval. That is a common shorthand, but strictly the true value is fixed—it is the interval that changes from sample to sample. Either phrasing has been accepted in the exam.
3. The "100 Times" Rule (Crucial Exam Definition)
If an exam question asks you to interpret what a 95% confidence interval literally means, you must memorise and use this exact phrasing:
"If you were to repeat the study 100 times, 95 of these times the true population mean (or proportion) would lie within the interval"
4. Confidence Intervals Containing Zero
- When testing the difference between two groups (like in a t-test or z-test), the null hypothesis usually assumes the difference between them is exactly zero.
- If your calculated 95% confidence interval ranges from a negative number to a positive number, it contains zero.
- An interval containing zero means zero is a plausible value for the difference, so there is not enough evidence of a difference between the groups. This is consistent with a decision to fail to reject the null hypothesis.
Worked examples
Example 1
Example 1: The Literal Interpretation
A marine biologist takes a sample of adult dolphins and calculates their mean weight. The statistical software generates a 95% confidence interval for the mean weight of [155.2 kg, 168.4 kg]. Explain what is meant by a 95% confidence interval in this context.
If the biologist were to repeat the study 100 times, they would expect the true population mean weight of the dolphins to lie within the calculated interval 95 times.
Example 2
Example 2: Intervals Containing Zero
A nutritionist runs an independent t-test to see if there is a difference in the mean sugar content between Brand A and Brand B cereal. The software generates a p-value of 0.612 and a 95% confidence interval for the difference in means of [-1.45g, 2.15g].
- (a) State whether the nutritionist should reject or fail to reject the null hypothesis.
- (b) Explain how the confidence interval supports this decision.
(a) Since the p-value (0.612) is greater than 0.05, the nutritionist must fail to reject the null hypothesis.
(b) We can be 95% confident that the true mean difference in sugar content lies between -1.45g and 2.15g. Because this interval contains zero, zero is a plausible value for the difference, so there is not enough evidence of a difference in the mean sugar content between the two brands.
Example 3
Example 3: Combining p-values and Confidence Intervals
A driving instructor believes that pupils who take lessons in the morning have a higher pass rate proportion than those who take lessons in the evening. They run a 2-sample test for equality of proportions. The software outputs a p-value of 0.014 and a 95% confidence interval for the difference of [0.035, 0.125].
- (a) Interpret the p-value and the result of the test in the context of the instructor's belief.
- (b) Explain how the confidence interval supports this conclusion.
(a) Since the p-value (0.014) is less than 0.05, we reject the null hypothesis. Therefore, there is statistically significant evidence to suggest that there is a difference in the proportion of pupils passing in the morning compared to the evening, supporting the instructor's belief.
(b) We can be 95% confident that the true difference in the proportions lies between 0.035 and 0.125. Because this interval does not contain zero (both numbers are positive), it provides further evidence that a true statistical difference exists between the two proportions.