Statistics · Topic 8 of 13
Hypothesis Testing & Errors
Theory
Hypothesis testing is a statistical method used to draw conclusions about a whole population based on a smaller sample of data. It removes subjective guesswork and allows us to make decisions free from bias.
1. Formulating Hypotheses
Every statistical test starts with two opposing statements:
- Null Hypothesis (H0): A statement that nothing unusual is happening. It usually states that there is no difference, no relationship, or no effect.
- Alternative Hypothesis (H1 or Ha): A statement that something is happening. It states that there is a difference, relationship, or effect.
2. The p-value and Making a Decision
When you run a test in statistical software, it generates a p-value. This is the probability of getting data as extreme as the data you actually observed, if the null hypothesis is true. We test this against a standard 5% significance level (a 0.05 threshold):
- If p < 0.05: The result is statistically significant. You must reject the null hypothesis.
- If p ≥ 0.05: The result is not statistically significant. You must fail to reject the null hypothesis.
Massive Exam Trap:
Never write "we accept the null hypothesis". The test only ever looks for evidence against H0, and finding none is not the same as showing H0 is true. Always use the phrase "fail to reject".
3. Statistical Errors
Because hypothesis testing is based on probabilities, we will sometimes reach the wrong conclusion. At the 5% significance level, if the null hypothesis is actually true, we will wrongly reject it about 1 time in 20. There are two ways a test can be wrong:
- Type I Error (False Positive): Rejecting the null hypothesis when it is actually true. (e.g., Concluding a drug works when it actually does nothing).
- Type II Error (False Negative): Failing to reject the null hypothesis when it is actually false. (e.g., Concluding a drug does nothing, when in reality it does work).
4. Confounding Variables
Sometimes a test shows a significant result, but the two variables are not actually causing the change in each other. A confounding variable is an outside, hidden influence that is changing both variables simultaneously, potentially leading to incorrect conclusions.
Worked examples
Example 1
Example 1: Writing Hypotheses in Context
A sports scientist wants to investigate whether a new brand of running shoe changes the mean sprint times of professional athletes. State the appropriate null (H0) and alternative (H1) hypotheses for this study.
H0: There is no difference in the mean sprint times of athletes wearing the new shoes compared to their normal shoes.
H1: There is a difference in the mean sprint times of athletes wearing the new shoes compared to their normal shoes.
(Note: Always include the context and specify the measure, such as 'mean', to get the mark).
Example 2
Example 2: Interpreting a p-value < 0.05
A researcher runs a statistical test to determine if there is a relationship between the number of hours students spend studying and their final exam score. The software generates a p-value of 0.014. Interpret the p-value and the result of the hypothesis test in context.
Since the p-value (0.014) is less than 0.05, we reject the null hypothesis.
Therefore, there is statistically significant evidence to suggest that there is a relationship between the number of hours students spend studying and their final exam scores.
Example 3
Example 3: The "Fail to Reject" Trap
A botanist is testing whether a new type of soil increases the mean height of tomato plants. An independent t-test generates a p-value of 0.182. A student concludes: "Since 0.182 is greater than 0.05, we accept the null hypothesis. The soil makes no difference."
Explain why the student's conclusion is statistically incorrect, and state what they should have written.
It is incorrect because you can never definitively "accept" a null hypothesis.
The student should have written: "Since 0.182 > 0.05, we fail to reject the null hypothesis. There is not enough evidence to suggest the soil increases the mean height."
Example 4
Example 4: Type I and Type II Errors
A tech company is testing a new battery for their latest smartphone. Their null hypothesis is that the new battery lasts exactly the same amount of time as the old battery.
Explain, in the context of this scenario, the meaning of:
- (a) A Type I error.
- (b) A Type II error.
(a) A Type I error (false positive) would be concluding that the new battery lasts longer (or shorter) than the old one, when in reality there is no difference in battery life.
(b) A Type II error (false negative) would be concluding that the new battery makes no difference to battery life, when in reality it actually does last longer (or shorter).