Understanding P-Values and Their Role in Hypothesis Testing

Research Fundamentals #3

Learning Objectives:

Define p-values and their role in hypothesis testing.
Understand how p-values relate to alpha (α) levels.
Interpret p-values correctly in research studies.
Recognize the limitations of p=values and the importance of effect size.

What is a P-Value?

p-value represents the probability of obtaining results at least as extreme as the observed data, assuming the null hypothesis (H₀) is true. It helps determine whether the observed effect is statistically significant.
Low p-value (≤ α): Strong evidence against H₀, leading to its rejection.
High p-value (> α): Weak evidence against H₀, leading to a failure to reject it.
Common thresholds:
p ≤ 0.05 (Standard in many research fields)
p ≤ 0.01 (Goldb standard used in medical research)

How P-Values Relate to Alpha (α)

The significance level Alpha (α) sets the threshold for decision-making in hypothesis testing.
p ≤ α → Reject H₀ (Statistically significant result)
p > α → Fail to reject H₀ (Not statistically significant)
However, p-values do not measure the strength of an effect; only whether it is likely due to random chance.

This is my pet peeve with p-value reporting. “We found a statistically significant effect.” Can be very misleading if you do not understand the values being reported. Example: a correlation of Pearson’s r of .10 Yes! You found that there was a relationship between two things of roughly 5%, and we care because why? If you test enough relationships, one is likely to be significant! Now research has focused so much on the significance in publishing it has taken away the significance of the effect size or relationships between variables. I will search for recent publication examples. I have found research published as significant without discussion of effect size!
– Dr. Santangelo
Tweet

Common Misinterpretations of P-Values

(True, but)

“A p-value of 0.05 means there is a 5% chance that the null hypothesis is true.”
True: The p-value indicates how likely the observed data would occur if H₀ were true. but: does NOT provide the probability that H₀ itself is true or false.
“A small p-value proves that the alternative hypothesis (H₁) is true.”
“A p-value of 0.05 means there is a 5% chance that the null hypothesis is true.”
True: A small p-value suggests evidence against H₀, but: does Not confirm H₁. Other factors, like effect size, must also be considered.
“A non-significant result means there is no effect.”
True: A high p-value indicates insufficient evidence to reject H₀. but: Does NOT prove that H₀ is true.

A rather weak explanation for not achieving statistical significance you hopefully will not encounter as a researcher is an underpowered study, meaning most likely you did not elect enough participants.
– Dr. Santangelo
Tweet

P-Values and Beyond: The Role of Effect Size

A statistically significant result (small p-value) does not always mean a practically meaningful effect. Remember, the Effect size measures the magnitude of the observed effect, not p-values, “statistical significance”.
-Dr. Santangelo
Tweet

Examples of effect size measures: (more on these later).
Cohen’s d (mean differences), Pearson’s r (correlations), Odds ratio (for categorical outcomes)
Reporting both p-values and effect sizes provides a more complete understanding of study results.

Key Takeaways

P-values help determine statistical significance but do not measure effect size or prove causation.
A p-value ≤ α suggests rejecting H₀, but interpretation must be careful.
Avoid common misinterpretations. remember. p-values do not confirm hypotheses or measure effect size.
Consider effect size alongside p-values for a better understanding of results.

Next Lesson: Type I and Type II Errors

In the next lesson, we will explore Type I and Type II Errors, understanding the risks of false positives and false negatives in hypothesis testing, and how researchers control these errors.