Two errors are possible. A Type I error rejects a true null, a false alarm. A Type II error misses a real effect.
Step 1: Let's Learn
Read it, or press Listen and follow the words.
They trade off
Demanding stronger evidence reduces false alarms and increases missed effects. There is no threshold that avoids both.
Which is worse depends
For a smoke alarm, a missed fire is far worse than a false alarm. For a criminal conviction, the balance runs the other way.
Misleading graphs
A bar chart whose axis starts above zero exaggerates small differences. Always check where the axis begins.
Testing many things
Test twenty unrelated ideas at the 0.05 level and about one will look significant by chance. That is why a lone striking finding needs replication.
Significant is not important
A huge sample can make a trivial difference statistically significant. Always ask how large the effect actually is.
Misleading graphs
A truncated vertical axis exaggerates differences. A three-dimensional pie chart distorts proportions. Neither involves a false number, and both are designed to produce a false impression.
Selective reporting
Running many tests and reporting only the significant one guarantees a finding. Choosing the time period that shows the trend you want is the same failure. The calculations are correct and the conclusion is not.
Choosing the flattering average
Mean, median and mode can differ by a great deal in skewed data. Reporting whichever supports your case is technically honest and substantively misleading, which is why readers should ask which was used.
How to read a statistical claim
Ask who was sampled, how, what was measured, what the comparison group was, and what was left out. Most misleading statistics fail at least one of those questions, and asking them is the whole defence.
Step 2: Try It Yourself
Tap and try it out.
Brand A
Brand B
Brand B has the most. It has 2 more than Brand A.
Step 3: Watch an Example
One step at a time.
Watch Kofi Spot a Misleading Chart
A chart shows Brand B towering over Brand A, with values 50 and 48.
- Step 1
The actual difference is 2 out of about 50, which is roughly 4%.
Step 4: Your Turn
Practice makes it stick.
The False Alarm
Problem 1 of 2
Rejecting a true null hypothesis is which error? 1 Type I, 2 Type II.
The Miss
Problem 2 of 2
Failing to detect a real effect is which error? 1 or 2.
Spot the Problem
1 of 8
A smoke alarm sounds with no fire. Which error? 1 Type I, 2 Type II.
2 of 8
A smoke alarm stays silent during a fire. Which error? 1 or 2.
3 of 8
Testing 20 ideas at the 0.05 level, how many look significant by chance on average?
4 of 8
Does a bar chart axis starting above zero exaggerate differences? 1 yes, 0 no.
5 of 8
Does reducing Type I errors increase Type II errors? 1 yes, 0 no.
6 of 8
Can a trivial difference be statistically significant with a huge sample? 1 yes, 0 no.
7 of 8
Which are genuine statistical warning signs?
8 of 8
Testing 100 ideas at the 0.05 level, how many false alarms on average?
Step 5: Quick Check
Show what you know.
Question 1 of 2
Convicting an innocent person corresponds to which error? 1 Type I, 2 Type II.
Question 2 of 2
Why does testing many hypotheses cause trouble?
What You Learned
- A Type I error is a false alarm; a Type II error is a missed effect.
- Reducing one increases the other, so context decides the balance.
- Truncated axes, multiple testing, and confusing significance with importance are the usual misuses.