Random sampling and random assignment are different things, and they buy different rights. Choosing subjects at random from the population lets you generalize to it. Assigning treatments at random lets you claim the treatment caused the difference. An observational study can only report association.
Step 1: Picture It
Start with something you already know.
A pot of soup
- A cook stirs a big pot of soup until it is mixed all through.
- Now one spoonful, from anywhere, tastes like the whole pot.
- A spoonful skimmed off the top of an unstirred pot might taste only of oil.
- To test a pinch of salt, the cook has twenty bowls poured from an unstirred pot, some oily and some plain.
- A coin toss for each bowl decides which bowls get the salt.
Random sampling is the stirring: it lets a small taste stand for the whole pot, so a result may be generalized.
Random assignment is the coin: it spreads oily and plain bowls across both sides, so a difference in taste can be pinned on the salt.
Where the picture falls short: people are not soup, and a coin makes two groups alike only roughly, so a small gap could still be chance.
Step 2: Let's Learn
Read it, or press Listen and follow the words.
Try it first
At one school, 30 students chose to sign up for free tutoring and 30 did not. By the end of the semester, the tutored students’ average score had risen 8 points more than the others’. Kai says the tutoring caused the extra 8 points. Which is another reason the tutored group could have done better?
Stuck is fine. Let’s Learn shows the way next. Got it first time? You can jump ahead to Watch an Example.
Random sampling
A simple random sample gives every member of the population an equal chance. Stratified sampling randomizes within groups. Both let the sample stand for the population. A convenience or voluntary sample does not, however large.
Random assignment
Take the subjects you have and let a coin decide who gets the treatment. Known and unknown differences between people are then spread evenly across the groups, so a difference in outcome can be pinned on the treatment.
Four combinations
Random sample and random assignment: generalize and claim cause. Random sample only: generalize, association only. Random assignment only: cause, for these subjects only. Neither: describe these people, nothing more.
Why observation is not enough
A lurking variable may explain the association entirely. Stress could cause both the coffee and the poor sleep. No amount of data removes that possibility; only assignment does.
Bias
Undercoverage leaves part of the population out. Nonresponse loses those who decline. Voluntary response attracts strong opinions. Wording can lead. A bigger biased sample is a more precise wrong answer.
One question
Was anybody assigned a treatment? If yes, and at random, cause may be claimed. If no, association is the most that may be said. That settles what a study can claim.
Step 3: Try It Yourself
Tap and try it out.
Three studies: (a) 200 volunteers randomly assigned to a new diet or their usual one; (b) a random sample of 1000 adults asked about exercise and mood; (c) 50 gym members compared with 50 non-members on blood pressure. What may each claim?
- Answer(a) random assignment, volunteers: cause, for people like these volunteersno random sample, so no generalizing to everyone
Step 0 of 3
Treatment
Control
There are 11 in all.
Treatment has the most. It has 3 more than Control.
Step 4: In Real Life
A coffee-and-health headline
A study finds coffee drinkers live longer. It is observational, so it cannot say coffee is the cause: maybe coffee drinkers are wealthier. Only an experiment could claim more.
Step 5: Watch an Example
One step at a time.
Watch Leo Read a Reading-App Study
A school district has 3,000 students on its roster. It draws 100 names at random, then flips a coin for each one: heads, the student uses a reading app for a month; tails, the student reads as usual. On average, the app group gains 6 points more on a reading test.
- Step 1
Two questions settle what any study may claim: how were the subjects chosen, and who decided which group each one joined?
Watch Noor Read a Call-In Poll
A town radio station asks listeners to phone in: should the town build a skate park? 300 people call, and 240 of them say no. The station reports that 80% of the town is against the park.
- Step 1
Nobody was assigned anything here; people were only asked their opinion. So this is an observational survey, and the one question is whether the 300 callers stand for the town.
Step 6: Your Turn
Practice makes it stick.
The Coupon Test
Problem 1 of 17
A cafe tests a coupon worth $2 off a drink. For one week, a coin flip at the door decides whether each of 200 customers gets the coupon. Customers with the coupon spent $14 each on average; customers without it spent $11. How many dollars more, on average, did the coupon customers spend?
dollars
Dogs and Outdoor Gear
Problem 2 of 17
A random sample of 400 households from a city’s address list finds that households with a dog spend $35 a month more on outdoor gear, on average, than households without one. Which claim does the study support?
Read the Design
Ask how the subjects were chosen, and who decided their group.
Problem 3 of 17
A researcher has 60 volunteers. A coin flip decides which of them use a new sleep app for two weeks. Is this an experiment or an observational study?
Problem 4 of 17
A survey asks 500 adults how many hours of TV they watch each day and how they would rate their mood. Is this an experiment or an observational study?
Problem 5 of 17
A principal writes every student’s name on a slip, mixes the slips in a box and draws 50. Which kind of sample is this?
Problem 6 of 17
A reporter asks the first 30 shoppers who walk out of a mall what they think of a new parking fee. Which kind of sample is this?
Problem 7 of 17
A study draws 300 workers at random from a city’s list of all 9,000 of its employees, then a coin decides which of the 300 get a standing desk. The standing-desk group reports less back pain. What may the study claim?
Whose Step?
Each page has one line that slipped. Find it, then put it right.
Problem 8 of 17
A random sample of 400 teens from a city’s schools finds that teens who play an instrument have higher math grades. Leo wrote down what the study shows. One line is a slip. Which?
Leo’s working: what the instrument study shows
- 1400 teens chosen at random from the city’s schools
- 2nobody was assigned an instrument: observational
- 3so playing an instrument raises math grades
- Answerthat cause holds for every teen in the city’s schools
Problem 9 of 17
What would the instrument study need before line 3 could be written?
Problem 10 of 17
A news site asked readers to vote online: should the town build a pool? Maya wrote down what the poll shows. One line is a slip. Which?
Maya’s working: what the online pool vote shows
- 1the site posted the poll; 2,000 readers chose to vote
- 21,400 of 2,000 voted yes: 70%
- 3a sample this large stands for the whole town
- Answerabout 70% of the town wants the pool
Problem 11 of 17
Whose views does the 70% in the pool vote describe?
Problem 12 of 17
Eighty students answered a flyer asking for volunteers. A coin put each one on flash cards or on their usual way of studying, and the flash-card group scored 9 points higher on average. Sam wrote down what the study shows. One line is a slip. Which?
Sam’s working: what the flash-card study shows
- 180 volunteers; a coin assigned flash cards or usual studying
- 2random assignment: the 9-point gap may be pinned on the flash cards
- 3random assignment also makes the volunteers stand for every student in the school
- Answerflash cards raise scores for every student in the school
Problem 13 of 17
For whom may the flash-card result be claimed?
Pick the Move
Which design or method does the job? No full working is needed here.
Problem 14 of 17
A district with 40 schools wants to know whether a later start time causes better attendance. Which plan could answer that?
Problem 15 of 17
A college has 4,000 undergraduates and 1,000 graduate students. It wants a sample of 100 that is sure to include both kinds, with names chosen at random inside each kind. Which method fits?
Same Claim?
Judge each study by its two randoms, without working anything out.
Problem 16 of 17
Select every study that may claim the treatment caused the difference.
Problem 17 of 17
Select every study whose result may be said to hold for the whole group it names.
Step 7: Quick Check
Show what you know.
Question 1 of 1
Which is required to claim causation?
Ready for more?Optional. Past the lesson, for anyone who wants it.
Stretch 1 of 2
A survey of 200 adults finds 50 of 100 coffee drinkers sleep poorly, against 30 of 100 non-drinkers. Split by stress: of the coffee drinkers, 75 are stressed (45 of them sleep poorly) and 25 are not (5 sleep poorly). Of the non-drinkers, 25 are stressed (15 sleep poorly) and 75 are not (15 sleep poorly). Among the stressed adults only, how many percentage points higher is the coffee drinkers’ poor-sleep rate than the non-drinkers’?
percentage points
Stretch 2 of 2
A town has 20,000 adults, and 12,000 of them have only a cell phone, with no landline. A pollster draws numbers at random from the town’s landline directory, so only adults in it can be picked. For at most 10% of the town’s adults to be left out, how many of the 20,000 would need to be in the directory?
adults
What You Learned
- Random sampling earns the right to generalize; random assignment earns the right to claim cause.
- An observational study reports association, whatever its size.
- A bigger biased sample is a more precise wrong answer.
- Ask of any study: was anybody assigned a treatment?
For the grown-up
The misconception this lesson targets is that "random" is one thing that does every job. Students hear that a study was randomized and grant it every right: a coin-flip experiment on volunteers gets generalized to everyone, and a large random survey gets read as proof of cause. The cure is to ask two separate questions of every study, how were the subjects chosen and who decided their group, and to grant one right per random. Its cousin is "a big sample is a good sample": a call-in poll of thousands still hears mostly the people who care most. When a child reports a finding, ask which random earned it.