Simulation and Randomisation

Building a sampling distribution by hand.

12 problems · about 21 minutes · VAR-1.E, UNC-3.C

What this lesson teaches

The student designs simulations and uses randomisation distributions to assess significance.

A simulation imitates a random process many times.
A randomisation distribution shows what chance alone produces under the null.
The empirical p-value is the proportion of simulated results at least as extreme.

Straightforward practice. Get the method working first.

5 problems

20 of 1000 simulated results were at least as extreme. Estimated p-value?
Answer 0.02
Why 0.02.
What is assumed while building a randomisation distribution?
Answer The null hypothesis, that there is no effect.
Why The null.
50 of 1000 trials were as extreme. Estimated p-value?
Answer 0.05
Why 50 ÷ 1000.
8 of 200 trials were as extreme. Estimated p-value?
Answer 0.04
Why 8 ÷ 200.
300 of 1000 trials were as extreme. Estimated p-value?
Answer 0.3
Why 300 ÷ 1000.

The same ideas with more to keep track of.

3 problems

That p-value at alpha 0.05. Significant? 1 yes, 0 no.
Answer 0
Why 0.3 is far above 0.05.
Which gives a steadier estimate? 1 for 20 trials, 2 for 2000.
Answer 2
Why More trials settle down.
What is assumed true while simulating? 1 the null, 2 the alternative.
Answer 1
Why The same assumption a p-value makes.

Mixed problems. Work out what kind of question it is before you start.

4 problems

Put the simulation design in order.
Answer 1. State what one trial represents. 2. Decide how the randomness maps onto the situation. 3. Define what counts as a success. 4. Run many trials and compute the proportion as extreme.
Why You cannot count successes before defining one.
10 of 500 trials were as extreme. Estimated p-value?
Answer 0.02
Why 10 ÷ 500.
The Estimate: 30 of 1000 simulated results were at least as extreme. What is the estimated p-value?
Answer 0.03
Why 0.03.
The Decision: That p-value at alpha 0.05. Is the result significant? 1 yes, 0 no.
Answer 1
Why Yes.