The new year is here! For me, this signals the approaching start of a year 2 research methods module that I run at Keele University. The module consists of weekly lectures and weekly lab classes, wherein students engage with classic-experiment replication and statistical analysis.
Towards the end of the semester, students break into groups of three and initiate a small research (experimental) project addressing a cognitive research question. When students get to the planning stage of their experiment, instructors always hear the same question: “How many participants do we need?”. They look at us eagerly awaiting some peal of wisdom (and a direct answer to their question), but are disappointed to hear the response: “It depends”.
Sample size of a study depends on a number of factors, all of which I am going to ignore in this blog post. Instead, I am going to step back and ask the more general question of why we care about sample size at all. What are we trying to achieve? Most students know—or intuit—that the larger the sample the better; but WHY?
Before we get to the meat of the matter, I thought I would share a nice joke I recently heard:
Three professors (a physicist, a chemist, and a statistician) were meeting to discuss their research, when all of a sudden a fire broke out in a nearby rubbish bin. The physicist proclaimed, “I know what to do: we must cool down the materials until their temperature is lower than the ignition temperature and then the fire will go out”. Unimpressed, the chemist says “No! I know what to do! We must cut off the supply of oxygen so that the fire will go out due to lack of one of the reactants”. While the physicist and the chemist debated what course to take, they were both shocked to notice the statistician running around the room starting other fires. They both scream “What the hell are you doing?”, to which the statistician replies, “Trying to get an adequate sample size.”
In psychology, we are (mostly) interested in universal effects; that is, in how all humans “work”. However, when conducting research it is obviously impossible to get every human alive to take part in your study. That’s why we take a sample from the general population, and use these results to make inferences about the population as a whole. The “true” behaviour of the population as a whole is known as a population parameter, and is of course never known; with statistics, we are interested in estimating this population parameter.
It turns out that the estimate of the population parameter improves as the number of observations sampled from the population increases. This can be demonstrated nicely with a simulation (using ESCI software made available from Geoff Cumming’s SUPERB book on effect sizes, confidence intervals, and meta anlayses). Consider the figure below.
The bell-shaped curve at the top of this image—which all students of research methods should automatically recognise as a normal distribution—reflects the distribution of scores of all people in the population (simulated) on some arbitrary psychological test. The peak of the distribution is around the value of 50, which is the Population Mean; this is the population parameter we are attempting to estimate, and is reflected by the solid vertical line running through the figure.
As stated, we can’t test everyone in the population, so we take a sample (i.e. run an experiment). In this simulation, the sample takes a fixed number random draw from the population distribution; each random sample in this example is represented by an open-blue circle below the population distribution. In this figure, the experimenter didn’t care too much about sample size, so only used 5 subjects in his experiment (i.e. there are 5 open-blue circles under the population distribution). The mean of this sample is represented by the green circle. As you can see, on this occasion, the sample mean is actually quite a good estimate of the population mean; but can we expect this all of the time?
The video below shows the simulation running multiple “experiments”; the means of each sample will be shown as a green circle; as the simulation progresses, keep a close eye on how close on average the sample estimates are to the true population parameter. Spoiler alert: they are very variable, and very often mis-estimate the population mean severely!
Not good, huh?! Why is this the case? With a small random sample, the samples can be taken from extreme ends of the population distribution, which will skew your average.
Now, here is another simulation video where we improve matters considerably: we increase the sample size. In this simulation, it has been increased from 5 to 100. What this does is it takes a more representative sample from the population distribution; true, you still might get a few “participants” sampled from the extremes of the distribution, but such an occurrence is rare, and counteracted by more samples being taken from the “bulk” of the distribution (just because there are more members of the population around this bulk).
As you can see, the sample means cluster much tighter to the true population parameter with large sample sizes.
Is a Large Sample Enough?
So, you know to take a relatively large sample when you next conduct some research. But does this always guarantee you are getting a good estimate of the population parameter? No. Your sample must be representative of the population as a whole; failure to do this means you will be taking samples from a subset of the population distribution (for example from a subset at the extreme ends of the distribution); if interested in the population’s IQ, it is not good enough to sample just from university students, as you will be taking your “random” samples from the extreme end of the distribution; hence, your sample’s mean will considerably overestimate the true population parameter.
And remember, as psychologists, the population parameter is often what we are most interested in. Remember that your sample is just estimating this important—but un-knowable—parameter. Remembering this will hopefully make you cognisant as to the importance of adequate sample sizes.