Sampling in Research: Preventing Bias and. Errors. Sandra L Siedlecki PhD RN
CNS. Senior Nurse Scientist. Cleveland Clinic. O Describe the issues related to ...
Sampling in Research: Preventing Bias and Errors
O Describe the issues related to sampling O Discuss how sample size and effect size
impacts power O Explain consequences of sampling problems
Sandra L Siedlecki PhD RN CNS Senior Nurse Scientist Cleveland Clinic
Sampling Plan O Define the population
O Why? O Be able to defend your
choice
O Identify the sampling
frame O Are they like your population? O Can you defend this? O Specify a sampling method O Determine the sample size
Sample O In all forms of research, it would be ideal to test
the entire population
O While random assignment might be the best
option, it is not really feasible O Money O Time O Resources
O In healthcare, convenience samples are the
norm
Convenience Sample A convenience sample is either a collection of subjects that are accessible or a self selection of individuals willing to participate in your study O As long as they look like the population you are interested in, it is probably a good choice O But this is always a limitation of your study O
O
You cannot generalize your findings beyond your sample
Misconception O A study without a random sample is not any
good O Most RCTs do not use a random sample O In fact most healthcare research does not
use a random sample
O Rather we control for bias in an RCT by
using random assignment
Misconception O I want to study the effect of exercise on BMI in
patients with DM
O If I use a number generator to select my subjects
from a list of patients with diabetes on my unit, I am doing random sampling. O False O Random sampling means everyone has an equal chance of being in the study, you only sampled from your unit O You still can only generalize to the patients on that unit. Did random assignment help you? O You could have just included all patients on the unit until you reached your sample size
Example O How about telephone sampling using a random
number generator
O It will miss people who do not have a phone. O It may also miss people who only have a cell phone that
has an area code not in the region being surveyed.
O And it will likely miss people who do not wish to be
surveyed, those who monitor calls on an answering machine and don't answer those from telephone surveyors.
O Thus the method systematically excludes certain
types of consumers in the area.
O Is this really a random sample?
Sampling O A sampling method
is called biased if it systematically favors some outcomes over others.
Example O I conduct a survey of high school students at
four High Schools in the city to measure teenage use of illegal drugs
O This could be a biased sample because it
does not include home-schooled students or dropouts.
O Where are these schools? O Do they represent different socio-economic
conditions?
O Is any one group under-represented?
Example O A sample is also biased if certain members are
overrepresented relative to others in the population. O For example, a "man on the street" interview which selects people who walk by a certain location is going to have an overrepresentation of healthy individuals who are more likely to be out of the home than individuals with a chronic illness.
Caveman Effect O Much of our understanding of ancient man
comes from cave paintings made 40,000 years ago.
O If there had been paintings on trees, animal
skins, or hillsides, they would have been washed away long ago. O Evidence most likely to remain intact until today was evidence protected from the elements (like in a cave) O Prehistoric people are associated with caves because that is where the data still exists, not necessarily because most of them lived in caves
Biased Sample O A biased sample causes problems O Almost every sample in practice is biased
O If the degree of underrepresentation is small the
sample can be treated as a reasonable approximation to a random sample.- it is probably not a big deal O If the group that is underrepresented does not differ a lot from the other groups then it is also probably not a big deal
Lesson Learned? O Pick your sample
carefully, it is important
O But how will you know?
Classic example O Headline DEWEY DEFEATS
TRUMAN, was a mistake
O The reason the Tribune
was wrong is that they trusted the results of a phone survey O
Telephones were not yet widespread, and those who had them tended to be prosperous and have stable addresses
O Over represented the
wealthiest voters
Sample Size O Too Big O You do not find anything, even though there
might be something to find (type II error)
O You conclude, “there is no difference”, when
there is
O Too Small O You find everything is significant (type I error) O You conclude there is a difference, when in
fact there is not a difference
Sample Size O Do a power analysis O This will tell you how many subjects (or how
many subjects per group) you need to detect a difference, if one exists
O Typically in this analysis, you set the power
to detect a difference at .80, and you set your probability (alpha) at .05 O G Power is a FREE program you can
download to your computer
Misconception O When it comes to sample size, bigger is
always better O False
O Too big can lead to a Type I error
O Using a sample larger than you need is
costly (dollars) and it there is a ceiling effect for advantages. O Once you reach the optimal sample size
adding more subjects makes only a miniscule difference in power to detect a difference
Misconception O Qualitative studies are underpowered
because they usually have only a small sample size O False
O The rules for sample size calculations are
for quantitative study designs only
Example O You would not judge
the quality of an Olympic diver based on the criteria for an Olympic figure skater.
O Qualitative studies use a different set of rules O Typically referred to as “data saturation”
When to do power analysis? O Do an power analysis in the early planning
stages to determine how many subjects you will need. O This is really just a guess
O Do a power analysis after you have analyzed
your data to see how much power you actually had O If you did not find a difference, this may
provide the explanation
Why does it matter? O The bigger the effect, the smaller your
sample will need to be to detect a difference O The smaller the effect, the bigger your sample will need to be
What is effect size? O When you run a power analysis, the program
asks you about the expected effect size. O How big of an effect do you think the
intervention will have on the variable you are measuring? O Small
.50
Example O I am comparing drug A and drug B on how
well they lower systolic BP on patients with systolic hypertension O If I expect a large difference in Systolic BP
(30 mm hg) between the two drug groups O Perhaps 20-30 per group will be enough
O I expect a small difference in systolic BP (2-5
mm hg) between the two drug groups
O Perhaps 200-300 per group will be needed
A word of Caution O Random sampling and Random assignment are
different O Random assignment is used for RCTs (decreases bias and sampling error)
O All subjects in the study have an equal chance of
being in either the control or experimental group O Role the die (odds-even) O Pick one of two envelopes O If the groups are exactly the same size, they probably did not use random assignment, even though they said they did
Summary O The sample is critical O Plan carefully
O Look first at your population of interest O Who so you want to be able to generalize to?
O Look at your available subjects
Take Home Message O Do your homework O Know your expected
“Effect Size” O Comes from clinical
experience
O Comes from
previous studies
Putting it together O Do hourly rounds decrease falls? O Fix the question O Identify the population O Clarify the outcome O Identify your options
O What are my options?
O Calculate your sample size
Putting it together What effect does listening to music (selfselected) an hour a day (for 7 days), have on feelings of depression in patients with chronic low back pain?
Putting it together Is there a difference in depression levels between people with chronic low back pain who listen to music for one-hour a day (for 7 days) and those who do not?
O Identify the population
O Clarify the outcome
O Clarify the outcome
O Identify the population
O Identify your options
O Identify your options
O How many groups do you need?
O How many groups do you need?
Questions
