Hi All
I posted a query to the list regarding sample size, and received several
replies, for which I am really grateful. As I understand it, the idea
of the list is for people who receive replies to then post a compendium
of the replies back to the list. I have produced that here, in the
order in which I received the replies (these are initial replies only -
in 4 cases, there were follow-up discussions). I have removed the names
of posters from their postings, but have listed (in alphabetical order)
the posters' names at the end.
Thanks very much for all who participated - you helped a great deal, and
I hope that others on the list will find the comments very useful.
Ken
----
I think that you (or anybody [no disrespect meant]) deserves hammering
for
regarding the 259 units as a random sample. The 2,600 would have been,
but you did not get what you planned to get from about 90% of them. This
is probably not your fault, but not knowing that the response rate (10%)
would be low ... is arguable.
The appropriate sample size is the one that uses the resources you have
(money, time, equipment, etc.) wisely. That is, the information you get
for the resources would be worth more than the investment. If the
sampling and recording (measurement) were cheap (requiring minimum of
resources), then the whole population should be in the sample.
The example about the Gallup poll is not a good one; the sample size
reflects the need to have estimates with a particular (prescribed)
precision AND the survey has to be done quickly, because the information
(ratings) are a perishable good -- it is valuable only when timely
(e.g.,
for the next day's papers). In your case, the perishability may not be
an
issue, but regard the necessity to have the results as 'time being of
essence' -- time is a valuable resource that is in short supply [in
Gallup's case].
I think that your database is not exactly the population that you would
like to survey, but that is the best [the closest thing] that you have.
I
conjecture that you have a list of contacts (customers/patients), and
you
would like to know about customers/patients (current and prospective) in
general. So your bosses are also imperfect ... (but don't tell them
because that would be yet another [costly] imperfection).
Should have done: Try to contact everybody in your database, because
only
about 2000 would respond. But this is (my) dishonest hindsight. Even if
you get the 2000 responses, which may appear to be a lot, they may be a
poor representation of the 18,000, because there may be systematic
differences between respondent and non-respondents. What to do then --
look for a competent statistician. This is a non-trivial business.
----
I think the only way you can salvage anything from this is it do a BIG
study of non-responders and show that the responding group is very
similar in all
characteristics relevent to your study.
------
The problem is the low response rate, not the proportion of the
population which you have. You have only 10% of your random sample
providing information. This could cause a huge bias. For example, you
might have a survey where only people who hold strong views bother to
respond and hence conclude quite wrongly that most people have strong
views. You need to consider whether there is a possibility of such bias
in your survey. You will find this discussed in books on survey
sampling. As for your Ph.D., you should be OK if you understand this
and discuss it in your thesis.
------
A couple of relevant points, one helpful to you, one rather
less so.
Your basic view is consistent with the mathematics behind
estimation from sample surveys. If N is the population
size and n is the sample size, the variance of an estimate
of the population mean is
( 1 - n/N ) SS / n ,
where SS is the square of the population standard deviation.
This also applies to estimates of proportions (which are in
effect special cases of means).
The effect of the sample size *relative* to the population
size comes in the ( 1 - n/N ) component, which varies only
very slightly with n, unless n is a very substantial fraction
of N (which almost never happens). The real effect of n itself
comes from its position in the denominator.
To make this point yourself, you can point out to those doing
the hammering that the Gallup example (1200 out of 250m) would
give essentially the same accuracy as it would if the US
population were 2.5m or 25000m, since (1 - 1200/2500000) and
(1 - 12000/25000000000) are both effectively 1.
The negative point is that your response rate (259 out of 2600)
does mean that you are pretty reliant on those who *did*
answer being representative of the population. In practice,
it's very hard to do this in a way which would satisfy an
academic statistician. I put it that way because lots of
real-world surveys do have awful response rates, so you do
seem to be in quite good company. And your query makes it
plain that you have done all the sorts of check which are
reasonable.
On a *very* minor point, you say in your "current response"
that a random sample (by definition) is a small group [from]
a population. Strictly, this isn't right; there's nothing
in the definition to require it to be small. Also, the
"strict" statistical term is "simple random sample", when the
population is finite - but there's no need to be too fussy
about that point.
----
I guess you've probably been swamped by responses about why, what you've
got is not particularly useful.
----
a priori there is no problem with the fact that your usable sample size
1,5%. however there are two issues. will you still get significant
results? there may be selection bias and this relates to the 10%
response rate given your random sample of 2,600. i understand that you
are happy with your confidence intervals. but are they based on
asymptotic theory (using the t test) or are they exact. in the latter
case there is no problem, in the former case 259 could not be large
enough in particular if you have covariates. in case you use the t test,
do you test for normality?
so i cannot advise you on defences until i learn more about whether
there is sth objectionable. give me more info.
-----
I don't normally reply, but was moved by your sad face :-(.
What % of the population you have surveyed is pretty much irrelevant (it
is only important when the population size is small). What are important
are (a) that the sample was truly random (which you say it was), (b) the
sample size you obtain (259 is okay) and (c) the response rate. If
someone was going to criticise your study, then they should have picked
on the response rate: 10%, which is low to be honest for a social
survey, although might be more usual for what you're doing.
Given that you have a database of 18,000 cases, you have, I assume, good
'population' estimates - i.e. you have good estimates for everyone that
was eligible for your survey. You could therefore use that to generate
post-survey adjustment weights so that the distributiona of the
characteristics of your survey match those the population. By making
your sample look more like the population for the measures for which you
know the population estimates, you (hope to) make it more likely that
the other estimates derived from your survey sample are accurate. Google
rim weighting for more information about this.
----
It is not possible to answer without knowing more about why you are
sampling in the first place. What is it you want to know from the data
that you couldn't find out form the entire data set? I suppose your
random sample of 2600 is okay but when response rate is included you
only get 10% of the random sample and that sample is not representative
of the population (being younger).
---
Just a quick bit of support and only minor help, but possibly not good
news (?).
259 can be an excellent sample size, especially when fitting simple
statistical models: constant means, regression with few explanatory
variables, proportions, logit models, simple time series, etc. 259 can
lead to very tight confidence intervals on parameter estimates and
strong conclusions. No problem there.
The problem is low the response rate I'm afraid. You chose 2600
randomly, which should nicely represent the population of 18K, but only
259 responded. This is effectively a self-selected sample (the 259). It
is likely to be biased in how it responds to the questions asked, even
if not in any demographic characteristics (like Age, location, etc).
Your attackers may feel this survey is like a TV survey question (e.g.
Is Jesus relevant in modern society?), to which only highly interested
people (i.e. Catholics and atheists) are likely to bother responding to:
thus giving a highly biased sample and set of responses (but since
atheists and Catholics are of all ages, locations, etc this will appear
as a demographically accurate sample!). Can you see the similarity with
your survey? Having a very low response rate usually points to bias and
self selection issues.
Can you do anything to increase the response rate somewhat
(reward/incentive of chocolate, raffle tickets, free pass to the zoo,
something?, etc)? Such considerations, and methods/tactics to help
ensure an acceptable response rate, should have been considered at the
absolute birth of the decision to use surveys (before construction of
questions, etc) and in consultation with an experienced statistician.
Sorry to point this out at this time, but it is an oft repeated theme,
something that users of this list see every day unfortunately. Perhaps
this actually was done and you PhD supervisor was not deficient on this
issue, in which case I apologise for my speculation. Regardless, I
sincerely hope you can resolve the low response rate: increasing it is
the only action I could recommend.
The only way to check validity of your current 259 sample, in regards to
your research questions, is to somehow get the non-responders to answer
the survey and compare results L. Which is a bit silly, since you would
just combine the samples and analyse the lot together in that case.
-----
By coincidence, I read about just this question yesterday morning and
attach the cover and sample page from "Teaching Statistical Concepts".
I'm sure your inquisitors will be pleased to hear they are naïve and
confused in their understanding of statistical inference.
Moore(1990) is "The skills challenges of the nineties". JRSS A 153(3)
265-85
Your comment about non-responders should be followed up, to look for
evidence that they are "missing at random" (MAR) or might indicate a
direction of bias.
----
>"A random sample, by definition, is a small group of a population."
No, it isn't. A random sample can be large or small, and can be a large
or small proportion of the population. In the (fairly rare)
circumstances where you do sampling with replacement it can even be
larger than the population.
>"A sample’s size (in this case 15%) relative to the population is not an indication...
Good.
However, the big problem is not the sample size as such, but the
possibility of BIAS. You selected a random sample, but the people who
responded are probably *not* a random sample. With a high rate of
non-response this has to be a real concern.
-----
Firstly this is a PR problem that many, many statisticians have
struggled with.
You are right in saying that it is the sample size, rather than the
sample fraction, that is the key determinant of the accuracy of a random
sample.
One non-technical explanation is the following. The validity of an
opinion poll (etc) doesn't depend on asking everyone. It depends on
whether the people who weren't asked would have given similar answers to
those who were asked. If the question is a simple yes/no or
Labour/Conservative/Democrat choice, then 2500 randomly chosen people
are more than enough to get a reasonably accurate estimate of the
population proportion of 'yes' answers, etc.
What is however of greater concern is that the response rate was quite
low. Again it's not the fraction of people who responded that is of
concern; rather it's the possibility that the people who did respond
were different from those who didn't respond. This is actually a far
greater problem than the sample size.
----
Ken, what you say here seems generally reasonable to me. There is
nothing wrong in seeking to
recruit a random sample, nor in assessing whether responders differed
systematically from
non-responders - which is a useful check on whether non-response was
random.
But, of course, a 10% response rate of those invited is very
regrettable. There would be nothing
wrong in sampling 1.5% of a very large population, all of whom respond -
but this is different, with
much greater risk of differential non-response. Since you sent out your
email last week, a colleague
of mine has mentioned to me a follow-up study to see how much that
dentists learned on a day course
on radiation protection was retained 6 months later. In that study, they
attempted to contact 284
dentists who had done the course, only 65 (23%) responded to an
invitation to do the follow-up, even
after a reminder, and even though a randomised carrot (£100 store
vouchers) was offered. So you're
not alone in getting a low response rate. My reaction to my colleague
was, the results are unlikely
to appeal to a highly rated journal, nevertheless let's see what they
tell us. Even the fact of the
low response rate is useful information - it should be possible to get
some idea of the main reasons
for it, and to take these into account in future studies - if only by
upgrading the size of the
random sample you invite to take part. Best practice is to plan the size
of the sample in such a way
that the confidence interval on the outcome of primary interest is
narrowed down to a specified
size. If you anticipate a low response rate, the number you invite to
participate should take this
into account, e.g. if you anticipated a 50% response this would prompt
doubling the size of sample.
Arguably, you can get a slightly purer check on differential response by
comparing the 259 vs. the
2600-259, rather than vs. the 18000-259. These two comparisons will be
practically identical in
sensitivity to detect a difference.
----
The posters (in alphabetical order)
Adrian Baddeley
Eryl Bassett
Martin Bland
Ben Carter
Blaise F Egan
Richard Gerlach
Nick Longford
Robert G. Newcombe
Zoann Nugent
Kevin Pickering
Allan Reese
Karl Schlag
Paul R. Swank
----
Regards
Ken
---------------------------
Ken Masters
IT Health Education
http://www.ithealthed.com
____/\/********\/\____
> -------- Original Message --------
> Subject: Sample size/percentage
> From: Ken Masters <[log in to unmask]>
> Date: Wed, November 19, 2008 6:10 pm
> To: [log in to unmask]
>
>
> Hi All
>
> (Thanks to Claire and Ali for their clarification).
>
> I have conducted a survey from a data base. The total size of the data
> base is roughly 18,000. My random sample size was 2,600. My usable
> response rate was 259.
>
> When I presented my results, I was hammered (and I mean HAMMERED) on the
> fact that my usable sample size is 1,5% of the population, and that the
> number is 259. I'm not a statistician, so my response is probably
> ham-fisted, and all comments are welcome.
>
> To the 1.5%, my response is currently: "A random sample, by definition,
> is a small group of a population. A sample’s size (in this case 15%)
> relative to the population is not an indication of the statistical
> validity of any arguments on the data obtained from a sample. For
> example, Gallup polls typically have a sample size of some 1200 people
> to represent the opinions of more than 250 million Americans. If the
> American public is take as the database, then this means than the sample
> size is 0.0005% of the data base. What is important in the sample
> size of this nature is the number of individuals (n). When the data
> are presented, the confidence interval at the confidence level is given,
> and gives an indication of the applicability of the statements to the
> wider population."
>
> Can anyone advise on this response? If this is a reasonable argument,
> does anyone also have examples that are "better" (i.e. academically
> effective) than a Gallup poll?
>
> To the low number of 259, I acknowledged that this was a low number, and
> ran comparisons from my sample against the data base based on
> geographical location, rural/urban, age and gender which, research has
> shown, might affect the type of response. Geographical location,
> rural/urban and gender were consistent, and the age was statistically
> different, but in a a way that would over-state rather than under state
> the case. (i.e. the survey measured usage, my sample was statistically
> younger than the data base, and, in this case, other research indicates
> that usage amongst the younger population is higher). Is this a fair
> way to check the validity and representivity if the response rate is so
> low?
>
> (I also contacted a number of non-responders to ask their reasons for
> non-response, but that's a question for another time, I think).
>
> BTW - not to put any pressure on anyone, but my PhD is riding on
> this.....:-(.
>
> All comments will be much appreciated.
>
> Regards
>
> Ken
>
> ---------------------------
> Ken Masters
> IT Health Education
> http://www.ithealthed.com
> ____/\/********\/\____
|
