My excessive concern was built iteratively - over a number of years
dealing with submissions where students paid cursory attention to the
creation of a representative sample (commonly without a clear definition
of the population of interest) and focus almost entirely on the
questionnaire.  Indeed, many (staff and students) interchange the terms
'questionnaire' and 'survey'.   Methodological limitations are then
skirted over in the write up to make way for ... tests of statistical
significance (although, on this foundation it is clear that these
students have a more limited understanding of what statistical
significance means).   The size of a sample is of concern (must be big
to be good) but generally rules of thumb are drawn upon rather than
sampling theory.  Sometimes, randomness is claimed ('I handed out the
questionnaires randomly to who ever came by') - not helped by the
increasing use of the 'random' Americanism (you were 'totally random'
there).

The curriculum evolved so that more time was devoted to the concept of
randomness and statistical significance ...with limited success - hence
my increasing entrenchment.

I do see it as valuable for our undergrads to experience designing and
conducting a questionnaire survey.  This is done in the first year -
students use seminars to design items for a questionnaire that is used
for a student survey; they conduct it, input the data and analyse the
results (only using descriptive statistics).  From the second year, to
get into more advanced and inferential statistics we turn to secondary
data (British Crime Survey, Youth Cohort, European Social Survey are the
three we are going with this year).  Students are taught statistical
inference and are expected to draw on this to critically assess the data
that they choose to analyse (in terms of sampling and measurement).   We
felt that our students were likely to have had a fair amount of
practical doing-surveys experience prior to the degree (Alevel or
before) and saw little value to proceeding with this (beyond the first
year) given the (fairly inevitable) data quality problems that arise.
The free and fairly painless access that students now have to secondary
survey data that uses random sampling is why we encourage those who wish
to undertake entirely quantitative dissertations in this direction.   Of
course, using secondary data results in a loss of autonomy - for those
who have particular questions / desire to undertake primary research we
strongly encourage that the statistical analysis remain descriptive (but
could include discussion / argument around generalising by drawing on
additional sources / evidence) and suggest that they supplement the
method with another (media analysis, in-depth interviews etc.).   I am
based within Sociology, Social Policy and Politics - where purely
quantitative dissertations are in the minority (and so I/we wanted to
ensure that they were of a certain standard).   

In psychology, purely quantitative projects are more common - and it was
the 'doesn't matter its just philosophical' perspective of psychology
tutors and the acknowledgement that generalisations are commonly made
from non-random samples in psychology journals that made me seek your
views.    

My understanding of claiming a finding (association, difference) to be
statistically significant is that the patterns observed in a sample are
unlikely to be due to chance (random sampling variation) and therefore
reflect a 'true' pattern within the parent population.  The test assumes
randomness is present (reasonable if the sample was random) and takes
sample size into account.  I take the point that a random sample does
not ensure a representative sample - although the larger the sample, the
lower the risk of such an unrepresentative sample (and the use of
improvements such as stratifying the sample also help).  It is
theoretically possible for a random and a convenience sample to produce
exactly the same participants - but in the long run, when possible,
randomness is the best way of getting representation (as long as the
sample is of sufficient size & response rate is OK).   Replication is
important - if a similar (statistically significant or insignificant)
finding emerges from multiple randomly sampled sources it clearly
strengthens the inference - I do have more doubts about whether this
would be the case with non-randomly sampled ones.

I am reading the discussion with interest, and reflecting on my
position.   And the Cathie Marsh lecture looks interesting - the
virtualising of the survey method seems to be associated with an
increase in non-random, self selecting samples.  The one that we (and no
doubt many of you) are told to pay attention to is the national student
survey (which is an attempt at a census) - an example of a tension
between democratic representation (everybody gets a chance to
participate) and statistical representation (where the sample is a
reflection of the population).

Best Wishes.

Sean

-----Original Message-----
From: email list for Radical Statistics [mailto:[log in to unmask]]
On Behalf Of John Whittington
Sent: 15 September 2008 14:22
To: [log in to unmask]
Subject: Re: Randomness, Statistical Significance and Generalisation

At 13:54 15/09/2008 +0100, Mike Brewer wrote (in part):

>Most posts have focused on the issue of generalising from a finding
that 
>is true, given on the data/sample to a finding that is true for the UK 
>population. It seems to me that this is never a good idea unless the 
>probability of being in the sample is random, given the observable 
>characteristics of the person being interviewed (isn't that what we
mean 
>by a random sample?).

I think this illustrates the confusion which often exists (particularly
in 
the general public) between 'random' and 'representative'.

As I said, what we mean by a (single) simple random sample is that it
can 
be ANYTHING, including totally unrepresentative of the parent
population.

The expectation is that a random sample will be representative of the 
population from which it is drawn, and the chance of that approximating
to 
being true obviously increases as the sample size increases.  However 
(regardless of how large the sample is), there is absolutely no
guarantee 
that a single simple random sample will be at all representative of the 
parent population - and the smaller the sample, the greater the risk
that 
it will be unrepresentative, even though random.

I therefore do not think it is theoretically safe to assume that a
sample 
is necessarily representative of the parent population (thereby
justifying 
generalisation of results obtained from that sample to the general 
population) just because sampling was at random.

Kind Regards,


John

----------------------------------------------------------------
Dr John Whittington,       Voice:    +44 (0) 1296 730225
Mediscience Services       Fax:      +44 (0) 1296 738893
Twyford Manor, Twyford,    E-mail:   [log in to unmask]
Buckingham  MK18 4EL, UK
----------------------------------------------------------------

******************************************************
Please note that if you press the 'Reply' button your
message will go only to the sender of this message.
If you want to reply to the whole list, use your mailer's
'Reply-to-All' button to send your message automatically
to [log in to unmask]
Disclaimer: The messages sent to this list are the views of the sender
and cannot be assumed to be representative of the range of views held by
subscribers to the Radical Statistics Group. To find out more about
Radical Statistics and its aims and activities and read current and past
issues of our newsletter you are invited to visit our web site
www.radstats.org.uk.
*******************************************************

******************************************************
Please note that if you press the 'Reply' button your
message will go only to the sender of this message.
If you want to reply to the whole list, use your mailer's
'Reply-to-All' button to send your message automatically
to [log in to unmask]
Disclaimer: The messages sent to this list are the views of the sender and cannot be assumed to be representative of the range of views held by subscribers to the Radical Statistics Group. To find out more about Radical Statistics and its aims and activities and read current and past issues of our newsletter you are invited to visit our web site www.radstats.org.uk.
*******************************************************