JiscMail Logo
Email discussion lists for the UK Education and Research communities

Help for RADSTATS Archives


RADSTATS Archives

RADSTATS Archives


RADSTATS@JISCMAIL.AC.UK


View:

Message:

[

First

|

Previous

|

Next

|

Last

]

By Topic:

[

First

|

Previous

|

Next

|

Last

]

By Author:

[

First

|

Previous

|

Next

|

Last

]

Font:

Proportional Font

LISTSERV Archives

LISTSERV Archives

RADSTATS Home

RADSTATS Home

RADSTATS  September 2008

RADSTATS September 2008

Options

Subscribe or Unsubscribe

Subscribe or Unsubscribe

Log In

Log In

Get Password

Get Password

Subject:

Re: Randomness, Statistical Significance and Generalisation

From:

Paul Bivand <[log in to unmask]>

Reply-To:

Paul Bivand <[log in to unmask]>

Date:

Mon, 15 Sep 2008 11:01:13 +0100

Content-Type:

text/plain

Parts/Attachments:

Parts/Attachments

text/plain (244 lines)

I agree, but think we should remember the context of undergraduate final year. Others who teach students at that level are better placed than me to advise on the correct level needed.

I'm inclined to think that experience of conducting a set of survey interviews might be very useful, although what might be more useful is combining that with cognitive testing of the questions. The widespread problem of survey respondents providing the answers they think the interviewer/survey designer wants might usefully be assessed. Equally, there is place for demonstrating technical competence in statistical analysis, while the important bit we are discussing is how or whether one can generalise from the results.


---------------------------------------------------------
Paul Bivand
Head of Analysis and Statistics
Direct Line: 020 7840 8335

Inclusion has a launched a new website, please visit: www.cesi.org.uk

Inclusion
3rd floor, 89 Albert Embankment, London SE1 7TP
Tel: 020 7582 7221
Fax: 020 7582 6391
Inclusion website: www.cesi.org.uk

Consider the environment - do you really need to print this email?

The contents of this e-mail and any attachment(s) are intended solely for the use of the addressee. If you are not the intended recipient please return the e-mail to the sender and delete from your mailbox.

The Centre for Economic & Social Inclusion is a company limited by guarantee. Registered in England & Wales number 2458694. Registered address: 89 Albert Embankment, London SE1 7TP


-----Original Message-----
From: email list for Radical Statistics [mailto:[log in to unmask]] On Behalf Of Ray Thomas
Sent: 15 September 2008 10:43
To: [log in to unmask]
Subject: Re: Randomness, Statistical Significance and Generalisation

I agree with Jane's message.

But the question raised in my mind is about why students are expected to
conduct a survey as part of their dissertation?   Isn't a survey conducted
by an individual without any resources unavoidably trivial?   The population
is akin to 'the man in the street'.

Much better to write a plan for a proper survey as Jane suggests.

Would it not also be more educative to evaluate one, or two, existing surveys or evaluate one, or two, of the thousands of existing statistical series?

Ray Thomas, Open University
*********************************

-----Original Message-----
From: email list for Radical Statistics [mailto:[log in to unmask]] On Behalf Of Jane Galbraith
Sent: 14 September 2008 09:22
To: [log in to unmask]
Subject: Re: Randomness, Statistical Significance and Generalisation

Dear Sean and all,
Everything Sean says and everything Martin says is right. The trick is to make the students answer the questions: "What is the population from which this might be regarded as a random sample?" and "How does that population differ from the one in which I am interested?"

Psychology students and others need to understand the concept of BIAS first. This is the major problem arising from selection effects and non-response (as well as non-sampling sources of bias).

The concept of RANDOMNESS is important in estimation and testing, in judging whether there is enough information in the sample to draw useful conclusions about the population from which the  achieved sample might be regarded as a random sample.

My solution to student projects would be to get the students to write them up as preliminary investigations, possibly the end point would be a plan for a well designed survey. Estimates and tests could be included as illustrative and/or as aids in designing the survey. (eg do we need to stratify by age?).

I have found student (and many professional) questionnaires are often badly designed. The sampling scheme is not the only problem is drawing inferences. A student project should evaluate the questionnaire used (and preferably alternative versions), and consider other sources of bias.

Furthermore, as Martin has pointed out, we may wish to generalise to populations from which we could not draw a random sample.

Best wishes
Jane

PS This year's Cathie Marsh lecture might be relevant:

Cathie Marsh Memorial Lecture
18th November 2008
5pm-7pm (Tea and Registration from 4.30pm) Royal Statistical Society
12 Errol Street, London EC1Y 8LX

Are web-based surveys the survey method of the future?


>>>>>>>Martin Bland wrote>>>>>>>>


> I do not think that random sampling is required for significance
testing.  If it is then I have been wrong all my life!  Clinical trials , for example, are never done on random samples, but on the patients who happen to turn up and agree to take part.  Agricultural experiments, where Fisher started, are not done on random samples.  We would hire a corner of a field and set up a series of plots.  Treatments were applied to plots chosen randomly, but they were a random sample only of plots, not the field, let alone the crop across the nation.  We might say that the results apply to the population of which the sample might be considered a random sample.  Where random sampling is more important is in estimation, as a confidence interval can apply only to a population of which this sample is representative.  Even so, when estimating the difference between means of two groups, as in both clinical and
> agricultural experiments, we assume that as the two randomised groups
come from the same population before treatment, a difference is likely to apply to other bits of the field too.  Even in non-randomised studies, when it was found that people with lung cancer were
> significantly more likely to have smoked cigarettes than people who
> had
other diseases, we did not worry that they were not a random sample. The point was that we would be unlikely to get a difference of this size in a sample if in the population which they represent smoking and lung cancer were not related.  In these studies we are most concerned about patients we haven't yet met, crops we haven't yet planted, people who have not started to smoke. The population is an infinite one stretching into the future and we could never take a random sample of it.
>
> Where your students would go wrong is in estimating a mean or a
> proportion from a convenience sample and then applying it to the
> general
population.  If you ask what proportion of the population think there should be a congestion charge, a convenience sample at a bus stop might give a very different answer to a convenience sample at a car park.  The confidence intervals would be meaningless for the general population.
>
> Martin
>
> Demack, Sean wrote:
>> Hi All
>> I am primarily seeking advice regarding the use of tests of
>> statistical
significance to generalise from social surveys.  My concerns relate to the use of the survey method and the assumption of a random sample.  Many surveys (most?) do not use random sampling.  This may be due to practicalities such as a lack of (or difficulty of obtaining) a sample
frame.   Student feedback 'surveys' are often attempts at a census -
questionnaires are emailed out or suck on a website (or virtual learning
>> environment) and all are encouraged to participate - but response is
never (even close to) 100%.   The national student survey also uses this
>> census approach.  Within my department, psychology students and staff
(in particular) use some fairly sophisticated statistical techniques (awash with p-values) on non-random (often, convenience / self
>> selecting) samples.
>> These approaches are pragmatic.  Their widespread use and seemingly
lack
>> of concern from those that use them has made me ponder on my own
dogmatic perspective.   The psychology degree has (a highly valued)
accreditation from the British Psychology Society and the degree is
designed / developed in consultation with this society's guideline.    Am
I precious for being concerned? - I see psychology as a subject area with an increasing social influence (on social policy for example) and a
>> lack of concern for fundamental assumptions (or cursory
>> consideration)
makes me wonder.
>> My background is in applied statistics; I am part of the social
>> science
research methods group and have responsibility for teaching (primarily
quantitative) research methods across the undergrad and postgrad
programs.    A few years ago it became fairly apparent that our
students
>> commonly had a rather underdeveloped idea of randomness - and limited
appreciation of how this relates to statistical inference.  As a group we focused on this - stressing the need of a sample frame and some form of random selection and that standing on a street / part of the campus selecting (perhaps in a haphazard way) the sample was not random. As well as developing students understanding of sampling (and how it relates to generalisation through statistical inference), we really wanted to deter students from undertaking a final year (undergrad) dissertation
based solely on a student designed (non-random sampled) survey.   To try
to get students to appreciate that statistics from such
>> surveys could not be (validly) generalised from.   Students are now
(strongly) encouraged to supplement a survey with another methodology (such as in-depth interviewing, focus groups or secondary data
>> analysis), honestly discuss their sampling and avoid entering the
>> world
of test-led analyses.  Students who wish to undertake a dissertation with an essentially quantitative methodology are directed towards data archives
and secondary data sources.   At one point I became so
inundated with queries from students asking about 'which test to use' on
>> their (non-random) samples I put together a 2-page handout that
attempted to (fairly strongly) dissuade and explain why (I have attached
>> this).
>> Things seemed to be fairly successful (although the widespread media
use
>> of 'margins or error', 'significant difference' etc. on clearly
non-random samples must serve to confuse/scupper this).   Within the
dissertation we run drop-in workshops (at the design and analysis
stages) - within one of the analysis ones the discussion (inevitably) came
round to statistical significance and generalisation.   The result was a
number of (psychology joint) students who became anxious about how
>> this impacted on their final submission.  This was followed by a
plethora of emails from the supervisors and students in which the assumption of random sampling was regarded as 'philosophical', the student should not be concerned about it and proceed with their MANOVA or whatever.  I see it differently but also did not see it as a reason for bringing a student's marks down (as they were following their tutors' advice).  I then went to discuss the issue informally with the head of methods for psychology students - who stated that much (most) psychological peer-reviewed quantitative research ignored the random sampling assumption but still went ahead using tests of statistical
significance (even calculating power) .    I thought this may be an issue
that related to differences between experimentalists and survey researchers but it became clear that surveys and generalisation were the
>> main reasons for the use of p-values etc.
>> It seems to be a tension between the pragmatic and the dogmatic - and
my
>> main reason for emailing is to seek comment
>> - does it matter?
>> - am I over obsessing about something that is so widely ignored?
>> There
seems to be a (kind of macho) perspective that quantitative analyses need to be complex and p-value heavy for it to be regarded as 'quality' - and
hence attract high marks.   This runs counter to my perspective -
simplicity, clarity and critical thinking is all; p-values
>> (when appropriate) can be useful additions but the main story lies
within the descriptive analyses.  The most complex technique our students use is (binary) logistic regression - p-values are present in assessing the model but the story comes from the (simpler and clearer)
odds-ratios.    If they used this technique on a non-random sample they
would not use confidence intervals and stress that the findings related solely to their sample; if they used the British Crime Survey, Youth Cohort Study etc. they include the intervals and talk about statistical significance and generalisation.
>> Sorry this is a long one - this has been a nagging issue and I would
really appreciate members perspectives as the new academic year arrives.
>> Best Wishes
>> Sean Demack
>> Senior Lecturer in Sociological Research Methods
>> ******************************************************
>> Please note that if you press the 'Reply' button your message will go
>> only to the sender of this message.
>> If you want to reply to the whole list, use your mailer's
>> 'Reply-to-All' button to send your message automatically to
>> [log in to unmask]
>> Disclaimer: The messages sent to this list are the views of the
>> sender
and cannot be assumed to be representative of the range of views held by
>> subscribers to the Radical Statistics Group. To find out more about
Radical Statistics and its aims and activities and read current and past
>> issues of our newsletter you are invited to visit our web site
>> www.radstats.org.uk.
>> *******************************************************
>
> --
> ***************************************************
> J. Martin Bland
> Prof. of Health Statistics
> Dept. of Health Sciences
> Seebohm Rowntree Building Area 2
> University of York
> Heslington
> York YO10 5DD
>
> Email: [log in to unmask]
> Phone: 01904 321334     Fax: 01904 321382
> Web site: http://martinbland.co.uk/
> ***************************************************
>
> ******************************************************
> Please note that if you press the 'Reply' button your message will go
> only to the sender of this message.
> If you want to reply to the whole list, use your mailer's
> 'Reply-to-All' button to send your message automatically to
> [log in to unmask]
> Disclaimer: The messages sent to this list are the views of the sender
and
> cannot be assumed to be representative of the range of views held by
subscribers to the Radical Statistics Group. To find out more about Radical Statistics and its aims and activities and read current and past issues of our newsletter you are invited to visit our web site
> www.radstats.org.uk.
> *******************************************************
>


--
Mrs Jane Galbraith
Honorary Research Associate
Department of Statistical Science
University College London
Gower Street
London WC1E 6BT

[log in to unmask]

******************************************************
Please note that if you press the 'Reply' button your message will go only to the sender of this message.
If you want to reply to the whole list, use your mailer's 'Reply-to-All' button to send your message automatically to [log in to unmask]
Disclaimer: The messages sent to this list are the views of the sender and cannot be assumed to be representative of the range of views held by subscribers to the Radical Statistics Group. To find out more about Radical Statistics and its aims and activities and read current and past issues of our newsletter you are invited to visit our web site www.radstats.org.uk.
*******************************************************

******************************************************
Please note that if you press the 'Reply' button your message will go only to the sender of this message.
If you want to reply to the whole list, use your mailer's 'Reply-to-All' button to send your message automatically to [log in to unmask]
Disclaimer: The messages sent to this list are the views of the sender and cannot be assumed to be representative of the range of views held by subscribers to the Radical Statistics Group. To find out more about Radical Statistics and its aims and activities and read current and past issues of our newsletter you are invited to visit our web site www.radstats.org.uk.
*******************************************************

******************************************************
Please note that if you press the 'Reply' button your
message will go only to the sender of this message.
If you want to reply to the whole list, use your mailer's
'Reply-to-All' button to send your message automatically
to [log in to unmask]
Disclaimer: The messages sent to this list are the views of the sender and cannot be assumed to be representative of the range of views held by subscribers to the Radical Statistics Group. To find out more about Radical Statistics and its aims and activities and read current and past issues of our newsletter you are invited to visit our web site www.radstats.org.uk.
*******************************************************

Top of Message | Previous Page | Permalink

JiscMail Tools


RSS Feeds and Sharing


Advanced Options


Archives

March 2024
February 2024
January 2024
December 2023
November 2023
October 2023
September 2023
August 2023
July 2023
June 2023
May 2023
April 2023
March 2023
February 2023
January 2023
December 2022
November 2022
October 2022
September 2022
August 2022
July 2022
June 2022
May 2022
April 2022
March 2022
February 2022
January 2022
December 2021
November 2021
October 2021
September 2021
August 2021
July 2021
June 2021
May 2021
April 2021
March 2021
February 2021
January 2021
December 2020
November 2020
October 2020
September 2020
August 2020
July 2020
June 2020
May 2020
April 2020
March 2020
February 2020
January 2020
December 2019
November 2019
October 2019
September 2019
August 2019
July 2019
June 2019
May 2019
April 2019
March 2019
February 2019
January 2019
December 2018
November 2018
October 2018
September 2018
August 2018
July 2018
June 2018
May 2018
April 2018
March 2018
February 2018
January 2018
December 2017
November 2017
October 2017
September 2017
August 2017
July 2017
June 2017
May 2017
April 2017
March 2017
February 2017
January 2017
December 2016
November 2016
October 2016
September 2016
August 2016
July 2016
June 2016
May 2016
April 2016
March 2016
February 2016
January 2016
December 2015
November 2015
October 2015
September 2015
August 2015
July 2015
June 2015
May 2015
April 2015
March 2015
February 2015
January 2015
December 2014
November 2014
October 2014
September 2014
August 2014
July 2014
June 2014
May 2014
April 2014
March 2014
February 2014
January 2014
December 2013
November 2013
October 2013
September 2013
August 2013
July 2013
June 2013
May 2013
April 2013
March 2013
February 2013
January 2013
December 2012
November 2012
October 2012
September 2012
August 2012
July 2012
June 2012
May 2012
April 2012
March 2012
February 2012
January 2012
December 2011
November 2011
October 2011
September 2011
August 2011
July 2011
June 2011
May 2011
April 2011
March 2011
February 2011
January 2011
December 2010
November 2010
October 2010
September 2010
August 2010
July 2010
June 2010
May 2010
April 2010
March 2010
February 2010
January 2010
December 2009
November 2009
October 2009
September 2009
August 2009
July 2009
June 2009
May 2009
April 2009
March 2009
February 2009
January 2009
December 2008
November 2008
October 2008
September 2008
August 2008
July 2008
June 2008
May 2008
April 2008
March 2008
February 2008
January 2008
December 2007
November 2007
October 2007
September 2007
August 2007
July 2007
June 2007
May 2007
April 2007
March 2007
February 2007
January 2007
2006
2005
2004
2003
2002
2001
2000
1999
1998


JiscMail is a Jisc service.

View our service policies at https://www.jiscmail.ac.uk/policyandsecurity/ and Jisc's privacy policy at https://www.jisc.ac.uk/website/privacy-notice

For help and support help@jisc.ac.uk

Secured by F-Secure Anti-Virus CataList Email List Search Powered by the LISTSERV Email List Manager