This is an interesting strand. Our museum has for several years run a self-completion survey to gather data on visitor demographics and reactions, the effectiveness of marketing, and so on. The survey is in English only, and we have always been aware that those who do not complete it consists of disproportionate levels of overseas visitors and English-speakers who do not fit a survey matrix of white, educated, middle-class and willing to fill in a post-visit form.
We know we can't correct the data, so when each year's survey is written up we overtly acknowledge that it almost certainly under-represents the numbers of overseas and C2DE visitors.
Another statistical issue we have with the survey is where an otherwise-useful form gives no response to one or two questions, for reasons we don't know. What we have done with this is to disregard forms that are substantially incomplete, but to cover the gaps in those that are mainly complete by ascribing them values which reflect the proportional split of other respondents. Thus, we ask "Are away from home on holiday, for at least one night?". If there is no answer to this question but 78% of respondents said yes, in "yes" we fill in a value of 0.78 for each person covered by the form (group size and makeup is questioned elsewhere), and 22% in "no". This enables us to sustain totals that balance without skewing the results, we feel. But I would be really interested in what others do in respect of that type of missing data.
Bob Clark
mailto:[log in to unmask]:
----
Jensen, Eric <[log in to unmask]>
To: [log in to unmask]
Date: Friday, July 22, 2016, 1:48:59 PM
Subject: Statistical method for managing missing data without bias risk
===8<==============Original message text===============
Hi Stephen,
I am a social scientist, and I teach social statistics and quantitative research methods at the University of Warwick (I also conduct a lot of quantitative research with museum audiences). I largely agree with Tom’s diagnosis here. No amount of fancy statistical tests are going to allow you to magically reverse engineer your data to identify what systematic biases might have been introduced in the non-response to demographic survey items.
For face-to-face surveys, best practice dictates the use of a ‘refusal log’, where you track any visible characteristics of the respondent (e.g. ‘white’ or ‘non-white’, ‘apparent gender’) to identify systematic biases that may have affected the data. If your survey is purely online and does not afford these options, you may be stuck just acknowledging this as a limitation of your data.
I am not aware of any robust evidence in the UK showing that there is a persistent pattern of non-response to demographic questions affecting one type of respondent more than another.
I would strongly advise against one of the possible solutions suggested by Tom below:
Could you re-run the survey without PNTA as option
This would be poor practice and could result in people exiting your survey altogether at this point, or putting down false information if they would in fact prefer not to answer.
His second suggestion of indicating how important this data is (and I would also stress what you are going to do with it) does sound like a promising approach:
preface it with a statement about how ticking the PNTA box might lead to skewed results
If you are getting high levels of non-response to demographic questions, it is worth reviewing the quality of the question and response options to ensure they are a good fit with your respondents and easy for them to answer (some pilot testing may be in order).
I also agree with Mia that your concern for the quality of your data is very admirable!
Best wishes,
Eric
---------------
Dr Eric Jensen, Fellow Higher Education Academy
Associate Professor (Senior Lecturer), Department of Sociology, University of Warwick
http://warwick.academia.edu/EricJensen
Latest book: - Doing Real Research (SAGE): https://us.sagepub.com/en-us/nam/doing-real-research/book241193
Check out a sample chapter here - https://us.sagepub.com/sites/default/files/upm-binaries/73894_Jensen_Chapter_6.pdf
Sociology at the University of Warwick ranked:
The Guardian, Complete University Guide and The Times Good University Guide – 3rd
QS World University Ranking - 23rd
On 22 Jul 2016, at 13:28, Bilson, Tom <[log in to unmask]<mailto:[log in to unmask]>> wrote:
Hi Stephen
I know this problem well, and am familiar with the ways of correcting for non ignorable non responses, but the inevitable question that comes to mind is why would you wish to? Surely if you’ve offered PNTA as a choice, then it too is a result. Personally, I don’t think it's possible to reverse engineer the scientific method to let results compensate for experimental design without introducing conditions and assumptions which, in many respects, undermine the purity of the data: otherwise you’re results are based partly on measurement and partly on speculation.
I take your point that removing PNTAs might introduce bias, but unless you have a crystal ball then you’ll never know whether this is accurate or misleading. I’m sure there’s existing research which shows how gender, age, race, sexuality, ethnicity, location plays a role in PNTA, but how to move from the general to the specific of the survey you’ve just run might be a problem.
I tend to be a PNTA sort of person if the survey starts to feel a bit intrusive, or looks like it has an agenda that makes me feel uneasy (and always assume that in doing so I’m ruling myself out of the prize draw at the end :) Could you re-run the survey without PNTA as option, or perhaps preface it with a statement about how ticking the PNTA box might lead to skewed results. I’m sure this never crosses peoples’ minds when they choose this as an option?
Best, Tom
The Courtauld Institute of Art, Somerset House, Strand, London, WC2R 0RN
www.courtauld.ac.uk<http://www.courtauld.ac.uk>
Now Open at The Courtauld Gallery – Georgiana Houghton: Spirit Drawings
Until 11 September 2016
On 22 Jul 2016, at 10:03, Mia <[log in to unmask]<mailto:[log in to unmask]>> wrote:
I really appreciate your attention to these questions, but personally it's way beyond the realms of my knowledge! Are there others on the list who could suggest R or Python libraries?
Failing that, the Association of Internet Researchers list at [log in to unmask]<mailto:[log in to unmask]> might have some suggestions.
Cheers, Mia
Sent from my handheld computing device
On 22 Jul 2016, at 08:28, Stephen McConnachie <[log in to unmask]<mailto:[log in to unmask]>> wrote:
Hi everyone,
I have a statistical methodology question - what could be more exciting for a damp warm Friday? I realise it's not entirely in the comfort zone of this group, but I thought I'd try before exploring it with statistician contacts and broader research online.
It's about managing missing data in survey response, where the missing data is Missing Not At Random (MNAR) aka nonignorable nonresponse. I'm interested in any established models to correct for bias. Maybe those of you who have conducted surveys have come across this and found a good, understandable solution?
I'll explain the problem. Imagine you're conducting a survey where some of the questions are within the 'sensitive data' realm: race, gender, sexuality, disability. Imagine you're getting high 'prefer not to answer' levels , eg 50%. One flawed approach is listwise deletion, meaning that the 50% PNTA is simply excluded from analysis. This introduces a bias risk, because it's unlikely that the nonresponse is random, it's more likely to be meaningful - eg you might argue that over-represented cases - white, heterosexual males without disability - are slightly more likely to PNTA than under-represented cases. So deleting the PNTA is likely to introduce bias in your analysis, even if that nonrandomness is low level. A concrete example: removing 50% PNTA from the gender question might bias your analysis towards misleadingly high % female.
There are complex statistical methodologies for approaching the management of this problem - multiple imputation, maximum likelihood estimation, etc - but the complexity is daunting to a non-statistician without a software package like Stata. So I wonder if any of you have done this and either found a simple solution or developed a complex solution which is transferable - in other words, does anyone have some Python they can give me / direct me to??
All the best,
Stephen
****************************************************************
website: http://museumscomputergroup.org.uk/
Twitter: http://www.twitter.com/ukmcg
Facebook: http://www.facebook.com/museumscomputergroup
[un]subscribe: http://museumscomputergroup.org.uk/email-list/
****************************************************************
****************************************************************
website: http://museumscomputergroup.org.uk/
Twitter: http://www.twitter.com/ukmcg
Facebook: http://www.facebook.com/museumscomputergroup
[un]subscribe: http://museumscomputergroup.org.uk/email-list/
****************************************************************
The Courtauld Institute of Art is a company limited by guarantee (registered in England and Wales, number 04464432) and an exempt charity. SCT Enterprises Limited is a limited company (registered in England and Wales, number 3137515). Their registered offices are at Somerset House, Strand, London WC2R 0RN. The sale of items related to The Courtauld Gallery and its collections is managed by SCT Enterprises Limited, a wholly owned subsidiary of The Courtauld Institute of Art.
****************************************************************
website: http://museumscomputergroup.org.uk/
Twitter: http://www.twitter.com/ukmcg
Facebook: http://www.facebook.com/museumscomputergroup
[un]subscribe: http://museumscomputergroup.org.uk/email-list/
****************************************************************
****************************************************************
website: http://museumscomputergroup.org.uk/
Twitter: http://www.twitter.com/ukmcg
Facebook: http://www.facebook.com/museumscomputergroup
[un]subscribe: http://museumscomputergroup.org.uk/email-list/
****************************************************************
===8<===========End of original message text===========
****************************************************************
website: http://museumscomputergroup.org.uk/
Twitter: http://www.twitter.com/ukmcg
Facebook: http://www.facebook.com/museumscomputergroup
[un]subscribe: http://museumscomputergroup.org.uk/email-list/
****************************************************************
|
