Thanks very much Tom and Eric, great food for thought of the usual MCG high standard!

The context is such that the high PNTA level is in itself a problem we will aim to address with advocacy, explanation, etc. It arguably sits in a context of survey fatigue, alongside the other influencing factors which lead to PNTA for sensitive data points.

But until we can improve the levels of PNTA, we need to make inferences from the data and want to find the model which yields most valuable insight (maximise statistical power and mitigate limitations) with least risk of bias. Not easy I know.

Eric I'm at least heartened by your reassurance from a position of domain expertise that there's no prominent evidence base indicating a pattern for non-responsiveness - eg over-indexed cases more likely to PNTA. Maybe it's Missing More At Random Than I Feared.....

Cheers,
Stephen 

Sent from my iPad

> On 22 Jul 2016, at 13:48, Jensen, Eric <[log in to unmask]> wrote:
> 
> Hi Stephen,
> 
> I am a social scientist, and I teach social statistics and quantitative research methods at the University of Warwick (I also conduct a lot of quantitative research with museum audiences). I largely agree with Tom’s diagnosis here. No amount of fancy statistical tests are going to allow you to magically reverse engineer your data to identify what systematic biases might have been introduced in the non-response to demographic survey items.
> 
> For face-to-face surveys, best practice dictates the use of a ‘refusal log’, where you track any visible characteristics of the respondent (e.g. ‘white’ or ‘non-white’, ‘apparent gender’) to identify systematic biases that may have affected the data. If your survey is purely online and does not afford these options, you may be stuck just acknowledging this as a limitation of your data.
> 
> I am not aware of any robust evidence in the UK showing that there is a persistent pattern of non-response to demographic questions affecting one type of respondent more than another.
> 
> I would strongly advise against one of the possible solutions suggested by Tom below:
> Could you re-run the survey without PNTA as option
> 
> This would be poor practice and could result in people exiting your survey altogether at this point, or putting down false information if they would in fact prefer not to answer.
> 
> His second suggestion of indicating how important this data is (and I would also stress what you are going to do with it) does sound like a promising approach:
> preface it with a statement about how ticking the PNTA box might lead to skewed results
> 
> If you are getting high levels of non-response to demographic questions, it is worth reviewing the quality of the question and response options to ensure they are a good fit with your respondents and easy for them to answer (some pilot testing may be in order).
> 
> I also agree with Mia that your concern for the quality of your data is very admirable!
> 
> Best wishes,
> Eric
> 
> 
> ---------------
> Dr Eric Jensen, Fellow Higher Education Academy
> Associate Professor (Senior Lecturer), Department of Sociology, University of Warwick
> http://warwick.academia.edu/EricJensen
> Latest book: - Doing Real Research (SAGE): https://us.sagepub.com/en-us/nam/doing-real-research/book241193
> Check out a sample chapter here - https://us.sagepub.com/sites/default/files/upm-binaries/73894_Jensen_Chapter_6.pdf
> 
> Sociology at the University of Warwick ranked:
> The Guardian, Complete University Guide and The Times Good University Guide – 3rd
> QS World University Ranking - 23rd
> 
> 
> 
> 
> On 22 Jul 2016, at 13:28, Bilson, Tom <[log in to unmask]<mailto:[log in to unmask]>> wrote:
> 
> Hi Stephen
> 
> I know this problem well, and am familiar with the ways of correcting for non ignorable non responses, but the inevitable question that comes to mind is why would you wish to? Surely if you’ve offered PNTA as a choice, then it too is a result. Personally, I don’t think it's possible to reverse engineer the scientific method to let results compensate for experimental design without introducing conditions and assumptions which, in many respects, undermine the purity of the data: otherwise you’re results are based partly on measurement and partly on speculation.
> 
> I take your point that removing PNTAs might introduce bias, but unless you have a crystal ball then you’ll never know whether this is accurate or misleading. I’m sure there’s existing research which shows how gender, age, race, sexuality, ethnicity, location plays a role in PNTA, but how to move from the general to the specific of the survey you’ve just run might be a problem.
> 
> I tend to be a PNTA sort of person if the survey starts to feel a bit intrusive, or looks like it has an agenda that makes me feel uneasy (and always assume that in doing so I’m ruling myself out of the prize draw at the end :) Could you re-run the survey without PNTA as option, or perhaps preface it with a statement about how ticking the PNTA box might lead to skewed results. I’m sure this never crosses peoples’ minds when they choose this as an option?
> 
> Best, Tom
> 
> The Courtauld Institute of Art, Somerset House, Strand, London, WC2R 0RN
> www.courtauld.ac.uk<http://www.courtauld.ac.uk>
> 
> Now Open at The Courtauld Gallery – Georgiana Houghton: Spirit Drawings
> Until 11 September 2016
> 
> 
> On 22 Jul 2016, at 10:03, Mia <[log in to unmask]<mailto:[log in to unmask]>> wrote:
> 
> I really appreciate your attention to these questions, but personally it's way beyond the realms of my knowledge! Are there others on the list who could suggest R or Python libraries?
> 
> Failing that, the Association of Internet Researchers list at [log in to unmask]<mailto:[log in to unmask]> might have some suggestions.
> 
> Cheers, Mia
> 
> Sent from my handheld computing device
> 
> On 22 Jul 2016, at 08:28, Stephen McConnachie <[log in to unmask]<mailto:[log in to unmask]>> wrote:
> 
> Hi everyone,
> 
> I have a statistical methodology question - what could be more exciting for a damp warm Friday? I realise it's not entirely in the comfort zone of this group, but I thought I'd try before exploring it with statistician contacts and broader research online.
> 
> It's about managing missing data in survey response, where the missing data is Missing Not At Random (MNAR) aka nonignorable nonresponse. I'm interested in any established models to correct for bias. Maybe those of you who have conducted surveys have come across this and found a good, understandable solution?
> 
> I'll explain the problem. Imagine you're conducting a survey where some of the questions are within the 'sensitive data' realm: race, gender, sexuality, disability. Imagine you're getting high 'prefer not to answer' levels , eg 50%. One flawed approach is listwise deletion, meaning that the 50% PNTA is simply excluded from analysis. This introduces a bias risk, because it's unlikely that the nonresponse is random, it's more likely to be meaningful - eg you might argue that over-represented cases - white, heterosexual males without disability - are slightly more likely to PNTA than under-represented cases. So deleting the PNTA is likely to introduce bias in your analysis, even if that nonrandomness is low level. A concrete example: removing 50% PNTA from the gender question might bias your analysis towards misleadingly high % female.
> 
> There are complex statistical methodologies for approaching the management of this problem - multiple imputation, maximum likelihood estimation, etc - but the complexity is daunting to a non-statistician without a software package like Stata. So I wonder if any of you have done this and either found a simple solution or developed a complex solution which is transferable - in other words, does anyone have some Python they can give me / direct me to??
> 
> All the best,
> Stephen
> 
> ****************************************************************
>    website:  http://museumscomputergroup.org.uk/
>    Twitter:  http://www.twitter.com/ukmcg
>   Facebook:  http://www.facebook.com/museumscomputergroup
> [un]subscribe:  http://museumscomputergroup.org.uk/email-list/
> ****************************************************************
> 
> ****************************************************************
>     website:  http://museumscomputergroup.org.uk/
>     Twitter:  http://www.twitter.com/ukmcg
>    Facebook:  http://www.facebook.com/museumscomputergroup
> [un]subscribe:  http://museumscomputergroup.org.uk/email-list/
> ****************************************************************
> 
> 
> 
> 
> The Courtauld Institute of Art is a company limited by guarantee (registered in England and Wales, number 04464432) and an exempt charity. SCT Enterprises Limited is a limited company (registered in England and Wales, number 3137515). Their registered offices are at Somerset House, Strand, London WC2R 0RN. The sale of items related to The Courtauld Gallery and its collections is managed by SCT Enterprises Limited, a wholly owned subsidiary of The Courtauld Institute of Art.
> 
> 
> 
> 
> ****************************************************************
>      website:  http://museumscomputergroup.org.uk/
>      Twitter:  http://www.twitter.com/ukmcg
>     Facebook:  http://www.facebook.com/museumscomputergroup
> [un]subscribe:  http://museumscomputergroup.org.uk/email-list/
> ****************************************************************
> 
> 
> ****************************************************************
>       website:  http://museumscomputergroup.org.uk/
>       Twitter:  http://www.twitter.com/ukmcg
>      Facebook:  http://www.facebook.com/museumscomputergroup
> [un]subscribe:  http://museumscomputergroup.org.uk/email-list/
> ****************************************************************

****************************************************************
       website:  http://museumscomputergroup.org.uk/
       Twitter:  http://www.twitter.com/ukmcg
      Facebook:  http://www.facebook.com/museumscomputergroup
 [un]subscribe:  http://museumscomputergroup.org.uk/email-list/
****************************************************************