Hi Bob,
With quantitative claims, I would have a hard time agreeing that a self-selected sample (e.g. comment card box in the corner) is better than nothing. (Especially if it gives you a sense of knowing something when you really don’t).
If that is all you can manage, I would recommend sticking with qualitative reporting.
I am sympathetic to the fact that feasibility / cost is a major barrier to gathering better data for many museums. I have recently written about how options for gathering good visitor data have improved with recent improvements in technology: http://jcom.sissa.it/archive/14/03/JCOM_1403_2015_C01/JCOM_1403_2015_C05
For example, if there is any kind of ticketing involved (as I suspect there would be in your case), setting an automated link to your ticketing software would be a way to intervene and be able to get a survey out to someone to complete online when they get back home. We are still talking about a budget that would need to exceed £0 most likely, but it doesn’t have to be super-expensive.
Also, the alternative of having people do the data entry and number crunching from the low quality comment card style data effectively costs something.
Best,
Eric
---------------
Dr Eric Jensen, Fellow Higher Education Academy
Associate Professor (Senior Lecturer), Department of Sociology, University of Warwick
http://warwick.academia.edu/EricJensen
Latest book: - Doing Real Research (SAGE): https://us.sagepub.com/en-us/nam/doing-real-research/book241193
Check out a sample chapter here - https://us.sagepub.com/sites/default/files/upm-binaries/73894_Jensen_Chapter_6.pdf
Sociology at the University of Warwick ranked:
The Guardian, Complete University Guide and The Times Good University Guide – 3rd
QS World University Ranking - 23rd
On 22 Jul 2016, at 15:43, Bob Clark <[log in to unmask]<mailto:[log in to unmask]>> wrote:
Eric,
That is all actually very helpful, and I'm pleased MCG has provided this opportunity for questions
and answers.
Yes, this is a self-selected sample, because that's the only sort of sample we have the resources
to obtain. That or nothing!
The point about filling in missing data has been a perceived need to have all the totals add up to
the same. But maybe the solution is always to talk in terms of the percentage of those who responded?
Bob Clark
Director
The Auchindrain Trust
mailto:[log in to unmask]
-----
Friday, July 22, 2016, 3:29:01 PM, you wrote:
Hi Bob,
A couple of clarifications about your data situation.
1. When you say ‘self-completion’, do you also mean a ‘self-selected’ sample? (For example,
using comment cards, freestanding kiosks or other methods that mean that people are not being
systematically asked to participate? If so, I am afraid the issues run deep, as the level of
sampling bias for such surveys is completely unknowable, which means any quantitative claims
from these data sources are completely suspect. I recently wrote about this here:
http://www2.warwick.ac.uk/fac/soc/sociology/staff/jensen/ericjensen/evaluation
2. I am not seeing the need to fill in missing data like this. Any statistical analysis you run
will use whatever data you have available that covers the variables you are analysing. So, for
example, just because you are missing income data for someone, doesn’t mean you can’t use the
other data you do have for them. In sum, I am not seeing the problem that needs solving here from a data analysis perspective?
Aside from that, I would strongly advise against the approach of filling in missing data with
an aggregate figure from the rest of your sample. I can see no methodologically or statistically
valid basis for doing this, and it is a textbook application of the ecological fallacy (taking
aggregate level results and assuming they apply at the level of the individual):
https://en.wikipedia.org/wiki/Ecological_fallacy
Best,
Eric
---------------
Dr Eric Jensen, Fellow Higher Education Academy
Associate Professor (Senior Lecturer), Department of Sociology, University of Warwick
http://warwick.academia.edu/EricJensen
Latest book: - Doing Real Research (SAGE):
https://us.sagepub.com/en-us/nam/doing-real-research/book241193
Check out a sample chapter here -
https://us.sagepub.com/sites/default/files/upm-binaries/73894_Jensen_Chapter_6.pdf
Sociology at the University of Warwick ranked:
The Guardian, Complete University Guide and The Times Good University Guide – 3rd
QS World University Ranking - 23rd
On 22 Jul 2016, at 14:16, Bob Clark
<[log in to unmask]<mailto:[log in to unmask]><mailto:[log in to unmask]>> wrote:
This is an interesting strand. Our museum has for several years run a self-completion survey
to gather data on visitor demographics and reactions, the effectiveness of marketing, and so on.
The survey is in English only, and we have always been aware that those who do not complete it
consists of disproportionate levels of overseas visitors and English-speakers who do not fit a
survey matrix of white, educated, middle-class and willing to fill in a post-visit form.
We know we can't correct the data, so when each year's survey is written up we overtly
acknowledge that it almost certainly under-represents the numbers of overseas and C2DE visitors.
Another statistical issue we have with the survey is where an otherwise-useful form gives no
response to one or two questions, for reasons we don't know. What we have done with this is to
disregard forms that are substantially incomplete, but to cover the gaps in those that are
mainly complete by ascribing them values which reflect the proportional split of other
respondents. Thus, we ask "Are away from home on holiday, for at least one night?". If there
is no answer to this question but 78% of respondents said yes, in "yes" we fill in a value of
0.78 for each person covered by the form (group size and makeup is questioned elsewhere), and
22% in "no". This enables us to sustain totals that balance without skewing the results, we
feel. But I would be really interested in what others do in respect of that type of missing data.
Bob Clark
mailto:[log in to unmask]:
----
Jensen, Eric <[log in to unmask]<mailto:[log in to unmask]><mailto:[log in to unmask]>>
To: [log in to unmask]<mailto:[log in to unmask]><mailto:[log in to unmask]>
Date: Friday, July 22, 2016, 1:48:59 PM
Subject: Statistical method for managing missing data without bias risk
===8<==============Original message text===============
Hi Stephen,
I am a social scientist, and I teach social statistics and quantitative research methods at the
University of Warwick (I also conduct a lot of quantitative research with museum audiences). I
largely agree with Tom’s diagnosis here. No amount of fancy statistical tests are going to allow
you to magically reverse engineer your data to identify what systematic biases might have been
introduced in the non-response to demographic survey items.
For face-to-face surveys, best practice dictates the use of a ‘refusal log’, where you track
any visible characteristics of the respondent (e.g. ‘white’ or ‘non-white’, ‘apparent gender’)
to identify systematic biases that may have affected the data. If your survey is purely online
and does not afford these options, you may be stuck just acknowledging this as a limitation of your data.
I am not aware of any robust evidence in the UK showing that there is a persistent pattern of
non-response to demographic questions affecting one type of respondent more than another.
I would strongly advise against one of the possible solutions suggested by Tom below:
Could you re-run the survey without PNTA as option
This would be poor practice and could result in people exiting your survey altogether at this
point, or putting down false information if they would in fact prefer not to answer.
His second suggestion of indicating how important this data is (and I would also stress what
you are going to do with it) does sound like a promising approach:
preface it with a statement about how ticking the PNTA box might lead to skewed results
If you are getting high levels of non-response to demographic questions, it is worth reviewing
the quality of the question and response options to ensure they are a good fit with your
respondents and easy for them to answer (some pilot testing may be in order).
I also agree with Mia that your concern for the quality of your data is very admirable!
Best wishes,
Eric
---------------
Dr Eric Jensen, Fellow Higher Education Academy
Associate Professor (Senior Lecturer), Department of Sociology, University of Warwick
http://warwick.academia.edu/EricJensen
Latest book: - Doing Real Research (SAGE):
https://us.sagepub.com/en-us/nam/doing-real-research/book241193
Check out a sample chapter here -
https://us.sagepub.com/sites/default/files/upm-binaries/73894_Jensen_Chapter_6.pdf
Sociology at the University of Warwick ranked:
The Guardian, Complete University Guide and The Times Good University Guide – 3rd
QS World University Ranking - 23rd
On 22 Jul 2016, at 13:28, Bilson, Tom
<[log in to unmask]<mailto:[log in to unmask]><mailto:[log in to unmask]>> wrote:
Hi Stephen
I know this problem well, and am familiar with the ways of correcting for non ignorable non
responses, but the inevitable question that comes to mind is why would you wish to? Surely if
you’ve offered PNTA as a choice, then it too is a result. Personally, I don’t think it's
possible to reverse engineer the scientific method to let results compensate for experimental
design without introducing conditions and assumptions which, in many respects, undermine the
purity of the data: otherwise you’re results are based partly on measurement and partly on speculation.
I take your point that removing PNTAs might introduce bias, but unless you have a crystal ball
then you’ll never know whether this is accurate or misleading. I’m sure there’s existing
research which shows how gender, age, race, sexuality, ethnicity, location plays a role in PNTA,
but how to move from the general to the specific of the survey you’ve just run might be a problem.
I tend to be a PNTA sort of person if the survey starts to feel a bit intrusive, or looks like
it has an agenda that makes me feel uneasy (and always assume that in doing so I’m ruling myself
out of the prize draw at the end :) Could you re-run the survey without PNTA as option, or
perhaps preface it with a statement about how ticking the PNTA box might lead to skewed results.
I’m sure this never crosses peoples’ minds when they choose this as an option?
Best, Tom
The Courtauld Institute of Art, Somerset House, Strand, London, WC2R 0RN
www.courtauld.ac.uk<http://www.courtauld.ac.uk><http://www.courtauld.ac.uk>
Now Open at The Courtauld Gallery – Georgiana Houghton: Spirit Drawings
Until 11 September 2016
On 22 Jul 2016, at 10:03, Mia <[log in to unmask]<mailto:[log in to unmask]><mailto:[log in to unmask]>> wrote:
I really appreciate your attention to these questions, but personally it's way beyond the
realms of my knowledge! Are there others on the list who could suggest R or Python libraries?
Failing that, the Association of Internet Researchers list at
[log in to unmask]<mailto:[log in to unmask]><mailto:[log in to unmask]> might have some suggestions.
Cheers, Mia
Sent from my handheld computing device
On 22 Jul 2016, at 08:28, Stephen McConnachie
<[log in to unmask]<mailto:[log in to unmask]><mailto:[log in to unmask]>> wrote:
Hi everyone,
I have a statistical methodology question - what could be more exciting for a damp warm Friday?
I realise it's not entirely in the comfort zone of this group, but I thought I'd try before
exploring it with statistician contacts and broader research online.
It's about managing missing data in survey response, where the missing data is Missing Not At
Random (MNAR) aka nonignorable nonresponse. I'm interested in any established models to correct
for bias. Maybe those of you who have conducted surveys have come across this and found a good, understandable solution?
I'll explain the problem. Imagine you're conducting a survey where some of the questions are
within the 'sensitive data' realm: race, gender, sexuality, disability. Imagine you're getting
high 'prefer not to answer' levels , eg 50%. One flawed approach is listwise deletion, meaning
that the 50% PNTA is simply excluded from analysis. This introduces a bias risk, because it's
unlikely that the nonresponse is random, it's more likely to be meaningful - eg you might argue
that over-represented cases - white, heterosexual males without disability - are slightly more
likely to PNTA than under-represented cases. So deleting the PNTA is likely to introduce bias in
your analysis, even if that nonrandomness is low level. A concrete example: removing 50% PNTA
from the gender question might bias your analysis towards misleadingly high % female.
There are complex statistical methodologies for approaching the management of this problem -
multiple imputation, maximum likelihood estimation, etc - but the complexity is daunting to a
non-statistician without a software package like Stata. So I wonder if any of you have done this
and either found a simple solution or developed a complex solution which is transferable - in
other words, does anyone have some Python they can give me / direct me to??
All the best,
Stephen
****************************************************************
website: http://museumscomputergroup.org.uk/
Twitter: http://www.twitter.com/ukmcg
Facebook: http://www.facebook.com/museumscomputergroup
[un]subscribe: http://museumscomputergroup.org.uk/email-list/
****************************************************************
****************************************************************
website: http://museumscomputergroup.org.uk/
Twitter: http://www.twitter.com/ukmcg
Facebook: http://www.facebook.com/museumscomputergroup
[un]subscribe: http://museumscomputergroup.org.uk/email-list/
****************************************************************
The Courtauld Institute of Art is a company limited by guarantee (registered in England and
Wales, number 04464432) and an exempt charity. SCT Enterprises Limited is a limited company
(registered in England and Wales, number 3137515). Their registered offices are at Somerset
House, Strand, London WC2R 0RN. The sale of items related to The Courtauld Gallery and its
collections is managed by SCT Enterprises Limited, a wholly owned subsidiary of The Courtauld Institute of Art.
****************************************************************
website: http://museumscomputergroup.org.uk/
Twitter: http://www.twitter.com/ukmcg
Facebook: http://www.facebook.com/museumscomputergroup
[un]subscribe: http://museumscomputergroup.org.uk/email-list/
****************************************************************
****************************************************************
website: http://museumscomputergroup.org.uk/
Twitter: http://www.twitter.com/ukmcg
Facebook: http://www.facebook.com/museumscomputergroup
[un]subscribe: http://museumscomputergroup.org.uk/email-list/
****************************************************************
===8<===========End of original message text===========
****************************************************************
website: http://museumscomputergroup.org.uk/
Twitter: http://www.twitter.com/ukmcg
Facebook: http://www.facebook.com/museumscomputergroup
[un]subscribe: http://museumscomputergroup.org.uk/email-list/
****************************************************************
****************************************************************
website: http://museumscomputergroup.org.uk/
Twitter: http://www.twitter.com/ukmcg
Facebook: http://www.facebook.com/museumscomputergroup
[un]subscribe: http://museumscomputergroup.org.uk/email-list/
****************************************************************
****************************************************************
website: http://museumscomputergroup.org.uk/
Twitter: http://www.twitter.com/ukmcg
Facebook: http://www.facebook.com/museumscomputergroup
[un]subscribe: http://museumscomputergroup.org.uk/email-list/
****************************************************************
****************************************************************
website: http://museumscomputergroup.org.uk/
Twitter: http://www.twitter.com/ukmcg
Facebook: http://www.facebook.com/museumscomputergroup
[un]subscribe: http://museumscomputergroup.org.uk/email-list/
****************************************************************
|