|
Hi Bob,
That sounds rather concerning.
On the final point, I would argue that you should care. If you are collecting data anyway, it would be great to have it provide accurate insights about what is working and why to inform your practice.
I have seen this be very useful for museums and other public engagement institutions, identifying key audience needs that had been missed for years and that could be addressed with relatively minor adjustments. Of course, evaluation is not a panacea: I would readily acknowledge that funding is a more pressing concern!
Best,
Eric
|
On 7 Sep 2018, at 22:27, Bob Clark <[log in to unmask]> wrote:
Re: Collecting visitor data
Eric,
I think we all look forward to the results of your work! At present, an organisation based in Scotland has won the hearts and minds of many funders and with their support is very widely rolling out "training" in their approach to museums and other third-sector bodies. When discussing the evaluation of one aspect of my museum's work, my concern was that we did not have the resources to gather information on the scale and at a depth that I instinctively felt should be necessary. The advice received was that I was greatly overcomplicating the matter, and that "this is not a social sciences study". In some ways I'm not sure I care, so long as we can continue to provide funders with data they find acceptable.
Bob
-----
Friday, September 7, 2018, 4:16:37 PM, you wrote:
|
Hi Bob,
No, that is not the case. I have supported a lot of evaluations that were far from properly resourced (ranging from £0 budgets).
Regarding the need to make good evaluation practice as logistically and financially feasible for museum professionals as possible: I am working on it!
Best,
Eric
|
On 7 Sep 2018, at 22:05, Bob Clark <[log in to unmask]> wrote:
Re: Collecting visitor data
Eric,
I'm very aware that you are far from just a theoretician when it comes to museum evaluation, but suspect you are in the fortunate position of only ever being called upon to do such work when it is properly resourced. You thus get good, scientifically-reliable, results.
There is, however, a fundamental difference of viewpoint between where you stand and what represents reality for most of us. You consider resources to be "wasted" when they produce data you feel to be "inaccurate and therefore not helpful for anyone". May I, however, offer you an alternative perspective? However flawed the data generated by vernacular surveys may be, time and again it satisfies the needs of governing bodies and funders, and museums like the one I run get grants. By my definition that is not money wasted, and regardless of how inaccurate it may be, it has certainly been helpful to a lot of people.
I suspect there isn't a single person on this forum who does not wholly accept what you say about the inaccuracy of what the great majority of us are doing. In my view however, until and unless the social sciences world can offer us methodologies that are logistically and financially realistic, we have no option but to meet internal and external demands for information with "inaccurate" results that satisfy the needs of those who require them.
Bob
-----
Friday, September 7, 2018, 2:59:16 PM, you wrote:
|
Hi Bob,
I assure you I am not chanting from the sideline. I've been involved directly in numerous museum evaluations in the UK and internationally (including in developing world contexts where things are even more challenging). I am not trying to be annoying: I just hate to see wasted resources on audience data that are inaccurate, and therefore not helpful for anyone. I am very sympathetic and personally experienced in the practical dilemmas involved.
I agree, sometimes paper surveys are appropriate to include as an option. I just see them used in far more cases than they actually need to be (with associated costs to the museum that I noted below).
Best,
Eric
|
On 7 Sep 2018, at 18:57, Bob Clark <[log in to unmask]> wrote:
Re: Collecting visitor data
Dr Jensen
Your points about the unknowable bias inherent in most visitor surveys are beyond challenge, and have been made often before.
If we had the very substantial resources that would be required to plan and run museum user surveys to the standards of a social sciences research project, I have no doubt that is what all of us would do. But for the great majority of museums, we don't have and will never get that sort of cash for these sorts of activities. At the same time, there is a constant, impossible-to-ignore, demand from strategic bodies, funders and governing bodies for information on who our visitors are, why they come (or don't), what they like (or don't), and so on, and no such bodies would accept us saying that because we cannot provide rigorously-collected, scientifically-reliable data, we are not going to provide any data at all. If we tried that, we'd just be sent away and told to do the best we can with what we've got. Which, most of the time, is what we do, and it is demoralising when the social scientists chant from the sidelines that it is all rubbish. What would be really useful is for the social scientists to undertake a study of common limited-resource survey methodologies, and try to place quantum on the extent and nature of the inherent bias.
I do not believe there is the slightest suggestion here - most certainly there is not on my part - that paper-based surveys generate more scientifically-accurate results than tablet-based surveys. The issue is simply that at least in some locations, paper-based surveys provide more data.
Bob Clark
Director
The Auchindrain Trust
mailto:[log in to unmask]
-----
Friday, September 7, 2018, 12:12:47 PM, you wrote:
|
Yes, it is all about getting the small details right in survey research and evaluation. I would not advise leaving the tablet at a reception desk without a systematic sampling plan (and detailed instructions and training for the reception staff) in place.
My default approach for visitor surveys is to intercept people on their way in to the site (30 seconds - 2 minutes), following a systematic sampling procedure. People complete a relatively short survey on the way in with a data collector either holding the tablet or handing it over temporarily. Then at the end of the day, they automatically receive a post-visit survey request (with automatic reminders). Observable data are collected on non-respondents by the data collector (called a ‘refusal log’) in order to calculate response rate and identify any systematic sampling bias that may be visible.
In order to do this kind of fundamental quality assurance, a person is needed to administer the initial survey (or you need advance access to people’s details). Without this kind of systematic approach, quantitative claims about visitors cannot legitimately be made (e.g. % of people satisfied with their experience) as the level of sampling bias is unknowable.
Of course, this is challenging when you do not ever have a data collector available, so creative solutions have to be identified. I don’t see how paper survey forms per se make these issues better though (and someone then needs to put the time into typing up the data, there is risk of data entry errors, you cannot use skip logic elegantly, etc.).
Best,
Eric
---------------
Dr Eric Jensen, Fellow Higher Education Academy
Associate Professor (Senior Lecturer), Department of Sociology, University of Warwick
http://warwick.academia.edu/EricJensen
Latest book: - Doing Real Research (SAGE): https://us.sagepub.com/en-us/nam/doing-real-research/book241193
Check out a sample chapter here - https://us.sagepub.com/sites/default/files/upm-binaries/73894_Jensen_Chapter_6.pdf
Sociology at the University of Warwick ranked:
The Guardian, Complete University Guide and The Times Good University Guide – 3rd
QS World University Ranking - 23rd
|
On 7 Sep 2018, at 16:20, Elin Bornemann <[log in to unmask]> wrote:
It has been our experience that many visitors are reluctant to interact with tablets as well. We tried doing visitor surveys via a tablet for a while, but it was not a success. We never hit on the right app to use either, they all had problems. It might have worked better if we had used the tablet in conjunction with a person, i.e. have a person approach visitors with tablet in hand. The way we tried it, however, was with the tablet set up on the reception desk, and we didn’t get much data that way. We have now gone back to paper.
Elin Bornemann
Collections Officer
Abingdon County Hall Museum
Market Place
Abingdon
OX14 3HG
01235 523703
From: Museums Computer Group [mailto:[log in to unmask]] On Behalf Of Jensen, Eric
Sent: 07 September 2018 03:36
To: [log in to unmask]
Subject: Re: Collecting visitor data
Just a quick note that I have sometimes found visitors are reluctant to have their data collected via smartphone. (Perhaps because these are so associated with being personal devices)
So I would recommend a low cost android tablet as the on-site data collection device. You can get these got as little as £45 each. (Though if you can afford the £100-120 level, the added processor speed does make the data collection interaction smoother)
Eric
|
|
|
|
|
|
|
|