Hi Bruce,

Thanks for asking. I appreciate that available time and resources for evaluation are typically highly limited for science communication practitioners outside of the larger institutions in the UK (although the larger national institutions are not necessarily producing the best evaluation they could be- see below).

There are two main suggestions I would like to emphasise: 

1. 

In principle there is no reason that evaluation data have to be collected from every single participant in your intervention / event / etcetera. By using a systematic form of sampling, you can get a good approximation of the experience and impacts of the population you are measuring. Sampling allows limited resources to be focused on collecting and analysing higher quality data from a smaller number of people (e.g. you might collect data from four randomly selected classes over the course of the year, rather than thousands of pupils). 

2. 

If you would like to measure the impacts from your intervention, you have two main options:

(A) Repeated Measures Design: Direct measurement of your target outcomes (at least) pre- and post-visit.

(B) Experimental Design: In this case, random assignment of individuals or classes to receive the intervention or not. In its simplest form, this design only requires post-visit data collection.

I have provided an example of surveys I designed for pupils within a repeated measures design, which I have placed at this web address in Word .doc format in case anyone would like to see/'borrow'/adapt the template: http://www2.warwick.ac.uk/fac/soc/sociology/news/scd/publicengagement/#Survey_Example

This website houses the videos and resources from the seminar series I organised in 2011-12 on 'Evaluating impacts of informal learning and public engagement', which goes through different evaluation options in detail.

There is also a more general issue about the quality of the survey questions that are used, as poorly designed questions can invalidate any data that are collected. For example, the CREST evaluation that Adrian mentioned makes impact claims (indeed it is labelled an 'impact report'), yet the closed-ended survey questions used to produce the impact findings are highly problematic. The particular impact claim Adrian mentioned was based on the question, 'Did CREST change the way you feel generally about the idea of a future career in STEM?', which was asked within a post-visit only survey. This question is flawed: It asks for retrospective self-report which requires the respondent to accurately assess how they felt about STEM subjects before the intervention (even though they are being asked after the intervention). The most likely explanation for answers to this question is that the respondents were able to clearly see what they were expected to say by the organisers/evaluators and they largely obliged. The simplest valid way of collecting this data would have been to assess sentiment towards STEM careers directly, both before and after the intervention. The bare minimum that should have been done with the data collected in this case is to describe it as 'self-reported benefits or changes', rather than as actual 'impact'. 

I have written a commentary about the importance of improving quality in science communication evaluation:

- Very short version on the Nature site: http://www.nature.com/nature/journal/v469/n7329/full/469162c.html

- On my academia.edu website: http://www.academia.edu/423049/Evaluate_Impact_of_Communication

- Longer version here: http://www.britishscienceassociation.org/people-science-magazine/june-2011/regular-items-june-2011/evaluate-engagement

I hope this is helpful.

Best wishes,

Eric