Hi Tom

Actually it is an easy-peasy evaluation. You have no choice. It has to be a theory-driven evaluation (supported, perhaps, with a realist synthesis).
The overall strategy is to try to pin down the ‘programme theory’ in operation at GMC and to evaluate the extent to which it comes to pass.

I imagine that you are surrounded by all sorts of policy narratives (statutes, mission statements, guidance documents). You need to select a sample of these and ask yourselves (formal interviews with the good, the great and the gaulieters at GMC) – How do we suppose that these ideas will be influential? What are the ways and means through which they will pass into practice? The responses are the programme theory (these might include ideas on why sometimes the policy ideas fall on stony ground).

Then, having decided on a subset of programme theories, you need to put them to some kind of empirical test (OK this is not so easy). The simplest way is to ask groups on the ‘receiving end’ about how they have acted on (or fail to act upon) the regulations, the missives, the pep-talks. Much harder would be operationalise some of the expectations within the regulations and measure any change that has ensued. Of course, you don’t have a counterfactual but the combination of explanatory narratives from all stakeholders plus some administrative data displaying change might help to establish some causal patterns.

The other starting point is with existing ‘regulation theory’. I’m no expert in this area but everything under the sun has been put to critical analysis and this literature is also a higher level source of programme theory (which sometimes will have received formal evaluation).

Sources for this? I recall somewhere that John Braithwaite has written a lot about ‘responsive regulation’. Didn’t that closet realist, Keiran Walshe, write about HNS regulation? Many of the ideas on regulation border on ‘public management’ theory, where Christopher Pollitt is your man. A synthesis of this literature would underpin your specific evaluation.

Have fun
RAY
________________________________________
From: Realist and Meta-narrative Evidence Synthesis: Evolving Standards [[log in to unmask]] On Behalf Of Tom Stocker (0207 189 5076) [[log in to unmask]]
Sent: Thursday, November 27, 2014 11:47 PM
To: [log in to unmask]
Subject: a possibly quite hard evaluation project

Hi RAMSES,
At the GMC we’re thinking about how we might approach a fairly comprehensive evaluation of what we do as a professional medical regulator. (We’re not decided that it will go ahead yet – depends on resource and priorities and what Directors think we’ll get from it). We’re thinking that a realist/ mixed methods approach would be best suited.
We’re just at the beginning, but aren’t finding many evaluation examples or much literature on the types of interventions we’ll be looking at (except perhaps legal reviews of regulators, or very limited evaluations we’ve done on parts of our functions or other UK professional regulators have attempted).
i.e. the effect of sanctions and the specific or background threat of them, applied globally, with no counterfactual, looking to change behaviours and outcomes for patients, with many functions attempting to control practices that are very hard to detect in the first place (many are actively hidden).
I thought you would be a great group of people to advise
a). Any tips for beginners looking for a search strategy for evaluations of this type or evaluations that will inform this type of evaluation
b). Any examples of evaluations that you think might be helpful
c). General tips, advice, and words of caution
I realise that these might be basic questions, so very happy for people to email me directly and save the list from having to read [log in to unmask]<mailto:[log in to unmask]>
Very best wishes and thank you,
Tom Stocker
Insight Analyst, General Medical Council
02071895076

From: Realist and Meta-narrative Evidence Synthesis: Evolving Standards [mailto:[log in to unmask]] On Behalf Of Gillian Westhorp
Sent: 27 November 2014 22:53
To: [log in to unmask]
Subject: Re: clarification

What a wonderful distillation of the ingredients! Thanks Nick,
Gill
Sent via BlackBerry® from Telstra
________________________________
From: Nick Emmel <[log in to unmask]<mailto:[log in to unmask]>>
Sender: "Realist and Meta-narrative Evidence Synthesis: Evolving Standards" <[log in to unmask]<mailto:[log in to unmask]>>
Date: Thu, 27 Nov 2014 15:06:27 +0000
To: <[log in to unmask]<mailto:[log in to unmask]>>
ReplyTo: "Realist and Meta-narrative Evidence Synthesis: Evolving Standards" <[log in to unmask]<mailto:[log in to unmask]>>, Nick Emmel <[log in to unmask]<mailto:[log in to unmask]>>
Subject: Re: clarification

Dear Lisa and everyone,
As a member of the LEEDS group, I think, I have to confess that I have no idea what the LEEDS group is, stands for, or the version of realism to which we all adhere. This is because, up here is Yorkshire (a.k.a. God’s own country), we are, to use one of Ray’s phrases, a disputatious community of truth seekers forever trying to work out how we investigate complex social systems. This means I cannot point you or the member of your academic committee to the methods cookbook, ‘How to do realist research of complex systems’. Cookbooks we use when we want to knock up a nice Spanish stew, not to design investigations. There are a number of methodological ingredients, however. We are faced with a similar challenge to that faced by a professional cook on a tea time cookery programme, who is presented with difficult to deal with ingredients and told to produce an edible meal, expertise, creativity, and experience are brought to bear on the cooking. I suggest our methodological ingredients include:
A contention that there is a reality independent of our knowing it.
Neither empirical observation, nor theories about the world account for social reality, it is far richer and deeper than that.
We most often describe reality as stratified, the empirical, the actual and the real.
In explanations we want to account for real causal mechanisms. These are, according to Roy Bhaskar (2008), the powers of things, which unlike events persist or are at least relatively enduring.
All researchers construct accounts of social objects. But as Joe Maxwell (2012: pg.13) contends, ‘our concepts refer to real phenomena, rather than being abstractions from sense data or purely our own constructions’.
These weak constructions raise consciousness about that which we seek to interpret and explain.
Investigation zigzags between ideas and evidence.
Methods and samples are always chosen in the service of testing, refining, judging, elaborating, …, ideas.
Explanation is an effort to work out the relations between ideas and evidence.
These relations we present as models (or less grandly, ideas on the backs of envelopes—cf. Greenhalgh et al., 2009), which can be transferred from one complex system to another to be tested and refined.
Interpretations and explanations—the insiders’ perspectives and the outsiders’ understandings—cannot be separated and they are always provisional.
Explanations are implicated in theories of the middle range, which seek to explain what works for whom in what circumstances and why (Pawson and Tilley, 1997; Pawson, 2006; 2013).
For some realists the idea of C+M=O, that ‘ugly circumlocution’ (Pawson, 2013:21), is the best way of checking the explanation is complete. But all realist explanations will explain the entanglement of how generative mechanisms act on social regularities in specific contexts to produce particular outcomes.
(after Emmel, 2013)
Good luck mixing the ingredients!

Best wishes

Nick

PS: I should tell you that we are in the process of setting up Realism@Leeds, another place for the disputatious community of truth seekers to interact. More to follow as we get established.
References

Bhaskar R 2008, A realist theory of science Verso, London.
Emmel N 2013, Sampling and choosing cases in qualitative research: a realist approach Sage, London.
Greenhalgh T, Humphrey C, Hughes J, Macfarlane F, Butler C, & Pawson R 2009, "How do you modernize a health service? A realist evaluation of whole-scale transformation in London", Milbank Quarterly, vol. 87, no. 2, pp. 391-416.
Maxwell JA 2012, A realist approach to qualitative research Sage, London.
Pawson R 2006, Evidence-based policy: a realist perspective Sage, London.
Pawson R 2013, The science of evaluation: a realist manifesto Sage, London.
Pawson R & Tilley N 1997, Realistic Evaluation Sage, London.


Nick Emmel | School of Sociology and Social Policy | University of Leeds |Leeds |LS2 9JT
+44 (0) 113 343 6958 | Twitter @NickEmmel | Blog http://realistmethods.wordpress.com/
Emmel ND (2013) Sampling and choosing cases in qualitative research: a realist approach. London. Sage. http://goo.gl/yYydFd


From: Realist and Meta-narrative Evidence Synthesis: Evolving Standards [mailto:[log in to unmask]] On Behalf Of Lisa Morgan
Sent: 26 November 2014 20:22
To: [log in to unmask]<mailto:[log in to unmask]>
Subject: clarification

Dear RAMESES members

I am a PhD student in Canada, where the realist approach is less well known. I attended the conference in Liverpool and left with a much clearer picture of my research going forward. I met with my academic committee yesterday, of which one member is a U.K. citizen and PhD graduate of Edinburgh University. She is the one that suggested I examine the realist approach for my study. In our meeting yesterday, when I was explaining my coding based on contexts, mechanisms, and outcomes, she challenged me on my approach. She was clear with me that the approach that I am taking is advocated by one school of realist thought, the LEEDS group. She claimed that the CMO configuration is not widely supported and that this methodology lacks a foundation for dealing with complexity. She also informed me that  Pawson et al. is not the only version of a realist methodology and that there are others out there who do not adopt CMOs and have a richer viewpoint on applying their principles. This was news to me!

Are any of you aware of these alternate forms of realist enquiry? I need to defend my choice and can't exactly do that if I have not seen the "menu". I recently read The Science of Evaluation (2013) and feel that Dr. Pawson does a good job addressing complexity. I am afraid that she disagrees. Are there any other resources that anyone can recommend, especially around incorporating complexity into data analysis?

Thanks for your thoughts and assistance.

Lisa Morgan


Lisa Morgan, R.M., B.Sc., B.H.Sc., MA
Midwifery Education Program
Laurentian University
935 Ramsey Lake Road
Sudbury, ON Canada P3E 2C6
Tel: 705-675-1151 ext. 3969
Toll free: 1-800-461-4030
Fax: 705-675-4830
[log in to unmask]<mailto:[log in to unmask]>


Unless otherwise expressly agreed by the sender of this email, this communication may contain privileged or confidential information which is exempt from disclosure under UK law. This email and its attachments may not be used or disclosed except for the purpose for which it has been sent.

If you are not the addressee or have received this email in error, please do not read, print, re-transmit, store or act in reliance on it or any attachments. Instead, please email the sender and then immediately delete it.

General Medical Council

3 Hardman Street, Manchester, M3 3AW

Regents Place, 350 Euston Road, London, NW1 3JN

The Tun, 4 Jacksons Entry, Holyrood Road, Edinburgh, EH8 8AE

4th Floor, Caspian Point 2, Caspian Way, Cardiff Bay CF10 4DQ

9th Floor, Bedford House, 16-22 Bedford Street, Belfast, BT2 7FD

The GMC is a charity registered in England and Wales (1089278) and Scotland (SC037750)