 |
 |
Hi Vanessa You've started one of those really interesting discussions again! Justin's first para below is spot on - trying to understand the 'why' something is being done is essential to understanding the 'what'. It's only after that that one can start to ask the more detailed questions about 'in what circumstances', 'how', 'for whom' and so on. The only thing I could add to that is - in understanding the 'why', be very specific about the outcomes the follow-up is supposed to create/achieve. I have a slightly different take on the critical realism/scientific realism issue. In my understanding: both critical realism and scientific realism understand causation the same way: underlying mechanisms generate outcomes, but only when the circumstances are right. What that means is that taking a critical realist approach doesn't get us out of having to understand contexts, mechanisms and outcomes. What Pawson and Tilley did was operationalise the notion of 'mechanism' for social programs - programs offer different kinds of resources, opportunities or constraints to different decision-makers along the program implementation pathway, up to and including the participants/beneficiaries; those decision-makers 'reason' in response to those resources; their decisions generate different actions which generate different outcomes. I've regularly argued on this list that that is a very useful operationalisation of mechanisms, but not the only one we can use in understanding programs, and that different operationalisations are appropriate for different levels of systems/programs. So at the risk of sounding trite: both critical realism and scientific realism are realist, and use the realist understanding of causation. What's different is the 'critical' compared to 'scientific' slant. It's understanding what the various authors mean by those adjectives that tells us what the differences are. The papers that Sam Porter has provided are relevant to that (although I'd take issue with what some of what he argues, too!). I'd also take issue with the idea that realist evaluation is only relevant to "small scale interventions and programs", as Justin has suggested below, and that critical realism is only relevant to the large scale social perspectives. Realist evaluation is increasingly commonly used in international development, where some programs are anything but small scale. I'm currently involved in a consortium designing a realist evaluation of a very large program that operates across multiple countries, and I'm involved in and/or know of realist evaluations for whole health system interventions in states or countries (as are others on this list). RE is also of increasing interest in environment and climate change domains, where again, issues are large and programs can be too. By the same token, one can apply a critical realist approach to small scale programs. Cheers Gill From: Realist and Meta-narrative Evidence Synthesis: Evolving Standards [mailto:[log in to unmask]] On Behalf Of Jagosh, Justin Sent: Saturday, 29 August 2015 1:20 AM To: [log in to unmask] Subject: Re: Defining mechanisms/PhD newbie questions Vanessa, Is it correct to say that you want to know how sound the policy is? - to review post-stroke patients at 6 weeks, 6 months and yearly after that? Pawson states that every program has a theory, whether it is made explicit or not. So, there is a theory here, likely implicit. And to uncover it, you may start by asking the question " Somebody thought that a series of reviews over the course of time would be a good idea. Why would that be the case? What is the logic here?" Do you think you could, in very simple terms, formulate a programme theory about the policy? What are the desired outcomes, and the pathway to those outcomes? You might be surprized to find lots of mechanisms once you start collecting interview data. Mechanisms (in Pawson and Tilley terms)are the resources offered by the program and the reaction/response to those resources by participants, given their contextual enablers/constrainers. Critical realism and realist evaluation do differ in the treatment of mechanisms, but the easiest way to make the distinction is that Pawson and Tilley's application of realist logic is tied to socially contingent, small scale interventions and programs, whereas Critical Realism is about the larger picture of what underpins the reality we experience - not that you can't use it, but I think P&T makes a lot more sense if you are studying an implemented policy. Critical realism uses philosophical devices and concepts. Realist Evaluation is methodologically oriented. So, you could ask the question, what resources are created by this policy? And how to people respond to those resources? - including all the groups you mentioned (carer, provider, patient etc). Just an observation: your research questions 1 & 2 are constructivist in nature. Nothing objectionable about that, but anytime you ask the question 'what is the perspective of XX group', you are engaging with constructivism. A realist question is about 'What's really going on here?' and you can get the perspective of all these stakeholders for the purpose of building your program theory about 'what's going on, and why?'. Does that make sense? Best of luck as you advance your project, Justin Justin Jagosh, Ph.D Senior Research Fellow Director, Centre for Advancement in Realist Evaluation and Synthesis (CARES) University of Liverpool, UK www.liv.ac.uk/cares <http://www.liv.ac.uk/cares> From: Realist and Meta-narrative Evidence Synthesis: Evolving Standards [mailto:[log in to unmask]] On Behalf Of Vanessa Abrahamson Sent: August 28, 2015 1:24 AM To: [log in to unmask] <mailto:[log in to unmask]> Subject: Re: Defining mechanisms/PhD newbie questions Dear All, I've been following the discussions from PhD students but was reticent to join in and thus demonstrate my lack of understanding. However I am really struggling to differentiate between realism, subtle realism (Hammersley), critical realism (Sayer, Bhaskar) and realist evaluation (Pawson & Tilley). I'm leaning towards CR but still not sure of the exact differences in philosophy this involves or how it influences my method & data analysis. For example, in one email (2nd Jul) Gill made the point that interviews will only be realist if they are building, testing &/or refining realist theory and that needs to cover context, mechanism and outcome. I don't think this suits my research question but the basic realist philosophy appeals. I'm using a case study design, 2 areas, looking at a particular policy recommendation that states people who have had a stroke should be reviewed at 6 weeks, 6 months and yearly thereafter. Most people are still under the acute sector for the 6 month review, provision of the 6 month review is patchy, and few fund a yearly review. I'm mainly using interviews with some observation (where ethics allows) and document analysis. My research question: 'How does the review process support adults with long term need post-stroke?' & objectives: 1. What is the purpose of the review process from the perspective of patient, carer, provider and commissioner? 2. What are the intended and/or unintended outcomes of the review process from the perspective of patient, carer, provider and commissioner? 3. By what mechanisms does the review process achieve the intended outcomes? What are the enablers and barriers? By mechanisms, I was using it in the lay sense..but still not sure how critical realism would define versus Pawson & Tilley? With different perspectives (patient, carer, provider, commissioner) and an over-reliance on interviews I can't imagine that I can achieve clarity with the mechanisms and outcomes at this stage - it is more exploratory (but I'm not keen on using interpretivism either). So can I frame as a critical realist approach without tying myself in knots with CMO configurations that I won't have the time/resources to test and refine? And how do I ensure/demonstrate that this informs my method & data analysis. I'd be very grateful for your suggestions, Vanessa. V. Abrahamson ESRC PhD Candidate in Social Policy Centre for Health Services Studies University of Kent George Allen Wing Canterbury Kent CT2 7NF From: Realist and Meta-narrative Evidence Synthesis: Evolving Standards [mailto:[log in to unmask]] On Behalf Of Sonia Dalkin Sent: 11 August 2015 10:48 To: [log in to unmask] <mailto:[log in to unmask]> Subject: Re: Defining mechanisms as processes? What Realist findings look like? PhD newbie questions Hi Mandy and others, Thank you for posting your questions Mandy and for all the responses so far, they are all so interesting and can only help us to question our own understanding of realist methods and learn more. I would echo Trisha and Justin's comments - picking an examiner with realist experience is essential if you're using a realist framework and often methodological innovation is best left post PhD. A lot of PhD candidates can trip themselves up when trying to use hybrids as there can be clashes of philosophies and associated methodologies. This isn't to say that this can't be done though! Q1 - I guess the question is why would you want to reinvent the wheel - Pawson & Tilley provide a definition of mechanism and if you're using their methodology it's probably best to stick to their definition, unless you have a rationale for changing it. In your PhD it's always important to be able to justify any deviations from the traditional methodology. Why did you do it - how has it enhanced your research project? What would have been lacking had you not have done this? Q2 - I think you're using the term 'mechanism statements' as the equivalent of Context-Mechanism-Outcome configurations (CMOc) in Pawson & Tilley. Ana Manzano and Ray Pawson pick up on this further in their diagnostic workshop paper (attached). They state that C, M and O should not be listed (a common mistake) but should be in configurations - in this context, this mechanism is activated which leads to this outcome. A recent paper I wrote (attached) with colleagues from University of Leeds (Dr. Joanne Greenhalgh) and Northumbria University (Dr. Monique Lhussier, Dr. Anna Jones, Dr. Bill Cunningham) highlighted the need to outline resource and reasoning (another common mistake in RE) within CMOc and provided a rejig of the C+M=O formula to aid operationalisation. The paper still conforms to Pawson & Tilley's original ideas but just provides an alternative way of configuring the formula. In this paper we provide examples of 'mechanism statements' or what are usually known as 'CMOc' with resource and reasoning. You may find this helpful to see whether your definition of mechanism statement is the same as CMOc. I hope I've interpreted your questions correctly Mandy and hope this in some way helps! Best of luck, Sonia __________________________________________________________________________ Dr Sonia Dalkin Lecturer in Public Health, Department of Public Health, Faculty of Health & Life Sciences Email: <mailto:[log in to unmask]> [log in to unmask] Twitter: @SoniaDalkin Room H006, Coach Lane Campus East, Northumbria University, Newcastle upon Tyne, NE7 7XA, United Kingdom From: Realist and Meta-narrative Evidence Synthesis: Evolving Standards [mailto:[log in to unmask]] On Behalf Of Trish Greenhalgh Sent: 11 August 2015 07:33 To: [log in to unmask] <mailto:[log in to unmask]> Subject: Re: Defining mechanisms as processes? What Realist findings look like? PhD newbie questions I don't like the way 'motivation' is creeping in here. Can we say 'agency-dependent'? Motivation seems wrong paradigm. Trisha Greenhalgh Professor of Primary Care Health Sciences T: +44 (0)1865 289363 E: [log in to unmask] <mailto:[log in to unmask]> <http://www.phc.ox.ac.uk> www.phc.ox.ac.uk | <http://www.twitter.com/oxprimarycare> @OxPrimaryCare | <http://www.twitter.com/oxprimarycare> @trishgreenhalgh Nuffield Department of Primary Care Health Sciences, University of Oxford New Radcliffe House, Radcliffe Observatory Quarter, Woodstock Road, Oxford, OX2 6GG From: <Jagosh>, Justin <[log in to unmask] <mailto:[log in to unmask]> > Reply-To: "Realist and Meta-narrative Evidence Synthesis: Evolving Standards" <[log in to unmask] <mailto:[log in to unmask]> >, "Jagosh, Justin" <[log in to unmask] <mailto:[log in to unmask]> > Date: Tuesday, 11 August 2015 00:34 To: "[log in to unmask] <mailto:[log in to unmask]> " <[log in to unmask] <mailto:[log in to unmask]> > Subject: Re: Defining mechanisms as processes? What Realist findings look like? PhD newbie questions Dear Mandy, Your rigorous questioning on the concept of mechanism is excellent! I agree with Trish that it is vital to provide a definition of terms if you diverge from Pawson and Tilley. But Realist Evaluation is gaining recognition and credibility so if your reason to change the definition is because people in your field are not familiar with it, then as Trish says, you need to instead find the right supervisors/examiners and stick with it. Innovation is better done after you graduate :). If, on the other hand, you want to diverge from the definition because you don't feel it works for what you are studying, that's a different story and it's totally justified - but that would also need to be clearly explained. Is the definition of mechanism, as defined by P&T as reactions to resources offered - not adequate? Are you examining youth employment interventions? If yes, do they create resources via implementation of the program theory? How do people respond to these resources, given who they are, their backgrounds, and their contextual constrainers and enablers? If your research question is : what makes people 'employable?', and you are not studying a specific intervention (or class of interventions), then you do have to innovate P&T's work, because they have built realist methodology pinned around the evaluation/synthesis of interventions, services, policies etc. The question: "what creates youth employability?" is a sociological question that, in and of itself needs to be unpacked. So your definition of mechanism as: 'a generalisable, context-dependent and motivation-dependent interaction process, through which outcomes of interest are partly generated (or strongly influenced?)' to provide conceptual scaffolding for non-realist readers? is a bit muddy to my mind. Are the concepts 'generalizable' and 'context-dependent' not antithetical when put together in this statement? Also 'motivation-dependent interaction process' is also a bit muddy. Is there such a thing as a motivation-independent interaction process when it comes to people interacting with each other and with policies? Also do people interact with policies? Or just react to them? We are all reacting to our environment all the time, and even when we think we're not - (because it's also happening subconsciously). Now if you want to say that mechanism has to do with how people react/respond within an interaction process - in a context - perhaps that might be clearer, I'm not sure. You might have a look the figure in our paper, linking CMO configurations in a ripple effect - (attached). In that paper, we too looked at a process over time, and suggest that you can link CMOs to explain this kind of emergent causal chain - one in which reactions lead to outcomes, which impact on the context over time or across intervention phases. It's still rather crude, but it has helped us create realist middle-range theory in our subject area that is more closely aligned with "what's really going" .(which can be considered one of the possible overarching goals of RE).. Please feel free to clarify if I haven't grasped adequately your question! And hopefully others will stir the pot too. Justin For example, my research question could ask: What can be clarified about key processes that shape individual 'employability' and employment/unemployment outcomes; particularly for youth populations and with a focus on vocational and 'at risk' youth subgroup outcome patterns? How is the variation in youth outcomes, as subgroup patterns, explained by certain types of intervention and/or context conditions affecting those processes? By the way - improvement in individual 'employability status' may be defined and studied as a harder to measure, long run outcome pattern of interest - for which improvements in an individual's employment outcomes over time are treated as one of a few key employability status/outcome indicators. A Rough Theory section proposes that employment outcomes, and improvements in individual states of employability (based on theorised indicators), could simultaneously be evaluated as intertwined outcome objectives. Question 2) When you describe mechanisms as research findings, don't they need to comprise mechanism statements? I.e. they must explain a mechanism's relevance to a set of outcomes - then also describe something about why, for which subgroup/s, or how at least one other context condition is relevant to understanding an outcome pattern that emerges from that mechanism (interaction process)? So a mechanism could be summarised as say a 1-3 word label - whereas a mechanism finding must comprise a full sentence statement distinguishing something about context conditions affecting the outcome patterns emerging from that mechanism, and/or affecting an intervention's workability in turn. Said another way - isn't it true that a complete mechanism statement/finding must distinguish and relate at least one Context variable to how a Mechanism works, and to an Outcome pattern in turn? Whereas a mechanism on its own can be summarised as say a 1-3 word label (e.g. Recruitment and Selection Mechanisms as being relevant to employment). I realise the above may be nit-picking but it helps to visualise what comprehensive Realist type 'findings' look like before trying to produce them. Many thanks in advance for anyone who wants to add their two cents to any of the above screeds! Regards Mandy McGirr PhD researcher School of Government, Victoria University of Wellington New Zealand