Hi all
My take on this breaks down into two parts.  The first picks up on the idea of mechanisms as processes.  The second picks up earlier comments on the focus and purpose of the evaluation.

So I'd begin by remembering that all cats are animals but not all animals are cats.  That is: It's true that all mechanisms are processes, but not all processes are mechanisms in the realist sense of the word.  If we go back to earlier descriptions of mechanisms we see that they are a) invisible and b) underlying processes. You can't see them.  

Process evaluations are not usually focused on the invisible level: they are usually, in fact, targeted at what is visible. For example, 'fidelity focused' process evaluations aim to see whether what is described in the 'implementation plan' (or manual) has been/is being done in the way it is described - almost always with 'observable' or otherwise assessable criteria. 

Another way of saying this is that process evaluations focus on the actions taken by the program. Realist evaluations take those actions into account, but they are not the main focus.  In evaluations using Pawson and Tilley's construct of program mechanisms, those actions are important either because they change the context in some way or because they provide a new or more accessible 'resource' to the target group, which then affects the internal response of decision-makers (the 'reasoning' component of mechanisms). You don't have to look at that in a non-realist process evaluation.  

The other 'main' categories of process evaluations focus on improving the efficiency or effectiveness of the processes used (again, a focus on activity, what is done); or on selecting the most efficient/effective intervention to achieve a given set out of outcomes. Again - you can do realist versions of these, but you don't have to.  You don't necessarily have to look at how or for whom or in which context something works in order to say: On average, "x" intervention gets there faster/more often/more cost-efficiently (or whatever the question was) than "y". 

So most unsurprisingly, I end up at the answer to Becky's refined question  "when to choose a realist process evaluation or a non-realist process evaluation" with the usual most basic answer to evaluation design questions:  It depends on the purposes of the evaluation and the questions you are trying to answer.  Do you need to understand underlying causal processes and contexts in which things work in particular ways? If so, do a realist process evaluation. If not, don't.

But I'd add the final comment:  Process evaluations are a broad category of evaluations, defined by their focus which in turn is defined by their purpose.  Realism is not a method or a category of evaluation, it is a philosophy with implications for methods.  So in a sense, comparing process evaluations with realist evaluations is comparing apples and oranges.  But it is realism's nature (ie as a philosophy) that means that one can, as Becky says, do a 'realist any category of evaluation' (I'd say: 'almost any, so long as the underlying assumptions are not incompatible with realism').

Cheers
Gill 

-----Original Message-----
From: Realist and Meta-narrative Evidence Synthesis: Evolving Standards [mailto:[log in to unmask]] On Behalf Of Hardwick, Rebecca
Sent: Friday, 26 January 2018 4:10 AM
To: [log in to unmask]
Subject: Re: In what circumstances would you do a realist evaluation over a process evaluation?

Thanks for the responses so far.  

The question I suppose could be better framed, as several have sent me or pointed out projects which have been 'realist' process evaluations.  This does make sense to me, seeing as you can have a realist [insert method] anything can't you?!  

However, process evaluations do have guidance, from the MRC, developed by people who wanted to set out the rubric for PE, and I wonder to what extent they, and 'we' are talking about the same thing (generative causation and explanation), but just giving it different names?

So in a sense the question 'when would you choose to do a process evaluation over a realist evaluation'  depends on whether you are investigation the process of how some intervention reaches its objectives, or interested in finding out more about what works, how it works for whom, in what circumstances and why....  but are these not really the same thing?!  Or is it more a matter of emphasis - any kind of process evaluation is (necessarily) only focussed on the "how", at the exclusion of all else (unless they offer explanatory traction to the "how")... ?

Is it just a matter of taste?  Do we need to employ the realist CMOCs when we can get to the same conclusions using theory of change, or logic models.... ? 

I guess my question runs a bit deeper then - how do we delineate realist from other kinds of evaluation....?  Or is it like trying to say how is realist interviewing different from interviewing? 

In earlier RAMESES posts, I know that it always seems to come back to the programme theory, looking for then testing out generative mechanisms.... but is this not what a process evaluation does too?  Are we saying then that all process evaluation are realist in nature because they do these things?  

So many more questions and I still feel like I'm missing something obvious....!  

(I am so glad I'm not involved in running the seminar :)) 

Thanks again for comments, 

Best wishes
Becky. 


Rebecca Hardwick
PhD Student

01392 727408
Email:  [log in to unmask]

-----Original Message-----
From: Realist and Meta-narrative Evidence Synthesis: Evolving Standards [mailto:[log in to unmask]] On Behalf Of Mikkel Møldrup-Lakjer
Sent: 25 January 2018 12:10
To: [log in to unmask]
Subject: SV: In what circumstances would you do a realist evaluation over a process evaluation?

Hello everyone,

I believe the answer to the question from Rebecca Hardwick does indeed depend on the meaning of the question itself, because "process evaluation" is not a well-defined evaluation model upon which there is agreement between institutions and evaluators. 

Process evaluation may mean an "implementation evaluation" as Craig Chambers suggests, and may focus either on barriers and helpers for an implementation with high fidelity, on assessing the fidelity itself, or both.

Or we may take the definition from Patton's /Utilization-Focused Evaluation/ (4th edition: 324): "...looking at /how/ a product or outcome is produced rather than looking at the product itself". (I would add: But of course in order to look at the process, you have to take into account the products/outputs/outcomes, because they are the fulfillment of the process).

This means asking process questions rather than variance questions according to the typology of Joseph A. Maxwell. But is a mechanism not also a process? Maxwell explicitly states so: “For critical realists, in contrast, the concept of “mechanism” (in the social sciences, “process” is the usual term) is central to explanation, and these mechanisms and processes are seen as real phenomena, rather than simply as abstract models“ (Maxwell 2012:8-9).

Summing it all up, I would suggest that realist evaluation is a member of the family of process evaluations which comes with specific foundations, principles and perspectives for looking at "processes". So the choice between realist evaluation and process evaluation would be a choice between a process evaluation with a focus on CMO's or a process evaluation with a focus on other questions.

Best regards,

Mikkel Møldrup-Lakjer
Evaluation Consultant
Denmark


-----Oprindelig meddelelse-----
Fra: Realist and Meta-narrative Evidence Synthesis: Evolving Standards [mailto:[log in to unmask]] På vegne af Craig Chambers
Sendt: 24. januar 2018 13:49
Til: [log in to unmask]
Emne: Re: In what circumstances would you do a realist evaluation over a process evaluation?

Hi Rebecca,

my very naive response, having never conducted a process evaluation and being only just over a year's worth of part-time PhD into realist evaluation, would be the answer is partly in the question. Is a process evaluation not an evaluation of whether a process has been implemented correctly/successfully or not? This would not have to consider outcomes as they may be contingent upon some other parameter. The focus instead is along the lines of "have the constituent components of this programme been implemented as the programme developers stipulated/intended".

Thanks for the interesting question and look forward to being shot down in flames for my ineptitude and naivety ;-)

best regards,

Craig Chambers
Part-time PhD, full time physio