PLEASE NOTE:

When you click 'Reply' to any message it will be sent to all RAMESES List members.

If you only want to reply to the sender please remove [log in to unmask] from the 'To:' section of your email.



Dear Elina, 



You present a very real world issue: that the things which we seek to evaluate just wont stay still and let us evaluate them: they're always changing, evolving, and so on.  



Thankfully there are at least two realist responses to this: realist philosophy and abstraction.



You ask if its possible to even "do" programme theory for this programme.  I'd suggest it is, and further, that you already have: but that you'd need to explain that in realist terms, our understanding of 'the mind independent reality' aka: what is real, is open to constant refinement and change - so whilst there might be a mind-independent reality, our knowledge of it, whether from experience, evaluation, evidence or what my mum says means that knowledge is constantly shifting of what 'is'.  The implication here is that no programme theory is ever able to fully be the 'final' resting point of a programme - all we have, we hope, are better and better theories about what 'it' is, how it works, for whom and why...  



Therefore, I'd suggest going back through what you've done and looking at the various ways that it has been conceptualised, both in the literature, through programme architects, and then modestly, but firmly say "This is our initial rough programme theory about how this programme works, why, for whom" and so on.  Then you have a starting point. 



Next step then is to think about abstraction: so abstraction is the step by which we move away from the 'one' programme to the many - so looking at the programme theory(ies) you've now identified - what are the mechanisms through which it works?  And then ask yourself - where else do we find these mechanisms/outcomes?!  In the example you mentioned, introducing new ways of working is old hat, so looking at evaluations which have evaluated other attempts at bringing in new models of working (in SW, but also nursing, medicine, vets, development research - any field in fact) to do that might be a place to start and work from.  You say its an adaptation of the RSW model - so how about looking at that too and working out in what ways its activating the same or different mechanisms... and then where its activating the same kinds of mechanisms, where has that been evaluated?  What can you learn about its effectiveness from other programmes which are activating the same mechanisms? In addition, part of how it works is to get behaviour to change - how does it attempt to do this: for example: is it through incentivising change, through preaching that they 'should' change, or through punishing lack of change (i.e. loss of status): these approaches to behaviour change of professionals have all been tried on different kinds of topics: what might they teach us then about this programme? 



(And if you want some reading to help illuminate this further, see chapter five of "The Science of Evaluation", where Ray sets out 'the realist response to complexity'.  If you haven't got the book already, then buy it and read/re-read the chapter.  If only for the lines: "We now career to the core of the chapter, confronting the conundrum of complexity by charting the creed of the comprehensive but corrigible realism.  To strike the delicate balance between pursuing an ever extending set of questions and the need to achieve closure in any particular inquiry requires a new set of organising principles... [he then goes on to explain TARMATO which he suggests can be the organising principles of evaluation science (I've just taken us through T for Theory, and A for abstraction)]) 



Good luck, 

Becky  



Dr Rebecca Hardwick

Post-Doctoral Research Associate

Realist Research, Evaluation and Learning Initiative (RREALI)

Charles Darwin University.



-----Original Message-----

From: Realist and Meta-narrative Evidence Synthesis: Evolving Standards <[log in to unmask]> On Behalf Of Aaltio, Elina

Sent: 18 April 2019 20:54

To: [log in to unmask]

Subject: Is it possible to create a realist programme theory for an unfinished co-created programme?





PLEASE NOTE:

When you click 'Reply' to any message it will be sent to all RAMESES List members.

If you only want to reply to the sender please remove [log in to unmask] from the 'To:' section of your email.



Dear experts in realist evaluation,



I am struggling with a PhD project, where I am trying to evaluate a social work practice model using RE. The programme under study is a practice model for statutory child protection, so called Systemic Model. It is a loose adaptation of the RSW model, that was developed in London 10 years ago. Both models aim at improving the well-being of service users and practitioners by changing the social work practice (approach, tool, methods, values, attitudes) and it's organizational context (values, resources, management, supervision, organizational culture). 



The Systemic Model has been piloted nation-wide in Finland for two years now and I have conducted a national evaluation of this pilot. One result was that practitioners saw the potential of the model, but don’t really know how to put the ideas of the model in everyday practice. There's no manual or uniform training material.



At this point, national stakeholders agreed that creating a programme theory for the Systemic Model might be a good idea both for the future evaluation and dissemination purposes. However, after several programme theory workshops with key stakeholders (people in charge of national training and dissemination) I have now come to the conclusion, that this model is in constant change. No one seems to be (nor wants to be) in charge of making final (or semi-final) decisions about the content. New ideas come along all the time, and even the key components are having new features that have implications for other key components.



My question is this: is it even possible to create a programme theory for something like this? How to respond to this obscurity - the obscurity of the programme and the obscurity of the development process?



I would be very grateful for any thoughts or advice!



Elina Aaltio

Doctoral researcher

Department of Social Sciences and Philosophy University of Jyväskylä, Finland







To UNSUBSCRIBE please see: https://www.jiscmail.ac.uk/help/subscribers/faq.html#join



To UNSUBSCRIBE please see: https://www.jiscmail.ac.uk/help/subscribers/faq.html#join