On 2 Feb 2017, at 11:22, Gill Westhorp <[log in to unmask]> wrote:

Hi Veronica

Welcome out into the light! J

 

Your steps look logical to me (assuming that ‘the study’ you refer to is the intervention rather than your study of it).  It is certainly possible to do retrospective realist studies of interventions that didn’t have program (or formal) theory underpinning them – I’ve done several myself.

 

I’ve found that it’s best if you can have more qualitative data when you have to work retrospectively – you want to understand the diversity of ‘reasoning’ for people at the different levels that you’re thinking about.  So – what did workers really do differently as a result of the intervention – and why?  What in the resources that were produced / used was new to the participants (assuming there that if they’d already had it, the program wouldn’t cause change) and how did that affect their reasoning?  Etc etc.

 

The other thing to think about is that each of those components has turned up in hundreds of similar interventions before (providing people with information, setting up workplace policies etc) and so there is a LOT of literature – and a lot of theory – that you could be drawing on.  There’s a lot about multi-component interventions too.  The fact that it’s cannabis related might or might not turn out to be relevant to which bits work for whom, how and why; likewise ‘remoteness’ might, or ‘Indigeneity’ might – but there’s literature and theory about all of those as well.  So if you can possibly increase the resource allocation for literature review and synthesis for developing your initial theories – I’d certainly do that.  (There are lots of resources on the Rameses website about how to do that if you’re not already familiar with them).

 

Cheers

Gill 

 

From: Realist and Meta-narrative Evidence Synthesis: Evolving Standards [mailto:[log in to unmask]] On Behalf Of Veronica Graham
Sent: Thursday, 2 February 2017 9:40 PM
To: [log in to unmask]
Subject: Retrospective realist evaluation of pragmatic, multiple component, community based intervention

 

Hi everyone,

This is my first question for this group.

I'd like to check my methodology for a realist evaluation.

 

We would like to evaluate a program retrospectively through a realist lens, but we have no mid-range  theory from the literature.

 

The program was a pragmatic, community-based multi-component intervention targeting high rates of cannabis use. It aimed to stimulate local agencies to deliver five core components - health worker training; school-based intervention; locally developed information resources; workplace policy in remote Indigenous Australian communities. Impact was measured by before and after survey of cannabis use and attitudes towards cannabis + field notes from consultation and implementation for process evaluation.

 

Currently, I plan the following steps:

1. Articulate the program theory of the implemented study.

2. Try to understand the theoretical CMO clusters intended for each component (study design, previous pragmatic multi-component studies in similar contexts). 

• Context will be considered through four levels (individual participant; community;  organisational and wider policy; project team). 
• Mechanism identifies resources created + reasoning of participants.

3. Reflect on what really happened in the implementation context by interviewing three key project staff and plus qualitative data.

4. Propose refined theoretical CMOs with empirical basis from the interviews and data.

5. Propose a refined program theory.

 

Key figures and tables:

1.Figure. Initial program logic

2. Table presenting the refined theoretical CMOs with our assumptions discussed in the text 

     OR 

     Table contrasting initial CMOs and refined CMOs to show what we've learned

3. Refined program logic

 

I am particularly unsure whether step 2 is valid, or whether steps 2-4 would be better considered one step.

Grateful for any feedback, and also any literature on the use of realist evaluation for multiple component designs.

 

Thank you,

Veronica

PhD student, Public Health and Tropic Medicine

James Cook University, Cairns. Australia.