Hi all

Pawson discussed the tests and trials of new drugs and the timescale/complexity involved. In a clinic environment the drug can be tested to find its effectiveness. However, when the drug is approved and prescribed to an individual, it is then in a social system. The drug has potential to cause a reaction depending on the reasoning of the individual to take it as prescribed. Then the question is why some individuals take the drug as prescribed while others may forget or choose not to take the drug? An RCT would not answer this question.

I am not against RCT, as each research project is unique and it is more important to find the right study design that will answer your research question/aims and refine the program theory.

I am also a realist novice and have 2 years left on a PhD. My PhD is to develop an intervention that challenges the norms of binge drinking for 16 to 18 year olds. What is important for my research is to understand who my intervention works for, why, when and how. It is fundamental to focus on what has happened as a consequence of my intervention both intended and unintended, enabling a deeper understanding of the mechanism of change. To compare these findings to a control group would add more complexity, as the control group is embedded in a social system and there may be other influences which may impact the findings.
 
Kind regards
Michelle Hyatt
[log in to unmask]

-----Original Message-----
From: Realist and Meta-narrative Evidence Synthesis: Evolving Standards [mailto:[log in to unmask]] On Behalf Of Rasmus Ravn
Sent: 11 October 2016 09:55
To: [log in to unmask]
Subject: SV: 'Realist RCTs'

Dear all.

This is my first ever post on the forum, though I have been receiving the e-mails for a year and a half.

I attended the CARES conference and the presentations by Ray Pawson and Gill Westhorp made me reflect a lot.

I am inclined to take part in this debate, though I am only a realist novice.

In advance, sorry for any spelling or grammatical errors. English is not my first language. 

My interest in this debate stems from the fact that in my own research I am combining Realist Evaluation with an impact study using a quasi-experimental approach establishing the control group using statistical matching. Though not based on random allocation the impact study will provide an average effect size and also effect sizes for subgroups. The general logic is the the same as in the RTC. Each of the two evaluation "models" will be applied in their own right, and insight from the two inform the overall discussion and conclusion. In practice, what I do in the Realist Evaluation using before-after measurement and within/intra case/group comparison (the comparative method). The comparison is carried out selecting groups that, according to the programme theory, are hypothesized to have low and a high chance of achieving the desired outcome respectively. The evaluation is carried out using the following data:

Qualitative interviews with programme staff and programme participants (for development, refinement and testing of the programme theory - the main purpose being to unearth the generative mechanims) Three repeated surveys with programme participants and programme staff respectively (The purpose being to investigate the implementation process, but more importantly to elicit and test the programme mechanisms in different contexts. Several questions have been included in the questionnaire to "tap" each of the hypothesised mechanisms). 
Register data on participant level (information on each participants "status" and each member of the control group and other background information on a weekly basis).

The register data will be used for the impact study but by merging the surveydata and the qualitative interviews with the register data using an ID-number, my reasoning is, that it will possible to investigate 'what worked for whom under which circumstances'. 

I hope and believe that my approach is truthful to both Realist Evaluation and to impact studies (experimental and quasi-experimental).
As part of my PhD thesis I therefore need to produce an argument for combining the two evaluation models, which is why my attention was drawn to the above mentioned papers. 

To reply to Mark's question, I do believe that a Realist RTC is feasible - at least in theory. Though I am not entirely convinced that one should go about it like Bonnel et al. 

To me, what makes an evaluation genuinely realist is the notion of generative causality. Generative causality has to be taken seriously and this requires a deep qualitative understanding of the programme at hand. You need to understand how the participants changed their reasoning and behaviour because of the resources the were exposed to. Of course, this cannot be done quantitatively. This however, does not rule out the use of an RTC for the sole purpose of establishing outcome. 

In my view, the RCT should not be regarded as more than a method for establishing an outcome. Though a method many positivist and post-positivist have used. A Realist might object and say that an RTC is not at all a feasible method, because of the open systems dilemma and because of human volition and agency. In other words, the outcome of an intervention will always be affected by the context in which it unfolds. I am inclined to believe that this is not a valid argument because the contextual factors would affect both the control and the participants groups equally due to the randomisation. If we are of the opinion, that RTC can fruitfully establish outcomes, the natural consequence would be that Realist Evaluation could inform and be combined with an RTC. We should not exclude any method based on it prior use by positivist. As Pawson illustrated in his presentation at the conference, Evidence-Based Medicine operates in congruence with broad realist principles - though we are are often only presented with the "RTC stage" of the proces. In Social Science and Public Policy we are also often only presented with the RTC stage and with the stages that goes into designing the intervention in question. The RTC is regarded as the end result and used to make judgements of whether interventions are effective or not. In my opinion, this is not a problem inherent to the experimental or quasi-experimental approach. It should rather be viewed as a problem regarding the use actual use of the approach.

I view myself as a devoted realist. Through my reading of Realist Evaluation I have come to identify a problem with what I believe is a problem with practice - but not the approach! When reading Realist Evaluations I am often not convinced - or sufficiently persuaded to be believe - that it is actually the hypothesised mechanisms that have brought about the change. This is due to a lack of proper testing of the hypothesised mechanisms. 
A Realist RTC could be one way of sufficiently testing the mechanisms. Another way could be to take a more quantitative turn in general. Many statistical methods have ways of handling complexity, feedback and interactions with contexts. 

If we believe that mechanisms cannot be measures in any way (because they are invisible) - then they cannot be tested. If they cannot in any way be tested then we are basically left with CO instead of CMO. Which is broadly the same as the RTC recieved so much criticism for. (Though I won't deny that could could make a persuasive argument - that it was the mechanisms that brought about the change.) Therefore I would argue for a minor quantitative turn in realist evaluation - or at least careful qualitative comparative analysis (within or between cases) based on "informed" sampling on the basis of ones programme theory. 

I know that my above standing post might be considered controversial within the realist camp. I absolutely do not mean to offend anyone - and I should once again make it clear, that I consider myself a realist. But I do believe that disputatious and constructive debate and critique is the way forward in all spheres of research.

These are just my thoughts.  

Best regards,

Rasmus Ravn, Realist Novice, PhD Student, Aalborg University, Department of Political Science, Denmark.

________________________________________
Fra: Realist and Meta-narrative Evidence Synthesis: Evolving Standards [[log in to unmask]] på vegne af Mark Pearson [[log in to unmask]]
Sendt: 10. oktober 2016 14:30
Til: [log in to unmask]
Emne: 'Realist RCTs'

Dear all

Last week at the CARES conference in London, Ray Pawson explained how (in clinical research) explanation of treatment and delivery mechanisms is developed in-depth before going to a trial (RCT). Ray also considered that a similar 'programmatic' model of research could provide greater explanatory insight about social programmes. So is there such a thing as a 'Realist RCT', or is this like saying 'English French'?

There has recently been debate on this in the journal 'Trials', for which a reply to our reply was published last week. Chronologically, these papers are:

Jamal et al. 2015 The three stages of building and testing mid-level theories in a realist RCT: a theoretical and methodological case-example https://trialsjournal.biomedcentral.com/articles/10.1186/s13063-015-0980-y

van Belle et al. 2016 Can "realist" randomised controlled trials be genuinely realist? https://trialsjournal.biomedcentral.com/articles/10.1186/s13063-016-1407-0

Bonell et al. 2016 Realist trials and the testing of context-mechanism-outcome configurations: a response to van Belle et al. DOI 10.1186/s13063-016-1613-9

What do people think?

Mark

Mark Pearson PhD
Senior Research Fellow in Implementation Science NIHR CLAHRC South West Peninsula (PenCLAHRC) University of Exeter Medical School
E: [log in to unmask]