Hi Patricia (and all)
Now this is really important. Real causal inferences in science are made in the manner you suggest. Theories are built, contested and refined over time using a plethora of different empirical tests – none of them decisive in isolation. This is well described by the philosophers of science – Popper, Lakatos, Campbell (the real one) etc. Hill’s criteria capture the balance of many considerations that contribute to strong causal explanations and remain influential in the public health community. None of these sources translate simply into programme evaluation.
The great irony is that these ‘conjectures are refutations’ models describe perfectly the many different empirical tests that are applied in the lengthy process of developing clinical interventions. These start in basic science, pre-clinical work hypothesising the underlying disease pathology and potential mechanisms of action that might target the particular condition. Then there is a phase of therapeutic discovery in which compounds and techniques are tried, refined in an attempt to embody the conjectured mechanisms and then laboratory tested the see if the initial explanation holds promise. Then, still in pre-clinical phases, there are tests, for instance, about the absorption, distribution, metabolism and excretion of a drug. Only then we get to effectiveness with patients and the three phases of clinical trials, starting with proof of concept, dose-finding and safety studies, moving eventually to RCTs and regulatory proof. There is even a further stage about long-term follow-up and the detection of ‘rare events’. Failure is commonplace at any stage – in which case the hypotheses are refined and the hypotheses-testing cycle is resumed.
Here’s the problem. A) Cochranites have managed to convince themselves (and much of the medical establishment) that only the penultimate stage counts in assessing causality and effectiveness. B) Even those keenly attuned to the full cycle explanation often don’t like to use the language of ‘theory’ testing (they often prefer rather technical or mechanical accounts of each individual stage).
So you are right about the desperate need for good examples of non-counterfactual causal explanations. Finding them in medicine would be the ultimate coup de grace. Rather late in the day I’m trying to become an amateur medic in trying to piece together the full story as above. Any good sources graciously welcomed!
Thanks by the way for the links to Julian and Jane’s excellent work.
From: Realist and Meta-narrative Evidence Synthesis: Evolving Standards [[log in to unmask]] On Behalf Of Patricia Rogers [[log in to unmask]]
Sent: Tuesday, November 11, 2014 9:36 PM
To: [log in to unmask]
Subject: Example of causal inference in evaluation without a counterfactual
Dear RAMESES colleagues,
After such great suggestions of examples of realist evaluations, I'd like to make a related request.
Can you suggest any good examples of an impact evaluation that uses non-counterfactual causal inference - that is, doing lots of small tests that the data fit the theory of a causal relationship, and ruling out alternative explanations?
While there's good material that discusses these strategies (eg Bradford Hill's classic 1965 paper,<http://www.edwardtufte.com/tufte/hill> on Edward Tufte's site, Julian King's new e-book<http://www.julianking.co.nz/wp-content/uploads/2014/08/140826-BHC-web.pdf> on the Bradford Hill criteria. and Jane Davidson's webinar<http://betterevaluation.org/events/coffee_break_webinars_2013#webinarPart5> on causal inference) I struggle to find good examples that can be used in a workshop to clearly demonstrate the logic of non-counterfactual causal inference, can be readily understood and are clearly relevant.
The old epidemiological examples of John Snow's investigation of cholera in London and the link between lung cancer and smoking explain the logic but are too often dismissed as being not relevant because they are "the cause of an effect, not the effect of a cause" - as in an impact evaluation of a known program. (I think this is a specious argument but that's a hard position to change).
I'd really appreciate suggestions of good examples I can add to the BetterEvaluation site <http://betterevaluation.org/plan/understandcauses> and share in workshops and presentations.