How very reassuring, Gill. I agree totally, BUT this
is an anathema to the traditional [stereotypical] systematic reviewer, who has
REPLACED reflection and theory-building with a technical lexicon of coding,
categorising, checking, double-checking and above all COUNTING. The
link between theory-building systematic review and reflective practice is key,
and one that has not previously been made explicit in many how-to-do-it guides.
Some of this is just the good old principles of qualitative
research, where validity (and scholarship more generally) are defined in terms
of reflexivity, making assumptions explicit, immersion in the data, critical
awareness of the inevitability of error, etc etc rather than in terms of the
'view from nowhere' and some universal, context-free truth.
However, I think we're getting somewhere beyond this.
The role of the systematic [theory-building] reviewer is not merely to reflect
(and thereby build theory) but also to prompt others to reflect. Dare I
introduce a Bakhtinian perspective: there is no story without an audience, and
the way the story is told is shaped and constrained by the relationship between
the narrator and the audience. When we produce a review, especially when
we do so for particular policymakers in particular contexts – and
especially when we are trying to pull out theories of change – we form a
triangle: the reviewer + the review + the audience, all of which exist in a
dynamic tension. This tension is perhaps most pressing when we’re
doing a ‘rapid review’ to a deadline for a painfully tiny
budget! But always the audience is present, even if only in a distant and
assumed way.
It’s also true that the [realist] evaluator exists
in a three-way tension: the evaluator, the policymakers/practitioners and the ‘case’.
The evaluator seeks to pull out the theories-in-use – and in order
to do so, s/he reflects and asks the practitioners to reflect. A
good evaluator will tease out theories of action and facilitate the process by
which the practitioners engage with and add to emerging interpretations. So
realist evaluation is inextricably bound up with reflection/facilitation.
As you say Gill, the realist reviewer
both loses something (direct access to the reflections of the practitioners and
the authors of the original studies) and gains something (a degree of critical
distance).
Can anyone take this thread forward or challenge it.
My day job too calls now!
(does anyone else find it ironic that we’re doing
our thinking outside our day
jobs?)
trish
Trisha Greenhalgh
Professor of Primary Health Care and Director, Healthcare Innovation
and Policy Unit
Centre for Primary Care and Public Health
Blizard Institute
Barts and The
t : 020 7882 7325 (PA) or 7326 (dir line)
f : 020 7882 2552
http://www.icms.qmul.ac.uk/chs/staff/trishagreenhalgh.html
-----Original Message-----
From: Realist and Meta-narrative Evidence Synthesis: Evolving Standards
[mailto:[log in to unmask]] On Behalf Of Gill Westhorp
Sent: 13 July 2011 00:33
To: [log in to unmask]
Subject: Re: How much should we impugn when authors don't make things explicit?
Hi all
Ah yes, I so agree. I consider theory building to be a necessary
part of science and therefore of good science. I consider good theory
building practice to include being as well informed as one can be, within all
the practical constraints. And I see realist (and probably meta-narrative
but I haven't tried one of those) approaches as being purpose designed for the
task... it's one of the reasons why I like it.
To expound: I believe that there is no such thing as 'interpretation
free' analysis - there are after all levels of interpretation and
decision-making reflected in every step of every piece of prior research or
evaluation that we synthesise, as well as every step of our review process
itself. I don't see a problem with taking that one step further and
making 'next step interpretations' (ie theory building) - so long as we are
explicit about the fact that that's what we're doing. In fact, I almost
consider it a duty. Who else is in a better position to do it than those
who have immersed themselves in the research and evidence? So while I
have sympathy for the desire to stay 'close to the literature', I see that as
'being as well informed as one can be' before proposing a theory that accounts
for the findings.
As for practitioners not having theory - here I in fact disagree.
I think they do have theories - NB plural - albeit at the naieve level
sometimes! It's possible in realist evaluation to unpack those. I
tell stories in my training about experiences in doing so - it's often when
practitioners find out that they're operating on different theories than their
colleagues, even if they work together all the time. This provides a
wonderful opportunity to deepen reflective practice, and to assist them to access
relevant MRT and evidence related to same. The difficulty in realist
synthesis is that we don't have the same direct access to the practitioners to
find out what their theories are, and those who did the primary
research/evaluation didn't always either find out or didn't record same.
That's both a strength and a weakness for the synthesis analysts - a
weakness because it reduces the clues about where to look for theory and
because it means we can't check whether interventions were in fact built on
'the same' theory (now there's an issue for reflective thought: what would
constitute adequate evidence that the theories were in fact 'the same'?). But
it can be turned into a strength because our job as synthesists is not
necessarily to test THEIR theories but to build and test theory ACROSS (insert
here whichever version you're doing) ... across manifestations of an
intervention; across interventions using similar theories; across interventions
using different theories but similar mechanisms; or in MNR, across similar
topics but from widely disparate theoretical bases...)
Gotta go. Day job calls.
Cheers
Gill