Hi y'all

Some of our debate around 'wringing out' theories brought this to mind:

Friedman V (2001) Designed Blindness: An action science perspective on program theory evaluation. American Journal of Evaluation 22(2) 161-181.

Enjoy - if you have not already come across it.

RAY

-----Original Message-----
From: Realist and Meta-narrative Evidence Synthesis: Evolving Standards [mailto:[log in to unmask]] On Behalf Of Mark Pearson
Sent: 25 July 2011 15:25
To: [log in to unmask]
Subject: Re: Theories - local or formal

Yes, there's certainly a difference between argument (exchange of blows) and argumentation (critical marshalling of diverse sources of evidence)! Hopefully, most research teams avoid the former and at least tend towards the latter... although I'll leave it to others to address the challenge of how we can have more of the good fights...

Mark
________________________________________
From: Trisha Greenhalgh [[log in to unmask]]
Sent: 22 July 2011 15:18
To: 'Realist and Meta-narrative Evidence Synthesis: Evolving              Standards'; Mark Pearson
Subject: RE: Theories - local or formal

Great posting.  Mostly agree but the following is surely an
oversimplification:
"2) Researchers not agreeing on the PTs that were 'wrung out' - Good!
Scientific inquiry is dead once researchers desist from employing their own
independent, critical faculties and begin practicing in a 'normative'
fashion (I forget whether this was Kuhn's or Popper's insight...). I think
the difficulties you had in reaching consensus on PTs is a sign of a healthy
and critical research team, if not exactly what a journal such as BMJ will
be looking for."

There's good fights and bad fights.  And one challenge is to work out how to
have fewer bad fights and more good fights!



Trisha Greenhalgh
Professor of Primary Health Care and Director, Healthcare Innovation and
Policy Unit
Centre for Primary Care and Public Health
Blizard Institute
Barts and The London School of Medicine and Dentistry
Yvonne Carter Building
Turner Street
London E1 2AT
t : 020 7882 7325 (PA) or 7326 (dir line)
f : 020 7882 2552
e: [log in to unmask]


http://www.icms.qmul.ac.uk/chs/staff/trishagreenhalgh.html


-----Original Message-----
From: Realist and Meta-narrative Evidence Synthesis: Evolving Standards
[mailto:[log in to unmask]] On Behalf Of Mark Pearson
Sent: 22 July 2011 14:06
To: [log in to unmask]
Subject: Re: Theories - local or formal

Having conducted health services primary research within a realist framework
(albeit not a full-blown RE) and currently being in the throes of a realist
review, I'm not sure I share Trish's misgivings about the difficulties of
'wringing out' programme theories from data (whether primary or secondary).
From where I sit, there are two important aspects (in primary research) for
eliciting PTs (both of which may well have been very difficult to attain in
a large RE):

1) Sufficient time for trust and rapport to be developed between researcher
and researched - people may well be insecure and nervous about their
positions, and it takes time for the confidence to be developed in which a
more open conversation about PTs can take place. This may well require
'ethnographic immersion in the field' and participating in the day-to-day
life of a field of practice (plainly, this is much easier to achieve in a
small-scale project with just one researcher...)
2) The willingness (or time!) to track back through policy history (using
primary documents and others' hard work in analysing and critiquing the
policy that arose from these) to better understand 'how we got to where
people think we are' in relation to the (emerging) findings of the primary
research

Point 1) is simply well-worn good research practice - although in my view
often forgotten when it comes to researching professionals, who we tend to
assume are articulate, thrusting and confident (rather than the insecure and
nervous individuals that they actually are). Point 2) acknowledges Trish's
point more about 'wringing out' PTs from data - inevitably, these PTs will
be incomplete and localised (not to mention more than a bit hazy), but
that's because they're only the starting point for investigating other
sources that enable a better understanding of the long chains of knowledge
(and power?) that underlie these brief utterances (a Foucauldian
'archaeology of knowledge'?). We may also have difficulty recognising PTs
because, to a greater or lesser extent, we inhabit a shared professional
world in which particular perspectives have become embedded and taken for
granted (doing a RE therefore requires more critical self-reflection than is
usual in evaluation?).

On Trish's other points:
1) Participants not agreeing on the PTs that were 'wrung out' - I don't
necessarily see this as a problem, as 'respondent-validation' is not
appropriate in this context. An outsider's perspective may produce insights
about day-to-day practice and assumptions that we find uncomfortable and
disconcerting - this doesn't necessarily make these insights 'wrong'!
2) Researchers not agreeing on the PTs that were 'wrung out' - Good!
Scientific inquiry is dead once researchers desist from employing their own
independent, critical faculties and begin practicing in a 'normative'
fashion (I forget whether this was Kuhn's or Popper's insight...). I think
the difficulties you had in reaching consensus on PTs is a sign of a healthy
and critical research team, if not exactly what a journal such as BMJ will
be looking for.


I would find it very interesting to hear more about:
a) why Trish/the team found it very difficult to 'get at' the PTs
b) others' experiences of 'getting at' PTs in the course of a realist review
(in ours, these potential PTs are multiplying daily - perhaps many of these
'PTs' are much more basic (low?) rather than middle range theories, but our
issue is more how to reign all these in/build them up coherently rather than
having to 'wring them out of the data')


Mark
(btw, definitely thinking on SDO-funded realist review time)

Mark Pearson PhD
Research Fellow
Peninsula Technology Assessment Group (PenTAG)
Peninsula College of Medicine & Dentistry (University of Exeter)
E: [log in to unmask]
T: 0044 (0) 1392 726079
http://sites.pcmd.ac.uk/pentag/staff.php?selstaff=mpearson

-----Original Message-----
From: Realist and Meta-narrative Evidence Synthesis: Evolving Standards
[mailto:[log in to unmask]] On Behalf Of Trisha Greenhalgh
Sent: 18 July 2011 06:43
To: [log in to unmask]
Subject: Re: Theories - local or formal

All I know is that [a] when I did a big realist evaluation (with Ray on the
team) of a major change programme, the theories of change had to be wrung
out of the data, they weren't evident at all, and neither did participants
(or researchers) agree on them.  I think in retrospect this was because the
project was extremely large and complex, but as I recall even Ray was
surprised.  [b] it is a very standard criticism of realist evaluation (by
non-realists) that these theories of change are harder to identify than
realists claim.  I've heard Mary Dixon-Woods waxing lyrical on this topic.

I find realist review a LOT easier than realist evaluation, as I think Ray
and others are right that once you've got some juicy written material to
study (i.e. pretty much anything other than a 2000-word RCT with a
two-paragraph methods section), you can divine the theories of change.  I
just think it's sometimes very hard to do it in a forward direction in a
primary study.


Trisha Greenhalgh
Professor of Primary Health Care and Director, Healthcare Innovation and
Policy Unit
Centre for Primary Care and Public Health
Blizard Institute
Barts and The London School of Medicine and Dentistry
Yvonne Carter Building
Turner Street
London E1 2AT
t : 020 7882 7325 (PA) or 7326 (dir line)
f : 020 7882 2552
e: [log in to unmask]


http://www.icms.qmul.ac.uk/chs/staff/trishagreenhalgh.html


-----Original Message-----
From: Realist and Meta-narrative Evidence Synthesis: Evolving Standards
[mailto:[log in to unmask]] On Behalf Of Gill Westhorp
Sent: 18 July 2011 02:36
To: [log in to unmask]
Subject: Re: Theories - local or formal

I'm intrigued, Trish, by your uncertainty that the theories are always out
there.  (Well, 'always' might be a bit strong, but you know...)  I thought
the variation in constructs and paradigms that underpins Meta-Narrative
Review would imply multiplicity of theories, rather than absence of same??

I'm also intrigued, Bruno, by your phrase (a couple of emails back) about
"theories and/or best practice that already exist".  Firstly - while one can
say that 'best practice' documents represent a kind of theory, I suspect
that theory in the sense we've been discussing lies at a somewhat different
level of abstraction than does 'best practice'.  More importantly, I'm not
100% convinced that the logic of 'best practice' (crudely summarised: do' x'
to get 'y' outcome) is congruent with realist philosophy...   Would be
interested to hear more of your thinking around that.

The other issue that I think arises in relation to theory or choice of
theories relates to the structure of the question one is trying to answer.
So a question that asks one to identify the mechanisms (plural) that
contribute or may contribute to an outcome or set of outcomes might require
one to draw on multiple theories across quite a wide range of disciplines.
A question that asks one to identify in what circumstances particular middle
range mechanisms are effective  (or more accurately, constructs within which
mechanisms lie - e.g. 'community accountability'; 'voice') might suggest a
narrower range of theories.  Both these questions draw on real RS examples,
but neither relies on a particular program or even 'family of programs'.
This suggests to me that the nature of the task for the reviewer, in drawing
the relationship between middle range theory and program theory might be
different, depending on what the task is...

Cheers
Gill



-----Original Message-----
From: Realist and Meta-narrative Evidence Synthesis: Evolving Standards
[mailto:[log in to unmask]] On Behalf Of BMarchal
Sent: Monday, 18 July 2011 3:26 AM
To: [log in to unmask]
Subject: Re: Theories - local or formal

Well, i would be tempted to follow Gill's 'definition' of mechanisms, in
which case there are a lot of theories in psychology, sociology and
economics (including their organisational sub-fields), but also in cognitive
sciences that can provide insights in how people act and why and then change
their situation (or not). In as far as human behaviour will never be fully
'explainable', there will always be theories lacking, but there is already a
great mass of knowledge out there...

-bruno


> From: Trisha Greenhalgh <[log in to unmask]>
> Reply-To: "Realist and Meta-narrative Evidence Synthesis: Evolving
> Standards" <[log in to unmask]>, Trisha Greenhalgh
> <[log in to unmask]>
> Date: Sun, 17 Jul 2011 18:29:53 +0100
> To: <[log in to unmask]>
> Subject: Re: Theories - local or formal
>
> I'm not as sure as Ray that the theories are always there....
>
>
>
> Trisha Greenhalgh
> Professor of Primary Health Care and Director, Healthcare Innovation
> and Policy Unit Centre for Primary Care and Public Health Blizard
> Institute Barts and The London School of Medicine and Dentistry Yvonne
> Carter Building Turner Street London E1 2AT t : 020 7882 7325 (PA) or
> 7326 (dir line) f : 020 7882 2552
> e: [log in to unmask]
>
>
> http://www.icms.qmul.ac.uk/chs/staff/trishagreenhalgh.html
>
>
> -----Original Message-----
> From: Realist and Meta-narrative Evidence Synthesis: Evolving
> Standards [mailto:[log in to unmask]] On Behalf Of BMarchal
> Sent: 17 July 2011 11:37
> To: [log in to unmask]
> Subject: Re: Theories - local or formal
>
> Hello all,
>
> I think that this exchange, and especially Ray's last response (
> evaluators keep their noses on the ground in immediate programme
> theories; reviewers can benefit more from middle-levels of abstraction
> because it allows them ponder variation of that family of programmes
> yet to cross the policy-maker's mind", neatly summons an answer to the
> initial question of Barend and Marjolein.
>
> In a review, one  focus first on what is reported but one can -- and
> probably should, in order to produce some added value -- reflect the
> findings and outcomes of the study under review against the theories
> and/or best practice that already exist. Confronting existing theory
> with evidence will integrate both and provide stronger theory-based
> frameworks for future interventions.
> One challenge is choosing the theories that are most usefully
> explaining the effect of the programme in question. With our team, we,
> too, found ourselves often questioned by political scientists or
> sociologists about the choice of
> theories: inevitably, the realist reviewer/evaluator only masters some
> disciplines and may tend to eclectically pick theories from other
> disciplines. This points to the need of having a multidisciplinary
> team or researchers with a broad knowledge and experience...
>
> Best,
> -bruno
>
> Bruno Marchal, MD, MPH
> Research Fellow
> Health Care Management Unit
> Department of Public Health
> Institute of Tropical Medicine, Antwerp (Belgium)
>
> Nationalestraat 155, B-2000 Antwerpen, Belgium
> +32.3.2476384
>
>
>> From: Raymond Pawson <[log in to unmask]>
>> Reply-To: "Realist and Meta-narrative Evidence Synthesis: Evolving
>> Standards" <[log in to unmask]>, Raymond Pawson
> <[log in to unmask]>
>> Date: Fri, 15 Jul 2011 13:54:10 +0100
>> To: <[log in to unmask]>
>> Subject: Theories - local or formal
>>
>> Short Friday afternoon answer.
>>
>> There isn't a new programme theory under the sun. So I may be the
>> first
> policy
>> maker to offer grants to help people install wind-turbines in their
>> back gardens but that would make me the 1000th policy maker to use
>> 'incentivisation' as the basis for change. I may be the first policy
>> maker
> in
>> the UK to advocate banning smoking in cars carrying children but this
> would
>> make me the 1000th policy maker trying to control tobacco by a
>> process of 'denormalisation'. Social science operates at a higher
>> level of
> abstraction
>> than evaluation and so, as you say, there are bodies of theory
>> already available which can be accommodated into the design and
>> analysis of a theory-driven review. Broadly speaking evaluators keep
>> their noses on the ground in immediate programme theories; reviewers
>> can benefit more from middle-levels of abstraction because it allows
>> them ponder variation of
> that
>> family of programmes yet to cross the policy-maker's mind (incentives
>> for dead-heading roses).
>>
>> Your question is about what happens if the family of programmes sits
>> potentially under several more abstract theories. As ever, my answer
>> is -
> of
>> course. For instance, there are generally theories which favour more
>> sociological or more psychological theories of change. Choose
>> whichever
> you
>> like. But which you like will take you into different bodies of
>> primary research. The idea is not to end with the total triumph of a
>> particular
> theory
>> but with a refined understanding of the one under review.
>>
>> That said - reference group theory is a total triumph.
>>
>> R
>>
>> -----Original Message-----
>> From: Realist and Meta-narrative Evidence Synthesis: Evolving
>> Standards [mailto:[log in to unmask]] On Behalf Of Gill Westhorp
>> Sent: 15 July 2011 11:07
>> To: [log in to unmask]
>> Subject: Re: Theories - magic or material?
>>
>> Thanks Ray - reassuring to hear this.  At this risk of making a
>> public
> turkey
>> of myself by replying without reading the attachments first:
>>
>> The question (little set of questions) I'm often asked is "and what's
>> the relationship between these (potentially informal) theories and
'formal'
>> (sociological/other substantive domain) theories? What if there
>> doesn't
> seem
>> to be a relationship to formal theory? How does one then choose a
>> formal theory to use to construct an MRT from the outcomes?"
>>
>> I'm thinking here of the apparent 'leap' you made from (hmm, doing
>> this
> from
>> memory after a glass of red on a Friday night)  was it from Naming
>> Shaming
> and
>> Faming to reference group theory? How to decide which theory to 'leap to'
>> there? I'd like to hear your ideas/strategy on the selection of the
>> formal theory...
>>
>> Meanwhile - a variation on the same strategy you've outlined: in a
>> current review, one of the things that I did was note the 'formal
>> theories' that
> were
>> referenced in the literature as I was reading it.  There were
>> several, relevant to different aspects of the question.  Some proved
>> more useful
> than
>> others for developing early propositions, and many of those 'useful
> theories'
>> turned out to be worthy search terms in their own right...
>>
>> Cheers
>> Gill
>
>
> Disclaimer: Http://www.itg.be/disclaimer


Disclaimer: Http://www.itg.be/disclaimer