 |
 |
Hi Arun I’ve just completed a review of programme documentation in support of a realist synthesis. All the documentation was from one organisation running one programme, stretching over a 7 year period. It included everything they had on file (planning, correspondence, reviews, mtg notes etc). There were around 3500 items. I set up an excel spreadsheet and extracted ‘data’ into 7 categories which were based on Pawson’s VICTORE. It gave me a great grounding in how the programme was run and managed and what/who influenced its direction over time. It didn’t give me much on CMOs for programme participants. It did though show me where I might find that data. In this case I found a deep well of blogs and case studies that had been assembled in newsletters and websites. There was also some uncoded open field data from several participant surveys they had done and that was insightful too. From the participant documentation sources I have identified several realist programme theories (CMO configurations). I approached the latter task by coding all outcomes mentioned in the participant documents and then revisited the documentation again to tease out specific contexts and possible mechanisms for each outcome. The mechanism identification felt like an art rather than science which was a little unusual to work through for me. I am sticking with it for now - trusting my gut and some mid range theories! I’m new to realist work (this is my PhD and I’m in my first year). I thought I’d share in the spirit of RAMESES network! There might be something useful for you in there. All the best Mary ________________________________ From: Realist and Meta-narrative Evidence Synthesis: Evolving Standards <[log in to unmask]> on behalf of Hardwick, Rebecca <[log in to unmask]> Sent: 18 April 2018 12:16:18 To: [log in to unmask] Subject: Re: Using realist synthesis to evaluate documents? Hi Arun, You might be interested in my MSc dissertation which was a realist review of project evaluation reports of services known as 'One Stop Shops', for women in the criminal justice system: I didn't use 'peer reviewed' journal articles, and instead, found a host of different service evaluation reports which I used to build and refine programme theory. I found these evaluations by reading the policy documents that were relevant to that domain (e.g the 'Corston Report' was a key one at that time), and by searching the references in those policy documents. (PM me if you want to see the whole dissertation, but the journal publication is here: https://www.emeraldinsight.com/doi/abs/10.1108/JICA-06-2013-0016 ) So it might be worthwhile look at what the policy arena you're working in has produced (so central govt department websites, research intermediary type organisations, third sector organisation reports and so on), and reading those to get a sense of the likely programme theories. As we know, mechanisms operate at a middle range of abstraction, so even if the setting is different, the pattern of outcomes across the evaluations will be pointing towards generative mechanisms that would be at work in your own area of enquiry. Good luck, Becky Rebecca Hardwick PhD Student 01392 727408 Email: [log in to unmask] -----Original Message----- From: Realist and Meta-narrative Evidence Synthesis: Evolving Standards [mailto:[log in to unmask]] On Behalf Of Arun Verma CSF Sent: 18 April 2018 11:04 To: [log in to unmask] Subject: Re: Using realist synthesis to evaluate documents? Hi Rob, Thank you for your reflections, I work in Commissioning in government so I have access to progress reports/meeting minutes to explore some outcomes as a way to begin to unpack causal assumptions, however I share your thoughts around this type of review perhaps giving more of an overarching view of the progam theor. Look forward to others' thoughts on using a 'realist synthesis approach to evaluation'. Best wishes, Arun -----Original Message----- From: Anderson, Rob [mailto:[log in to unmask]] Sent: 18 April 2018 10:53 To: Realist and Meta-narrative Evidence Synthesis: Evolving Standards <[log in to unmask]>; Arun Verma CSF <[log in to unmask]> Subject: RE: Using realist synthesis to evaluate documents? Hi Arun, I think the reason people may not have considered or voiced this question before is that the approach of Realist Synthesis does not presume that rich insights and findings about how programmes or policies work will reside in any particular types of document. It's all situation-specific. In fact, in my field of health care and public health, research articles (which are also simply documents of course) about a programme are often very poor at revealing the underlying rationale and causal assumptions that underlie a proposed or evaluated health service change. In contrast policy documents, or (for example) the minutes of policy or commissioning meetings, in some cases will provide the only or a more rich source of the underlying rationale of programmes and explicit statements about way people believe service changes will make things better for patients/service users. However, in my experience policy documents will often frame and express programme theory in higher level terms of policy rhetoric, without clearly specifying how the specific combinations of programme elements will work together to produce the expected outcomes. As for the other problem you allude to - how to find the particular types of policy/service documents which will provide rich data and insights about programme/policy theory and explaining programme/policy success or failure - there must be others on RAMESES who have some experience of doing this (I do not). Hope my reflections are useful. Good luck! Rob ESMI, University of Exeter Medical School -----Original Message----- From: Realist and Meta-narrative Evidence Synthesis: Evolving Standards [mailto:[log in to unmask]] On Behalf Of Arun Verma Sent: 18 April 2018 10:19 To: [log in to unmask] Subject: Using realist synthesis to evaluate documents? Good morning everyone, I'm starting to get to grips on the complexity of literature around realist methodologies. However, I have a question (which I'm not sure if it has come up before). I would like to evaluate policy/service documents for a project I'm doing using a realist framework. Is there a way in which one can conduct an evaluation of documents using a 'Realist synthesis' protocol? Are there any papers on this - I'm struggling to be creative with keywords in my search at the moment. Best wishes, Arun * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * This email and any attachments with it are intended for the addressee only. It may be confidential and may be the subject of legal and/or professional privilege. If you have received this email in error please notify the sender or [log in to unmask] The content may be personal or contain personal opinions and cannot be taken as an expression of the County Council's position. Surrey County Council reserves the right to monitor all incoming and outgoing mail. Whilst every care has been taken to check this e-mail for viruses, it is your responsibility to carry out any checks upon receipt. Visit the Surrey County Council website - http://www.surreycc.gov.uk * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *