Anna, optimizing the outputs of your realist review means customizing protocols in such a way to harness insights from existing literature or using that material to cultivate your own insights in your area of  investigation. As has been said, the realist paradigm supports our 'flattening' of the traditional hierarchy of research evidence because insights can come from anywhere - empirical studies, grey literature and commentaries. It would be very wise to include a table at the end of your study listing the articles retained and categorizing according to type of article (empirical, commentary) and methodology used, if any (qual, quant etc.) Transparency back through to your source documents is important. 



What counts as an insightful claim is a good question - but essentially should contain content implicitly relating to mechanisms, context elements that are important, or the C-M interaction. Read through your commentary pieces and determine how much insight they contain. If a commentary is simply saying 'we need more research in this area', that is not the kind of insight you are looking for. But a commentary piece written by an experienced practitioner in the field, reflecting on her or his experience about how something is working - you may score this higher in your appraisal than a large study on the topic that has a results section but is very thin on insights about why things happened the way they did. It's an odd thing - if a practitioner in the field writes a commentary piece and publishes this, it is largely considered anecdotal. Yet if you conducted a qualitative interview with this person in a formalized study, and the person said exactly the same thing in the interview as they did in the commentary, we would tend to give that more credibility, even though through the qualitative analysis we have cut up their narrative into a series of fragmented statements.  I'm being intentionally controversial here. There might be some justification for the hierarchy, but I think it is rather weak. Using 'insight' as our yardstick for success we can justify the inclusion of a variety of sources -



So the short answer to your question is that you should include commentaries and grey literature to the extent to which they help you make insightful claims about how, for whom, under what circumstances the programme works. 



Here is a relevant article that profoundly influenced my thinking that may help:

Pawson, R 2006. Digging for Nuggets: How 'bad' research can yield 'good' evidence. International Journal of Social Research Methodology, 9, 127-142.





Best of luck

Justin



Justin Jagosh, Ph.D

Director, Centre for Advancement in Realist Evaluation and Synthesis (CARES)

www.realistmethodology-cares.org

&

Honorary Research Associate

Institute for Psychology, Health and Society

University of Liverpool, UK

www.liv.ac.uk/cares







-----Original Message-----

From: Realist and Meta-narrative Evidence Synthesis: Evolving Standards [mailto:[log in to unmask]] On Behalf Of Anna Richmond

Sent: June 30, 2017 3:21 AM

To: [log in to unmask]

Subject: use of commentary and opinion only paper in realist synthesis



Dear all



I am performing a realist synthesis and my search has obtained around 150 papers after title and abstract screening that seem as they may help me refine/refute my programme theory. However, on a brief scroll through, many are commentaries, review articles or opinion papers etc. 



I just wondered what the usefulness of these types of non-empirical articles to a realist synthesis? Is there a place for them or not?



Thanks



Anna Richmond