There are so many rules for conducting evaluations - most of which seem to be unmet in practice. We have tried to apply them in evaluating (systematic reviews of) evaluations!
 
My personal checklist looks a bit like this:
 
Predicted outcome / change (otherwise no evidence of effect) i.e. prior goals!
Adequate measurement of baseline states and proper description (detailed) of all key variables
Adequate coverage of confounders/covariates (i.e. things going on at the same time as the 'intervention' that might have an effect: is my grass growing too long because of the rain or the warmth or the interaction of the two, or just because it has been so wet I cannot get the lawnmower out to cut it?) (NB days have also got longer - so look for 'contextual' changes as alternative explanations.)
Explicit theory (this is what is so often lacking) that links the intervention with the (expected) outcomes - e.g. 'Psychological link between trigger events and action, so increased probability of a trigger event like seeing a reminder will lead to attendance for checkup' (etc)
Enough time to demonstrate effect - and followup to see if effect is maintained when stimulus is withdrawn
 
publication in peer reviewed /publically accessible sites
 
'they liked it' / we gave out N packs ... etc is NOT an evaluation.
 
Look at the CONSORT / EQUATOR/ etc sites for more ...
 
 
lEQUATOR (International network)
lCONSORT (Reporting standards for RCTs)
lSTROBE (for Observational studies)
lRATS (for qualitative studies)
lPRISMA (Systematic reviews)
 
l“Enhancing the quality and transparency of health research” Groves T, 2008 - BMJ 337: 66
lBut see also the new NICE practice (NICE 2007, 2010) and comments by Tugwell et al 2010 – focus on ‘fitness for purpose’ and equity concerns.
 
In NICE we used the following checklist to appraise evidence:
 
 
lSection 1: theoretical approach1.1 Is a qualitative approach appropriate? For example:• Does the research question seek to understand processes or structures, or illuminate subjective experiences or meanings?• Could a quantitative approach better have addressed the research question? Appropriate Inappropriate Not sure Comments: Aimed to explore healthcare professionals experiences.
l1.2 Is the study clear in what it seeks to do? For example:• Is the purpose of the study discussed – aims/objectives/research question(s)?• Is there adequate/appropriate reference to the literature?• Are underpinning values/ assumptions/ theory discussed? Clear Unclear Mixed Comments: Introductory paragraphs outlines all these points.
lSection 2: study design2.1 How defensible/rigorous is the research  design/ methodology?
lFor example:• Is the design appropriate to the research question?• Is a rationale given for using a qualitative approach?• Are there clear accounts of the rationale/ justification for the sampling, data collection and data analysis techniques used?• Is the selection of cases/sampling strategy theoretically justified? Defensible Not defensible Not sure Comments: Design is appropriate to question.  Rationale given for purposeful sample and use of focus groups
 
 
l3.1 How well was the data collection carried out? For example:•
lAre the data collection methods clearly described?• Were the appropriate data collected to address the research question?• Was the data collection and record keeping systematic? Appropriate Inappropriate Not sure/inadequately reported Comments: Probably appropriate, but section too brief to be sure. Audiotapes and verbatim transcriptions of focus groups were kept.
l4.1 Is the role of the researcher clearly described? For example:• Has the relationship between the researcher and the participants been adequately considered?• Does the paper describe how the research was explained and presented to the participants? Clear Unclear Not described Comments: No discussion of relationship between researcher and participants.
Invitation and info about the study was sent to potential participants via local service contacts
l.4.2 Is the context clearly described? For example:• Are the characteristics of the participants and settings clearly defined?• Were observations made in a sufficient variety of circumstances?• Was context bias considered? Clear Unclear Not sure Comments: Table of characteristics given, settings clearly defined.  Focus groups took place in a range circumstances.  Context bias considered: “..findings must be interpreted with regard to the study context...”
l.4.3 Were the methods reliable? For example:• Were data collected by more than one method?• Is there justification for triangulation, or for not triangulating?• Do the methods investigate what they claim to? Reliable Unreliable Not sure Comments: More than 1 composition of focus groups – professional groups and multidisciplinary teams.  No other information given.
 
 
Is the data analysis sufficiently rigorous? For example:• Is the procedure explicit – is it clear how the data were analysed to arrive at the results?• How systematic is the analysis – is the procedure reliable/ dependable?• Is it clear how the themes and concepts were derived from the data? Rigorous Not rigorous  Not sure/ not reported Comments: Constant comparison.  Deviant cases sought to assess integrity of categories.
l5.2 Are the data ‘rich’? For example:• How well are the contexts of the data described?• Has the diversity of perspective and content been explored?• How well have the detail and depth been demonstrated?• Are responses compared and contrasted across groups/sites? Rich Poor Not sure/not reported Comments: Boxes 1-4 compare and contrast within themes. Contexts well described.
l5.3 Is the analysis reliable? For example:• Did more than one researcher theme and code transcripts/data?• If so, how were differences resolved?• Did participants feed back on the transcripts/data? (if possible and relevant)• Were negative/discrepant results addressed or ignored? Reliable Unreliable Not sure/ not reported Comments: All 3 researchers, from different backgrounds, coded and themed. Discussion and agreement. Summary of results sent to all participants for comment. 7 attended a follow-up focus group to check validity of interpretation. Deviant cases sought to assess the integrity of the categories identified.
l5.4 Are the findings convincing? For example:• Are the findings clearly presented?• Are the findings internally coherent?• Are extracts from the original data included?• Are the data appropriately referenced?• Is the reporting clear and coherent? Convincing Not convincing Not sure Comments: Clearly presented, original extracts included and referenced.5.5 Are the findings relevant to the aims of the study? Relevant irrelevant Partially relevant Comments:
l5.6 Are the conclusions adequate? For example:• How clear are the links between data, interpretation and conclusions?• Are the conclusions plausible and coherent?• Have alternative explanations been explored and discounted?• Does this study enhance understanding of the research subject?• Are the implications of the research clearly defined?• Is there adequate discussion of any limitations  encountered? Adequate Inadequate  Not sure  Comments: Clear links between data, interpretation and conclusions.  Enhances understanding of the subject.  Discussion of limitations and generalisability.
 
l6.1 How clear and coherent is the reporting of ethical considerations? For example,• Have ethical issues been taken into consideration?• Are ethical issues discussed adequately – do they address consent and anonymity?• Have the consequences of the research been considered; for example, raising expectations, changing behaviour?• Was the study approved by an ethics committee? Clear Not clear Not sure/ not reported
lComments: Ethics committee approval gained. Written informed consent given by participants.
 
 
lNICE checklist is based on checklists in:
lSpencer L. Ritchie J, Lewis J, Dillon L (2003) Quality in qualitative evaluation: a framework for assessing research evidence. London: Government Chief Social Researcher’s Office. Available from:
lwww.strategy.gov.uk/downloads/su/qual/downloads/qqe_rep.pdf
lPublic Health Resource Unit England (2006) Critical Appraisal Skills Programme (CASP) – making sense of evidence: 10 questions to help you make sense of qualitative research . Available from:
lwww.phru.nhs.uk/Doc_Links/Qualitative%20Appraisal%20Tool.pdf
lNational Training and Research Appraisal Group (NTRAG); contact: www.ntrag.co.uk
lBritish Sociological Association (BSA); contact: www.britsoc.co.uk
lI Methodology checklist: qualitative studies
l© National Institute for Health and Clinical Excellence (January 2009)
 
See also
 
lTugwell P, Petticrew M, Kristjansson E, Welch V, Ueffing E, Waters E, Bonnefoy J, Morgan A, Doohan E, Kelly MP 2010 “Assessing equity in systematic reviews: realising the recommendations of the Commission on Social Determinants of Health” BMJ 341: c4739  doi: 10.1136/bmj.c4739

lWelch V, Tugwell P, Petticrew M, de Montigny J, Ueffing E, Kristjansson B, McGowan J, Benkhalti Jandu M, Wells GA, Brand K, Smylie J. 2010 How effects on health equity are assessed in systematic reviews of interventions.  Cochrane Database Syst Rev. 2010 Dec 8;12:MR000028

 

Hope all this helps...

 

(sorry about the berserk formatting: don't know what is happening here)

 

Mark
Moderator, Minority-Ethnic-Health Discussion List
www.jiscmail.ac.uk/minority-ethnic-health