There is a validated scoring system developed by Guyatt and Oxman (2 papers
published with appropriate validation) for systematic reviews.
We are using our adaptation of this for RCTs (I can provide if it interests
you, but it has not been appropriately validated or used elsewhere, but it
is serving us well) since the Jadad score appears insufficient in light of
recent developments such as empirical evidence for bias associated with
allocation concealment, the importance of blinding, and the intention to
The sytematic review references:
J Clin Epidemiol 1991;44(11):1271-8
Validation of an index of the quality of review articles.
Oxman AD, Guyatt GH.
Department of Family Medicine, Faculty of Health Sciences, McMaster
University, Hamilton, Ontario, Canada.
The objective of this study was to assess the validity of an index of the
scientific quality of research overviews, the Overview Quality Assessment
Questionnaire (OQAQ). Thirty-six published review articles were assessed by
9 judges using the OQAQ. Authors reports of what they had done were compared
to OQAQ ratings. The sensibility of the OQAQ was assessed using a 13 item
questionnaire. Seven a priori hypotheses were used to assess construct
validity. The review articles were drawn from three sampling frames:
articles highly rated by criteria external to the study, meta-analyses, and
a broad spectrum of medical journals. Three categories of judges were used
to assess the articles: research assistants, clinicians with research
training and experts in research methodology, with 3 judges in each
category. The sensibility of the index was assessed by 15 randomly selected
faculty members of the Department of Clinical Epidemiology and Biostatistics
at McMaster. Authors' reports of their methods related closely to ratings
from corresponding OQAQ items: for each criterion, the mean score was
significantly higher for articles for which the authors responses indicated
that they had used more rigorous methods. For 10 of the 13 questions used to
assess sensibility the mean rating was 5 or greater, indicating general
satisfaction with the instrument. The primary shortcoming noted was the need
for judgement in applying the index. Six of the 7 hypotheses used to test
construct validity held true. The OQAQ is a valid measure of the quality of
PMID: 1834807 [PubMed - indexed for MEDLINE]
2: J Clin Epidemiol 1991;44(1):91-8
Agreement among reviewers of review articles.
Oxman AD, Guyatt GH, Singer J, Goldsmith CH, Hutchison BG, Milner RA,
Department of Clinical Epidemiology & Biostatistics, Faculty of Health
Sciences, McMaster University, Hamilton, Ontario, Canada.
OBJECTIVE. To assess the consistency of an index of the scientific quality
of research overviews. DESIGN. Agreement was measured among nine judges,
each of whom assessed the scientific quality of 36 published review
articles. ITEM SELECTION. An iterative process was used to select ten
criteria relative to five key tasks entailed in conducting a research
overview. SAMPLE. The review articles were drawn from three sampling frames:
articles highly rated by criteria external to the study; meta-analyses; and
a broad spectrum of medical journals. JUDGES. Three categories of judges
were used: research assistants; clinicians with research training; and
experts in research methodology; with three judges in each category.
RESULTS. The level of agreement within the three groups of judges was
similar for their overall assessment of scientific quality and for six of
the nine other items. With four exceptions, agreement among judges within
each group and across groups, as measured by the intraclass correlation
coefficient (ICC), was greater than 0.5, and 60% (24/40) of the ICCs were
greater than 0.7. CONCLUSIONS. It was possible to achieve reasonable to
excellent agreement for all of the items in the index, including the overall
assessment of scientific quality. The implications of these results for
practising clinicians and the peer review system are discussed.
PMID: 1824710 [PubMed - indexed for MEDLINE]
From: Gero Langer [mailto:[log in to unmask]]
Sent: Monday, August 13, 2001 3:32 AM
To: [log in to unmask]
Subject: validated instruments for critical appraisal
I am looking for some validated instruments to critically appraise
studies. It is important to find the 'best' studies, and a (validated)
rating system for all kinds of questions (intervention, diagnosis,
qualitative etc.) should be used.
Currently I am using the JAMA users' guides, but they are not validated
(or?) and comparisons between studies are difficult and subjective. For
RCTs I am working with the Jadad score. With all of those I could get a
'result', but not a 'comparing' (e.g. rated) solution.
We are developing a database for nurses in Germany and trying to offer the
best available evidence for some nursing problems -- but what is unbiassed
the best? A scoring system would be very useful... Does anyone know of
anything in this field?
Thanks in advance,
Martin Luther University Halle-Wittenberg
Institute for Nursing and Health Sciences
German Center for Evidence-based Nursing
Website: www.EBN-Zentrum.de E-Mail: [log in to unmask]