One additional quick comment, the sentence which reads: "An expert panel achieved only fair interrater agreement on the strength of recommendations based on a body of evidence (5)” is misinterpretation of the findings reported in our paper; in fact, we say opposite, even people with minimal training ( these were not "experts") and exposed to GRADE for the first time, achieve rather good agreement on key GRADE domains. This should have been obvious even from the title “GRADE guidelines system is reproducible when instructions are clearly operationalized even among the guidelines panel members with limited experience with GRADE.” https://www.ncbi.nlm.nih.gov/pubmed/26845745. The key message of the paper is that when instructions are clear, people do tend to agree with each other. Ben djulbegovic From: Evidence based health (EBH) [mailto:[log in to unmask]] On Behalf Of Philipp Dahm Sent: Wednesday, December 07, 2016 8:54 AM To: [log in to unmask] Subject: Re: GRADE: time to change I would encourage interested individuals to also review the 2 commentaries that have been posted in reference to this article, one of which I helped write. http://annals.org/aim/article/2552985/grade-methods-guideline-development-time-evolve David – I don’t understand why the issue you raise isn’t EXACTLY the type of issue a systematic approach to rating the quality of evidence such as GRADE provides allows you to address. Ph* Philipp Dahm, MD, MHSc Professor of Urology, University of Minnesota Director for Surgery/Specialty Care Service Line Research Activities Coordinating Editor, Cochrane Urology Group Minneapolis VA Health Care System, Urology Section 112D One Veterans Drive Minneapolis, MN 55417 Phone: 612 467 3532 Fax: 612 467 2232 Email: [log in to unmask]<http://umn.edu/> Twitter: EBMUrology [cid:image001.png@01D2506E.A7F52050] From: "Evidence based health (EBH)" <[log in to unmask]<mailto:[log in to unmask]>> on behalf of David Richard Leslie Cawthorpe <[log in to unmask]<mailto:[log in to unmask]>> Reply-To: David Richard Leslie Cawthorpe <[log in to unmask]<mailto:[log in to unmask]>> Date: Wednesday, December 7, 2016 at 6:48 AM To: <[log in to unmask]<mailto:[log in to unmask]>> Subject: Re: GRADE: time to change I concur. In an online evidence-based medicine evaluate course (contact me personally if you want to link to it), one of the feature studies illustrated how clinical outcomes are influenced by selection bias in randomized controlled trial's, and in the case of the featured paper-published! While the authors had the temerity to publish the selection bias in the comparison of groups, they failed to comment on the fact that the selection bias completely confounded the results. When these types of results make it into meta-analytic review designed to influence policy, even though (in the case study) the selection bias contributed to a significant finding, it becomes no longer possible to trust meta-analytic findings at face value without precise examination of each sample. This type of examination is necessarily qualitative even though the consequences are quantitative. Hence, approaches such as GRADE will remain limited, and may more reflect the human condition than anything else, given the progress (or lack of it) in evidence-based medicine. Evidence remains something that will require context and interpretation. David Sent from my iPhone On Dec 7, 2016, at 5:08 AM, Juan Gérvas <[log in to unmask]<mailto:[log in to unmask]>> wrote: GRADE Methods for Guideline Development: Time to Evolve Although GRADE methods are evolving (7), they are not currently applicable to many questions that guideline developers face, including those about assessing risk and causality, establishing risk thresholds, or assessing animal studies. Further, GRADE does not provide explicit guidance for complex interventions or when the evidence is linked across a causal pathway, and conceptual frameworks are generally absent. There is only limited GRADE guidance on how to assess the quality of a body of evidence addressing resource use. The GRADE approach is also challenging to apply to different types of data because it was developed for quantitative data with a pooled estimate and CI for each outcome. When data are qualitative or the outcomes cannot be pooled due to heterogeneity, GRADE must be adapted, although the same framework and elements can still be applied. No GRADE guidance is available on how to assess the quality of data from mathematical models or how to incorporate the results of modeling into the development of recommendations. http://annals.org/aim/article/2552985/grade-methods-guideline-development-time-evolve -un saludo juan gérvas