 |
 |
Fantastic, Paul Let's do it- happy to prepare the material. So, can I ask those interested to send me forest plots that we should include (it would be nice if the identifiers such as the name of interventions or trials id can be removed). Also, we can extend this invitation to other volunteers as well (so far, we have you and Martin)- so, I am asking those who'd like to participate to send me their contacts. This is going to be a fun- it is great to see that one off-hand remark can potentially proceed to an actionable project. Best Ben Ps Timetable: since I will be moving my office at the end of the month, it would be great to hear from everyone interested by then. I will then compile the material (including some of my own) sometimes in mid August -----Original Message----- From: Evidence based health (EBH) [mailto:[log in to unmask]] On Behalf Of Paul Glasziou Sent: Thursday, July 03, 2008 13:44 To: [log in to unmask] Subject: Re: quesetionable statistics in meta-analysis - ROT Hi Martin & Ben You have me fascinated - I would have thought this was easy. So I'd love to be a subject. But I imagine some forest plots will be hard and some easy. They should have more than 4 trials and vary by a. number of studies b. degree of significance and c. heterogeneity. Ben - are you happy to do this? We could also ask for volunteers to send you forest plots (but they can't take part in the guessing competition), Cheers Paul Martin Dawes, Dr. wrote: > > OK - good challenge > > But let's do this properly > > Probably the first step is a pilot of 10 or so - 5 volunteers (I am > happy to be one of these) will look at the plots and then guess the > stat signif and its size and direction without cheating > > Someone will prepare the 10 plots by removing the pooled estimates > (maybe Ben as he proposed the challenge) > > Then another impartial observer will report back the anonymous pooled > and individual results from the experiment. > > Martin > > Dr Martin Dawes > > Chair Family Medicine > > McGill University > > 515 Pine Avenue West, Montreal > > Quebec, Canada H2W 1S4 > > Tel 514 398 7375 x0469 > > Fax 514 398 4202 > > *** This communication is confidential. Please request explicit > permission before copying or forwarding. If you are not the intended > recipient, you are notified that any distribution or copying of this > email is strictly prohibited. If you have received this email in > error, please notify us immediately by return email and delete all > copies of the message. Thank you. *** > > ------------------------------------------------------------------------ > > *From:* Djulbegovic, Benjamin [mailto:[log in to unmask]] > *Sent:* 03 July 2008 10:00 > *To:* Martin Dawes, Dr.; [log in to unmask] > *Subject:* RE: quesetionable statistics in meta-analysis - ROT > > Martin/Michael > > I challenge both of you to select any number of forest plots (have > someone remove the pooled estimates) and then try to guess the > statistical significance of the treatment effect (its size or or its > direction). I bet that your visual impression will not be better than > chance. > > ben d > > ------------------------------------------------------------------------ > > *From:* Evidence based health (EBH) on behalf of Martin Dawes > *Sent:* Thu 7/3/2008 7:33 AM > *To:* [log in to unmask] > *Subject:* Re: quesetionable statistics in meta-analysis - ROT > > Hi > I agree - particularly with the central premis that if it isn't > obvious on the first eye-ball of data then it probably is not > going/should not see the clinical light of day. Maybe there should be > an additional questions similar to decision threshold for diagnostic > tests that are stated/agreed before the study is undertaken. THey > might include "Is this intervention feasible in practice" "What global > reduction in adverse outcome is required to change practice". > At the same time maybe we should be doing a model of the cost > effectiveness before the study is done. Maybe that is what should be > required by ethics committees > > Martin > > > On 03/07/2008 05:11, "Michael Power" <[log in to unmask]> wrote: > > Hi Martin > > I used the words "suggest an effect" deliberately because, ROTn-1: if > you plot the "raw" data for a meta-analysis (measure of central tendency > and confidence interval for each included study), you get a visual > impression that suggests the direction of effect. The meta-analysis > quantifies the visual impression of direction and provides a confidence > interval. If the meta-analysis is consistent with the visual impression, > it confirms its face validity. If the meta-analysis is NOT consistent > with your visual impression, you should check the numbers, graphing and > calculations! > > Michael > > > -----Original Message----- > From: Martin Dawes, Dr. [mailto:[log in to unmask]] > Sent: 02 July 2008 18:07 > Subject: Re: quesetionable statistics in meta-analysis - ROT > > I agree with the consensus that is accumulating in this discussion on > ROT's but wonder if we have evidence for this following statement > > ROTn: if the raw data or simple stats do not suggest an effect, then > the effect found by sophisticated statistics is unlikely to be important > in practice. > > Pragmatically this sounds reasonable but isn't that what the Cochrane > logo shows us - advanced stats on small studies (plural) give a result > that is meaningful so maybe it should have a proviso - "in single > adequately powered studies if the raw data....." > > Martin > > > Dr Martin Dawes > Chair Family Medicine > McGill University > 515 Pine Avenue West, Montreal > Quebec, Canada H2W 1S4 > Tel 514 398 7375 x0469 > Fax 514 398 4202 > > > > > Dr Martin Dawes > Chair Family Medicine > McGill University > 515 Pine Avenue West, Montreal > Quebec, Canada > Tel 514 398 7375 x0469 > Fax 514 398 4202 > -- Paul Glasziou Director, Centre for Evidence-Based Medicine, Department of Primary Health Care, University of Oxford www.cebm.net ph - +44-1865-289298 fax +44-1865-289287 This transmission may be confidential or protected from disclosure and is only for review and use by the intended recipient. Access by anyone else is unauthorized. Any unauthorized reader is hereby notified that any review, use, dissemination, disclosure or copying of this information, or any act or omission taken in reliance on it, is prohibited and may be unlawful. If you received this transmission in error, please notify the sender immediately. Thank you.