Hello Ben,
This is very challenging. It is difficult to believe that we can not identify a trend or pattern most of the time.
I would like to have a go. I'm ready to be a volunteer :)
Sent from my iPod
On 4 Jul 2008, at 00:21, "Djulbegovic, Benjamin" <[log in to unmask]> wrote:
Fantastic, Paul
Let's do it- happy to prepare the material. So, can I ask those
interested to send me forest plots that we should include (it would be
nice if the identifiers such as the name of interventions or trials id
can be removed). Also, we can extend this invitation to other volunteers
as well (so far, we have you and Martin)- so, I am asking those who'd
like to participate to send me their contacts.
This is going to be a fun- it is great to see that one off-hand remark
can potentially proceed to an actionable project.
Best
Ben
Ps Timetable: since I will be moving my office at the end of the month,
it would be great to hear from everyone interested by then. I will then
compile the material (including some of my own) sometimes in mid August
-----Original Message-----
From: Evidence based health (EBH)
[mailto:[log in to unmask]] On Behalf Of Paul Glasziou
Sent: Thursday, July 03, 2008 13:44
To: [log in to unmask]
Subject: Re: quesetionable statistics in meta-analysis - ROT
Hi Martin & Ben
You have me fascinated - I would have thought this was easy. So I'd love
to be a subject.
But I imagine some forest plots will be hard and some easy. They should
have more than 4 trials and vary by
a. number of studies b. degree of significance and c. heterogeneity.
Ben - are you happy to do this? We could also ask for volunteers to send
you forest plots (but they can't take part in the guessing competition),
Cheers
Paul
Martin Dawes, Dr. wrote:
OK - good challenge
But let's do this properly
Probably the first step is a pilot of 10 or so - 5 volunteers (I am
happy to be one of these) will look at the plots and then guess the
stat signif and its size and direction without cheating
Someone will prepare the 10 plots by removing the pooled estimates
(maybe Ben as he proposed the challenge)
Then another impartial observer will report back the anonymous pooled
and individual results from the experiment.
Martin
Dr Martin Dawes
Chair Family Medicine
McGill University
515 Pine Avenue West, Montreal
Quebec, Canada H2W 1S4
Tel 514 398 7375 x0469
Fax 514 398 4202
*** This communication is confidential. Please request explicit
permission before copying or forwarding. If you are not the intended
recipient, you are notified that any distribution or copying of this
email is strictly prohibited. If you have received this email in
error, please notify us immediately by return email and delete all
copies of the message. Thank you. ***
------------------------------------------------------------------------
*From:* Djulbegovic, Benjamin
[mailto:[log in to unmask]]
*Sent:* 03 July 2008 10:00
*To:* Martin Dawes, Dr.; [log in to unmask]
*Subject:* RE: quesetionable statistics in meta-analysis - ROT
Martin/Michael
I challenge both of you to select any number of forest plots (have
someone remove the pooled estimates) and then try to guess the
statistical significance of the treatment effect (its size or or its
direction). I bet that your visual impression will not be better than
chance.
ben d
------------------------------------------------------------------------
*From:* Evidence based health (EBH) on behalf of Martin Dawes
*Sent:* Thu 7/3/2008 7:33 AM
*To:* [log in to unmask]
*Subject:* Re: quesetionable statistics in meta-analysis - ROT
Hi
I agree - particularly with the central premis that if it isn't
obvious on the first eye-ball of data then it probably is not
going/should not see the clinical light of day. Maybe there should be
an additional questions similar to decision threshold for diagnostic
tests that are stated/agreed before the study is undertaken. THey
might include "Is this intervention feasible in practice" "What global
reduction in adverse outcome is required to change practice".
At the same time maybe we should be doing a model of the cost
effectiveness before the study is done. Maybe that is what should be
required by ethics committees
Martin
On 03/07/2008 05:11, "Michael Power" <[log in to unmask]>
wrote:
Hi Martin
I used the words "suggest an effect" deliberately because, ROTn-1: if
you plot the "raw" data for a meta-analysis (measure of central
tendency
and confidence interval for each included study), you get a visual
impression that suggests the direction of effect. The meta-analysis
quantifies the visual impression of direction and provides a
confidence
interval. If the meta-analysis is consistent with the visual
impression,
it confirms its face validity. If the meta-analysis is NOT consistent
with your visual impression, you should check the numbers, graphing
and
calculations!
Michael
-----Original Message-----
From: Martin Dawes, Dr. [mailto:[log in to unmask]]
Sent: 02 July 2008 18:07
Subject: Re: quesetionable statistics in meta-analysis - ROT
I agree with the consensus that is accumulating in this discussion on
ROT's but wonder if we have evidence for this following statement
ROTn: if the raw data or simple stats do not suggest an effect, then
the effect found by sophisticated statistics is unlikely to be
important
in practice.
Pragmatically this sounds reasonable but isn't that what the Cochrane
logo shows us - advanced stats on small studies (plural) give a result
that is meaningful so maybe it should have a proviso - "in single
adequately powered studies if the raw data....."
Martin
Dr Martin Dawes
Chair Family Medicine
McGill University
515 Pine Avenue West, Montreal
Quebec, Canada H2W 1S4
Tel 514 398 7375 x0469
Fax 514 398 4202
Dr Martin Dawes
Chair Family Medicine
McGill University
515 Pine Avenue West, Montreal
Quebec, Canada
Tel 514 398 7375 x0469
Fax 514 398 4202
--
Paul Glasziou
Director, Centre for Evidence-Based Medicine,
Department of Primary Health Care,
University of Oxford www.cebm.net
ph - +44-1865-289298 fax +44-1865-289287
This transmission may be confidential or protected from disclosure and is only for review and use by the intended recipient. Access by anyone else is unauthorized. Any unauthorized reader is hereby notified that any review, use, dissemination, disclosure or copying of this information, or any act or omission taken in reliance on it, is prohibited and may be unlawful. If you received this transmission in error, please notify the sender immediately. Thank you.
|