I like the use of this listserv to take a common question and develop a
research protocol, and I like Michael's overall approach to this.
But let's clarify what we are trying to learn with the research.
Let's assume we were discussing whether or not someone can visually
determine if a person is taller or shorter than 5 feet. If our dataset
were composed of 3-foot and 7-foot people then our test subjects should
be accurate in estimated whether they are taller or shorter than 5 feet.
If our dataset were compose of 4.99-foot and 5.01-foot subjects then our
test subjects should be no better than chance in their estimation.
We could get any answer we want by choosing the subjects.
A better question might be What is the minimal difference in height at
which most people can visually distinguish one person from another
5-foot person by visual estimation of height? If that were our
question then we would need a dataset with known heights and our
statistical approach would be considerably different.
So what is our real question:
A) When can visual estimation of a meta-analysis forest plot accurately
predict the overall results?
B) Can visual estimation of a meta-analysis forest plot meeting [fill in
defined criteria here] accurately predict the overall result?
C) Can visual estimation of an unselected meta-analysis forest plot
accurately predict the overall result? [[[For this question random
sampling could be used for the dataset rather than finding the
"difficult" or "easy" ones]]]
--------------------------------------
Brian S. Alper, MD, MSPH
Editor-in-Chief, DynaMed (www.DynamicMedical.com)
Medical Director, EBSCO Publishing
10 Estes St.
Ipswich, MA 01938
office (978) 356-6500 extension 2749
cell (978) 804-8719
fax (978) 356-6565
home (978) 356-3266
"It only takes a pebble to start an avalanche."
This e-mail and any attached files transmitted are confidential and
solely for the use of the intended recipient. It may contain information
which is covered by professional or other privilege. If you are neither
the intended recipient of this e-mail nor the person responsible for
delivering it to the intended recipient, be advised that you have
received this e-mail in error and that any use of it is strictly
prohibited. Please notify the sender immediately by reply e-mail and
then delete from your system. EBSCO Industries, Inc., its subsidiaries
and divisions, accept no responsibility for any loss or damage suffered
by any person arising from the use of this e-mail.
-----Original Message-----
From: Evidence based health (EBH)
[mailto:[log in to unmask]] On Behalf Of Michael Power
Sent: Friday, July 04, 2008 3:51 AM
To: [log in to unmask]
Subject: Re: quesetionable statistics in meta-analysis - ROT
Great idea, and I am keen to participate.
But, to prevent the project not meeting SS's first rule of thumb for
critical appraisal (first focus on how the data was collected), we
should
clarify the research question and protocol before hapahazardly
collecting
data.
So, here is an outline suggestion for the first draft protocol:
DRAFT
Question
Can eyeballing a meta-analysis's forest plot with the pooled effect
blanked out accurately estimate the presence/absence of significance in
the pooled estimate of effect.
Methods
Data collection: First set: 30 forest plots of the primary outcome
measure
in 30 Cochrane systematic reviews. Second set: 10 forest plots of the
primary outcome measure in 10 Cochrane systematic reviews where the
primary outcome was just signifiant AND 10 forest plots of the primary
outcome measure in 10 Cochrane systematic reviews where the primary
outcome was just NOT signifiant.
Primary outcome measures: sensitivity and specificy of eyeballing;
positive and negative likelihood ratios of eyeballing.
Any volunteers to make a numbered list of Cochrane SRs that could be
used
for randomizing selection?
Any volunteers to screen Cochrane SRs for forest plots where the
significance is just above/below 5%?
Michael
On Thu, 3 Jul 2008 17:21:02 -0400, Djulbegovic, Benjamin
<[log in to unmask]> wrote:
>Fantastic, Paul
>Let's do it- happy to prepare the material. So, can I ask those
>interested to send me forest plots that we should include (it would be
>nice if the identifiers such as the name of interventions or trials id
>can be removed). Also, we can extend this invitation to other
volunteers
>as well (so far, we have you and Martin)- so, I am asking those who'd
>like to participate to send me their contacts.
>This is going to be a fun- it is great to see that one off-hand remark
>can potentially proceed to an actionable project.
>Best
>Ben
|