I agree with the consensus that is accumulating in this discussion on
ROT's but wonder if we have evidence for this following statement
ROTn: if the raw data or simple stats do not suggest an effect, then
the effect found by sophisticated statistics is unlikely to be important
in practice.
Pragmatically this sounds reasonable but isn't that what the Cochrane
logo shows us - advanced stats on small studies (plural) give a result
that is meaningful so maybe it should have a proviso - "in single
adequately powered studies if the raw data....."
Martin
Dr Martin Dawes
Chair Family Medicine
McGill University
515 Pine Avenue West, Montreal
Quebec, Canada H2W 1S4
Tel 514 398 7375 x0469
Fax 514 398 4202
*** This communication is confidential. Please request explicit
permission before copying or forwarding. If you are not the intended
recipient, you are notified that any distribution or copying of this
email is strictly prohibited. If you have received this email in error,
please notify us immediately by return email and delete all copies of
the message. Thank you. ***
-----Original Message-----
From: Evidence based health (EBH)
[mailto:[log in to unmask]] On Behalf Of Michael Power
Sent: 02 July 2008 11:13
To: [log in to unmask]
Subject: Re: quesetionable statistics in meta-analysis
Steve
Thanks very much for your clear explanation of hierarchical bayesian
meta-
analysis, and for your practical and reassuring rules of thumb (ROT):
ROT1: the first rule in critical appraisal is to focus on how the data
was collected and not on how it was analyzed
ROT2: we have to trust the experts to some extent
I have no problems with your first ROT.
Your 2nd ROT is practical, but it needs to be balanced by a
complementary
ROT about when NOT to trust the experts.
- ROT3: never trust an expert --- unless you have no other option; or
they have shown themselves to be trustworthy (you hinted at this by
suggesting that Brian check that a statistician had been involved)
I would also suggest another ROT for critical appraisal:
ROTn: if the raw data or simple stats do not suggest an effect, then
the
effect found by sophisticated statistics is unlikely to be important in
practice.
I can think of examples in physics that support ROTn: neutrinos and
gravity waves need very clever machines to detect them, and they do not
affect our daily lives much.
In healthcare are there exceptions to ROTn's rule that important effects
are suggested by the raw data or simple stats?
In Brian's case does a graphic display of results of individual trials
suggest the effect measured by Bayesian hierachical meta-analysis (I
have
not seen the paper)? If no, any effect is unlikely to be clinically
important. If yes, you could be less sceptical about the results.
Michael Power
Clinical Knowledge Summaries Service www.cks.library.nhs.uk
standard disclaimers (plagiarised from steve)
|