Just a word of caution about interpreting statistical significance from confidence intervals. Comparing traditional p values to confidence intervals and point estimates, it becomes clear that overlapping CIs do not mean no significance. It is the overlap of the CI of one group with the point estimate of the other group that implies a lack of statistical significance. In fact, only if both group CIs overlap both point estimates is one really assured of no statistical significance. If there is overlap of one point estimate, but not the other, that is a borderline situation that may or may not be statistically significant, depending on the method of calculating a p value.
Another problem to watch for with CIs are intervals that are too wide to be useful. This implies low statistical power. There is probably low power if the CI of the control group surpasses what would be the smallest clinically significant difference, or if the experimental group point estimate is beyond the smallest clinically significant difference yet the CI of the experimental group still overlaps the control group point estimate. In the case of low power, a finding of "no significance" is inconclusive and does not mean no effect. A very informative way to examine power is to calculate the smallest difference that could be detected with statistical significance. If that difference is larger than the smallest clinically significant difference, then there is inadequate power to detect differences that would be useful clinically.
The "counter-null" point is also a useful observation with CIs. Even if the CI overlaps the point estimate, there is a point at the opposite side of the interval that has equal probability with the "zero difference" point where the interval overlaps the point estimate of the other group. In an adequately powered study with a finding of no significance, the counter-null point will be smaller than the smallest clinically significant difference.
While these problems with CIs may seem complicated and messy, they merely illustrate the messiness that underlies the overly simplistic concept of "statistical significance" based on p values, particularly in low-powered studies.
David L. Doggett, Ph.D.
Senior Medical Research Analyst
Health Technology Assessment and Information Services
ECRI, a nonprofit health services research organization
5200 Butler Pike
Plymouth Meeting, PA 19462, USA
Phone: (610) 825-6000 x5509
FAX: (610) 834-1275
e-mail: [log in to unmask]
From: padmanabhan badrinath [mailto:[log in to unmask]]
Sent: Sunday, August 18, 2002 5:20 AM
To: [log in to unmask]
Subject: Low protein trial in DM - therapy or prognosis?
Greetings from sunny East Anglia, UK. Please find below my thoughts on your
posting on the low protein trial.
The Study in question:
Hansen HP, Tauber-Lassen E, Jensen BR, Parving HH.Effect of dietary protein
restriction on prognosis in patients with diabetic nephropathy.Kidney Int
I have briefly summarised the design and findings.
Aim: The purpose of this study was to determine the effect of dietary
protein restriction on survival and progression to ESRD in diabetic
Design: Randomised controlled trial
Intervention: Low-protein diet (0.6 g/kg/day) compared with a usual-protein
Main outcome: The main outcome measures were decline in GFR and development
of ESRD or death.
Main results: The mean declines in GFR were 3.9 mL/min/year (2.7 to 5.2) in
the usual-protein diet group and 3.8 (2.8 to 4.8) in the low-protein diet
group. (As you can see the CI overlaps) ESRD or death occurred in 27% of
patients on a usual-protein diet as compared with 10% on a low-protein diet.
Conclusion: Moderate dietary protein restriction improves prognosis in type
1 diabetic patients with progressive diabetic nephropathy.
We can calculate the NNT to prevent the development of ESRD or death, but
the abstract does not provide the number of patients in the two arms.
As this is a classical randomised trial comparing two modalities of
treatment I sggest that you use the treatment worksheet for critical
appraisal. This will help you (to some extent) to decide whether the results
of this study are valid.
You might find most of the questions for a therapy article will be
appropriate to this study. Please see
http://www.med.ualberta.ca/ebm/ebm.htm (see under therapy/prevention)
These are my impressions. Others might have differing views and you might
hear from them. I hope you find the above of some use/help in your
discussions in the unit. Good luck with the critical appraisal.
Cheers & regards,
Specialist Registrar in PHM & Honorary Clinical Lecturer,
Suffolk Public Health Network & University of Cambridge,
PO Box 170, St.Clement's Hospital,
Ipswich IP1 4LA, UK.
Tel: 00 44 1473 329 570
Fax: 00 44 1473 329 090
"For an excellent review of the current medical literature, go to Journals
Scan www.uaeu.ac.ae/jscan/" - BMJ 3rd June 2000, Netlines
Chat with friends online, try MSN Messenger: http://messenger.msn.com