One problem with critical appraisal is that we are only critical when we
already disagree with the conclusion. If there's a result we don't like,
we proponents of Evidence-Based Medicine become some of the harshest and
hardest to please people. I'm afraid we may have an example here, as 
evidenced by Michael Power's comments.

I am critiquing Dr. Power's comments, not out of any love for
Naturopathy, but rather because it illustrates two important points for
Evidence Based Medicine.

First, allegations of fraud or even insinuations of fraud have no place
in the critical appraisal of a journal article. Fraud exists, but there
is no tool in the Evidence-Based Medicine tool kit for ferreting it out.

Second, arguing that a particular data analysis choice was wrong is a 
poor way to conduct a critical appraisal. Most medically trained people 
are unfairly mistrustful of legitimate statistical methodologies. Also, 
the emphasis of critical appraisal should be mostly on how the data was 
collected. If you collect the wrong data, it doesn't matter what 
analysis you choose. Furthermore, if you collect the data well, almost 
all reasonable analyses will tell you pretty much the same thing.

I also want to highlight Dr. Power's comments because this 
hypercriticism is a bigger problem for most proponents of Naturopathy 
and other forms of alternative medicine. There's a lot of mistrust of 
published research that prevents proponents of alternative medicine from 
weeding out the worst and most dangerous aspects of their practices. 
They take legitimate concerns about financial conflicts of interest by 
drug companies as evidence that all studies of pharmaceutical 
interventions are invalid. They disregard serious and carefully run 
studies that show negative results for alternative medicine by applying 
criticisms that are never considered for the positive studies.

> Randomization was conducted by the Canadian College of Naturopathic
> Medicine.

It's possible that there was some tampering with the randomization list,
but in almost every study I am familiar with, the randomization is
"controlled" by someone who may be tempted to commit fraud. We let drug
companies, for example, control the randomization of their clinical
trials.

There's a legitimate concern that this comment fails to mention 
directly. The authors should have used concealed allocation. So this is 
a weakness of the study.

> Participants were selected on the basis of higher ratios of total
> cholesterol to HDL cholesterol - because cholesterol measurements
> are quite variable, and selection seems to have been done on only
> one measurement, this would have introduced a risk of bias from
> "regression to the mean".

As someone else has already noted, regression to the mean is not a
possible source of bias.

> "The naturopathic doctors collected all biometric and validated
> questionnaire measures", and were not blinded to study group.

This is an odd comment, because Dr. Power misses the even greater threat 
to validity, the failure to blind the patients to treatment status.

> Tables 1 (baseline) and 2 (results) are not comparable. First, table
> 1 shows data plus/minus standard deviations, while table 2 shows data
> plus/minus standard errors of the means.

This is not a source of bias. Should they have been consistent and
always reported a standard deviation? I would say yes, but quite
honestly many papers follow this format. Certainly the tables are
labelled clearly enough.

My complaint is different. The authors should have computed a Number 
Needed to Treat in Table 2. That is far more important than whether a 
number is a standard deviation or a standard error.

> Secondly, and most problematically, table 1 shows real data, i.e.
> data with real numerators and denominators, while table 2 shows
> imagined, or at least engineered data - the real numbers have been
> adjusted for baseline measure of outcome variables, and missing data
> have been created by "a multiple imputation".

"Imagined data"? "Engineered data"?  What the authors did was to use 
"repeated-measures analysis of covariance in a mixed model by including 
the baseline value as a covariate for the continuous data and a 
generalized estimating equations approach for the binary data." This is 
a rather technical detail, but it falls clearly in the realm of standard 
statistical practice. If you do a PubMed search, for example for 
"generalized estimating equations" you will find thousands of 
publications, such as

Rogatko A, Babb JS, Wang H, Slifker MJ, Hudes GR. Patient
characteristics compete with dose as predictors of acute treatment
toxicity in early phase clinical trials. Clin. Cancer Res.
2004;10(14):4645–4651. doi:10.1158/1078-0432.CCR-03-0535.
http://clincancerres.aacrjournals.org/content/10/14/4645.long

Go to Amazon and you will find several books with the title "Generalized
Estimating Equations" written by prominent statisticians.

Multiple imputation is also a well accepted statistical methodology. It
sounds bad. To impute an unobserved value is surely a bad thing, our
intuition tell us. Well, our intuition is wrong. The imputation is done
using well established statistical methodologies.

It's easy enough to establish the reasonableness of multiple imputation. 
Take a data set that has no dropouts. Randomly introduce some dropouts 
and then apply multiple imputation. What you will find is that the 
multiply imputed results are consistent with the results with no 
dropouts. The confidence intervals are wider, of course, as they should 
be, but the method introduces no systematic bias, under most reasonable 
scenarios.

There are hundreds of references to imputation on PubMed, such as

Vinnard C, Wileyto EP, Bisson GP, Winston CA. First Use of Multiple
Imputation with the National Tuberculosis Surveillance System. Epidemiol
Res Int. 2012;2013. doi:10.1155/2013/875234.
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3645492/

Again there are several books at Amazon on this topic published by
prominent statisticians. It's also worth noting that the alternative to
multiple imputation is simply ignoring the missing data and this
effectively imputes a value as well, but a value that presumes that
people with missing values are no different than people without missing
values (technically, the term for this is "missing completely at random").

As a statistician, I find it disheartening that use of well-established
statistical methodologies are considered weaknesses of a study.

> This data engineering is particularly problematic given the opacity
> of the imputation process and the 30% drop out rate. With a 30% drop
> out rate the missing data rate would be > 30%. I wonder why a table 3
> with real outcome data was not published? If it had been, we would be
> able to see whethere or not the data engineering had created
> statistically significant results.

Now, I suspect that most people reading this list consider all advanced
statistical methods as being opaque. That's why I get paid more than the
minimum wage when I consult on statistical issues. But, really, this
comment could be applied to almost any research publication.

The 30% drop out rate is more troublesome, of course, and does represent
a legitimate criticism of the study.

As far as publishing a table with "real outcome data" this is not done
often enough, in my opinion. The correct term for this is "crude
estimate" because that's what it is. There is value in seeing this
number. If there is a discrepancy, however, between the crude estimate
and the adjusted estimate, anyone who trusts statistics as an
established and carefully tested methodology would prefer the adjusted
estimate.

> This seems to be the first trial of its kind - and we know that
> first publications often turn out to have results that are more
> extreme than subsequent trials.

This is an excellent comment. I agree 100%.

> The response to these issues, especially the data engineering,
> should have been to adjust the confidence intervals (i.e. widen
> them). But I can only speculate that the reason this was not done is
> that all significant differences would have evaporated faster than
> n-butyl glycol.

Given that the process is opaque, you have no idea whether the
confidence intervals have been appropriately widened. For all you know,
the confidence intervals might be too wide! So why all this mistrust? Do
you have an equal amount of mistrust when a study in Oncology (see the
first reference above) uses generalized estimating equations? Do you
have an equal amount of mistrust when a study in Tuberculosis uses
multiple imputation?

Maybe the field of Naturopathy is rife with fraud, but I have seen no
empirical evidence of this. Other fields in alternative medicine do have
problems, of course.

But an important issue here is that fraud is almost never captured
during a critical appraisal by an outside expert. It is caught by a
whistleblower on the inside or it is caught by a formal audit. For the
average person, there is no way that you can look at a publication and
tell it is fraudulent.

So, assessment of potential for fraud do not belong in the critical
appraisal step of evidence-based medicine. This is a bitter pill to
swallow (pun intended), as we know of many prominent cases of serious
research fraud. But none of them, as far as I know, were discovered by
someone like Michael Power or me.

Fraud is perhaps too strong a word. Perhaps Dr. Power was implying
something more subtle, such as running ten different analyses and
reporting the one with the smallest p-value. Or choosing a statistical
approach which is known to exaggerate the differences between groups.

Given the opacity of most statistical analyses, however, you have no
serious option here. If you mistrust generalized estimating equations
because they could be manipulated to produce bogus findings, well that
would potentially be true for anything more complex than a t-test. And
to be critical would require (if you are consistent) a rejection of
almost all currently published research.

There's an even more fundamental problem here. Dr. Power makes a mistake
that is fairly common in critical appraisal. He is focusing on how the
data was analyzed more than on how it was collected. Critical appraisal,
must look at research design issues first. Debating the merits of the
log rank test versus Cox regression is rarely helpful in my opinion.

Furthermore, there is a wealth to criticize in this article, but none of
it relates to the data analysis per se.

First, there is the issue of blinding.

Second, the intervention is highly heterogenous and poorly controlled.

Third, the dropout rate is high.

Fourth, the authors relied on a surrogate outcome.

Fifth, the treatment group got more attention than the control group.

Sixth, there are a large number of exclusions prior to randomization. Of
the 1125 patients screened, only 246 made it to randomization.

Seventh, there was no concealed allocation (e.g., no sequentially
numbered opaque envelopes).

Eighth, there was no discussion of whether the changes were clinically
important (e.g., no NNTs).

The strengths of the study are the use of randomization, intent to treat
analysis, and a reasonably long follow-up time (though two years would
have been better). In spite of Dr. Power's comments, I would argue that
the thoroughness of the statistical analysis, especially the use of
multiple imputation is another strength.

Although the weaknesses are troublesome, some (e.g., blinding) are a 
thorn in the side of many research studies. Almost all device trials, 
for example, are unblinded. Others (high dropout rate) are not all that
uncommon in other research areas. A 30% dropout rate is too high, but 
the standard that I use (no more than 10% dropouts) is almost never met 
in any long term trial.

All in all, I would call this a good study, but not a definitive study. 
Eight weaknesses and three or four strengths is actually better than 
average, in my experience. None of the weaknesses is so serious as to 
invalidate the entire study.

I'm not rushing out to visit a Naturopath on the basis of this study, 
but it is an interesting finding and indicates that additional research 
in this area might be helpful.

There's a lot more to debate here, but I've gone on for far too long.

Steve Simon, [log in to unmask], Standard Disclaimer.
Sign up for the Monthly Mean, the newsletter that
dares to call itself average at www.pmean.com/news