I have not been following this discussion too closely -- apologies in advance for redundancy and/or irrelevancy.

There is nothing wrong in testing for the "overall" effect of an intervention, even if the effect is merely a placebo effect. What matters is that the effect is clinically important. Why discard a treatment which effects a meaningful change by ways other than DIRECT drug-receptor interaction? All we should care about is that intervention should work. if a placebo effect of naturopathy is specific to naturopathy and is clinically meaningful then its no less important than any direct pharmacological effect of it. Having a placebo is a deliberate attempt to ignore other ways of effecting an outcome because we are pharmacologically primed from med schools.  

Pragmatic trials, some reason, should not be placebo controlled nor blinded.

CVS risk score is an outcome that does not invalidate the study ... it just decreases our confidence for decision-making compared to direct risk scores.  

Attrition bias of large magnitude is a problem -- ITT does not take care of it, neither does per protocol nor "as treated" analysis. Some recommend using instrumental variables or g-estimation to adjust for resultant confounding. Multiple imputations do not give "A" true estimate, at best some idea about the fragility of results.

In brief, what I have learnt of it, I don't see the study any more problematic than the routine drug trials I evaluate from.  But then I must admit, I have not personally read the paper in detail. Will for sure and perhaps have a Rounds on it in our centre. Must also admit, I am just a student of EBM, may be even a "newbie" compared to others; and that have previously collaborated with the lead author on one project a couple of years or so back.


m

(Ottawa)

________________________________

From: Evidence based health (EBH) on behalf of k.hopayian
Sent: Fri 17/05/2013 4:45 PM
To: [log in to unmask]
Subject: Re: why did CMAJ publish this study?


Hi Steve,
Nice to clear the air! However, having worked with Michael Power and having read his contributions here and elsewhere, I can vouch for the fact that he is as critical of poor orthodox medical research as he is of alternative. Please give him credit for that.

I  agree with you that the strongest criticisms of this study are to do with the design rather than the analysis. Where I disagree  is the repeated statement that the researchers used a surrogate measure. We calculate CVS risk with models derived from cohort studies. This trial showed that the exposed group had a lower CVS risk as defined by one such model. (Is this not what most practitioners in primary prevention do?). Had the authors claimed that naturopathy reduced CVS rates that would be a different matter.

Another thing to add to your list is one already mentioned, that the exposure was in fact not very different in content to that of the usual care so it is hard to see how the benefits can be attributed to naturopathy. Now a proper trial of naturopathy would compare usual care + 7 scheduled visits plus placebo (a practitioner who is not a naturopathy but using similar language) versus usual care + 7 scheduled visits with a naturopath. Why was there no such placebo?

Dr Kev (Kevork) Hopayian, MD FRCGP
General Practitioner, Leiston, Suffolk
Hon Sen Lecturer, Norwich Medical School, University of East Anglia
Primary Care Tutor, East Suffolk
RCGP Clinical Skills Assessment examiner

http://www.angliangp.org.uk/

On 16 May 2013, at 21:49, "Steve Simon, P.Mean Consulting" <[log in to unmask]> wrote:


One problem with critical appraisal is that we are only critical when we
already disagree with the conclusion. If there's a result we don't like,
we proponents of Evidence-Based Medicine become some of the harshest and
hardest to please people. I'm afraid we may have an example here, as evidenced by Michael Power's comments.

I am critiquing Dr. Power's comments, not out of any love for
Naturopathy, but rather because it illustrates two important points for
Evidence Based Medicine.

First, allegations of fraud or even insinuations of fraud have no place
in the critical appraisal of a journal article. Fraud exists, but there
is no tool in the Evidence-Based Medicine tool kit for ferreting it out.

Second, arguing that a particular data analysis choice was wrong is a poor way to conduct a critical appraisal. Most medically trained people are unfairly mistrustful of legitimate statistical methodologies. Also, the emphasis of critical appraisal should be mostly on how the data was collected. If you collect the wrong data, it doesn't matter what analysis you choose. Furthermore, if you collect the data well, almost all reasonable analyses will tell you pretty much the same thing.

I also want to highlight Dr. Power's comments because this hypercriticism is a bigger problem for most proponents of Naturopathy and other forms of alternative medicine. There's a lot of mistrust of published research that prevents proponents of alternative medicine from weeding out the worst and most dangerous aspects of their practices. They take legitimate concerns about financial conflicts of interest by drug companies as evidence that all studies of pharmaceutical interventions are invalid. They disregard serious and carefully run studies that show negative results for alternative medicine by applying criticisms that are never considered for the positive studies.



Randomization was conducted by the Canadian College of Naturopathic
Medicine.



It's possible that there was some tampering with the randomization list,
but in almost every study I am familiar with, the randomization is
"controlled" by someone who may be tempted to commit fraud. We let drug
companies, for example, control the randomization of their clinical
trials.

There's a legitimate concern that this comment fails to mention directly. The authors should have used concealed allocation. So this is a weakness of the study.



Participants were selected on the basis of higher ratios of total
cholesterol to HDL cholesterol - because cholesterol measurements
are quite variable, and selection seems to have been done on only
one measurement, this would have introduced a risk of bias from
"regression to the mean".



As someone else has already noted, regression to the mean is not a
possible source of bias.



"The naturopathic doctors collected all biometric and validated
questionnaire measures", and were not blinded to study group.



This is an odd comment, because Dr. Power misses the even greater threat to validity, the failure to blind the patients to treatment status.



Tables 1 (baseline) and 2 (results) are not comparable. First, table
1 shows data plus/minus standard deviations, while table 2 shows data
plus/minus standard errors of the means.



This is not a source of bias. Should they have been consistent and
always reported a standard deviation? I would say yes, but quite
honestly many papers follow this format. Certainly the tables are
labelled clearly enough.

My complaint is different. The authors should have computed a Number Needed to Treat in Table 2. That is far more important than whether a number is a standard deviation or a standard error.



Secondly, and most problematically, table 1 shows real data, i.e.
data with real numerators and denominators, while table 2 shows
imagined, or at least engineered data - the real numbers have been
adjusted for baseline measure of outcome variables, and missing data
have been created by "a multiple imputation".



"Imagined data"? "Engineered data"?  What the authors did was to use "repeated-measures analysis of covariance in a mixed model by including the baseline value as a covariate for the continuous data and a generalized estimating equations approach for the binary data." This is a rather technical detail, but it falls clearly in the realm of standard statistical practice. If you do a PubMed search, for example for "generalized estimating equations" you will find thousands of publications, such as

Rogatko A, Babb JS, Wang H, Slifker MJ, Hudes GR. Patient
characteristics compete with dose as predictors of acute treatment
toxicity in early phase clinical trials. Clin. Cancer Res.
2004;10(14):4645-4651. doi:10.1158/1078-0432.CCR-03-0535.
http://clincancerres.aacrjournals.org/content/10/14/4645.long

Go to Amazon and you will find several books with the title "Generalized
Estimating Equations" written by prominent statisticians.

Multiple imputation is also a well accepted statistical methodology. It
sounds bad. To impute an unobserved value is surely a bad thing, our
intuition tell us. Well, our intuition is wrong. The imputation is done
using well established statistical methodologies.

It's easy enough to establish the reasonableness of multiple imputation. Take a data set that has no dropouts. Randomly introduce some dropouts and then apply multiple imputation. What you will find is that the multiply imputed results are consistent with the results with no dropouts. The confidence intervals are wider, of course, as they should be, but the method introduces no systematic bias, under most reasonable scenarios.

There are hundreds of references to imputation on PubMed, such as

Vinnard C, Wileyto EP, Bisson GP, Winston CA. First Use of Multiple
Imputation with the National Tuberculosis Surveillance System. Epidemiol
Res Int. 2012;2013. doi:10.1155/2013/875234.
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3645492/

Again there are several books at Amazon on this topic published by
prominent statisticians. It's also worth noting that the alternative to
multiple imputation is simply ignoring the missing data and this
effectively imputes a value as well, but a value that presumes that
people with missing values are no different than people without missing
values (technically, the term for this is "missing completely at random").

As a statistician, I find it disheartening that use of well-established
statistical methodologies are considered weaknesses of a study.



This data engineering is particularly problematic given the opacity
of the imputation process and the 30% drop out rate. With a 30% drop
out rate the missing data rate would be > 30%. I wonder why a table 3
with real outcome data was not published? If it had been, we would be
able to see whethere or not the data engineering had created
statistically significant results.



Now, I suspect that most people reading this list consider all advanced
statistical methods as being opaque. That's why I get paid more than the
minimum wage when I consult on statistical issues. But, really, this
comment could be applied to almost any research publication.

The 30% drop out rate is more troublesome, of course, and does represent
a legitimate criticism of the study.

As far as publishing a table with "real outcome data" this is not done
often enough, in my opinion. The correct term for this is "crude
estimate" because that's what it is. There is value in seeing this
number. If there is a discrepancy, however, between the crude estimate
and the adjusted estimate, anyone who trusts statistics as an
established and carefully tested methodology would prefer the adjusted
estimate.



This seems to be the first trial of its kind - and we know that
first publications often turn out to have results that are more
extreme than subsequent trials.



This is an excellent comment. I agree 100%.



The response to these issues, especially the data engineering,
should have been to adjust the confidence intervals (i.e. widen
them). But I can only speculate that the reason this was not done is
that all significant differences would have evaporated faster than
n-butyl glycol.



Given that the process is opaque, you have no idea whether the
confidence intervals have been appropriately widened. For all you know,
the confidence intervals might be too wide! So why all this mistrust? Do
you have an equal amount of mistrust when a study in Oncology (see the
first reference above) uses generalized estimating equations? Do you
have an equal amount of mistrust when a study in Tuberculosis uses
multiple imputation?

Maybe the field of Naturopathy is rife with fraud, but I have seen no
empirical evidence of this. Other fields in alternative medicine do have
problems, of course.

But an important issue here is that fraud is almost never captured
during a critical appraisal by an outside expert. It is caught by a
whistleblower on the inside or it is caught by a formal audit. For the
average person, there is no way that you can look at a publication and
tell it is fraudulent.

So, assessment of potential for fraud do not belong in the critical
appraisal step of evidence-based medicine. This is a bitter pill to
swallow (pun intended), as we know of many prominent cases of serious
research fraud. But none of them, as far as I know, were discovered by
someone like Michael Power or me.

Fraud is perhaps too strong a word. Perhaps Dr. Power was implying
something more subtle, such as running ten different analyses and
reporting the one with the smallest p-value. Or choosing a statistical
approach which is known to exaggerate the differences between groups.

Given the opacity of most statistical analyses, however, you have no
serious option here. If you mistrust generalized estimating equations
because they could be manipulated to produce bogus findings, well that
would potentially be true for anything more complex than a t-test. And
to be critical would require (if you are consistent) a rejection of
almost all currently published research.

There's an even more fundamental problem here. Dr. Power makes a mistake
that is fairly common in critical appraisal. He is focusing on how the
data was analyzed more than on how it was collected. Critical appraisal,
must look at research design issues first. Debating the merits of the
log rank test versus Cox regression is rarely helpful in my opinion.

Furthermore, there is a wealth to criticize in this article, but none of
it relates to the data analysis per se.

First, there is the issue of blinding.

Second, the intervention is highly heterogenous and poorly controlled.

Third, the dropout rate is high.

Fourth, the authors relied on a surrogate outcome.

Fifth, the treatment group got more attention than the control group.

Sixth, there are a large number of exclusions prior to randomization. Of
the 1125 patients screened, only 246 made it to randomization.

Seventh, there was no concealed allocation (e.g., no sequentially
numbered opaque envelopes).

Eighth, there was no discussion of whether the changes were clinically
important (e.g., no NNTs).

The strengths of the study are the use of randomization, intent to treat
analysis, and a reasonably long follow-up time (though two years would
have been better). In spite of Dr. Power's comments, I would argue that
the thoroughness of the statistical analysis, especially the use of
multiple imputation is another strength.

Although the weaknesses are troublesome, some (e.g., blinding) are a thorn in the side of many research studies. Almost all device trials, for example, are unblinded. Others (high dropout rate) are not all that
uncommon in other research areas. A 30% dropout rate is too high, but the standard that I use (no more than 10% dropouts) is almost never met in any long term trial.

All in all, I would call this a good study, but not a definitive study. Eight weaknesses and three or four strengths is actually better than average, in my experience. None of the weaknesses is so serious as to invalidate the entire study.

I'm not rushing out to visit a Naturopath on the basis of this study, but it is an interesting finding and indicates that additional research in this area might be helpful.

There's a lot more to debate here, but I've gone on for far too long.

Steve Simon, [log in to unmask], Standard Disclaimer.
Sign up for the Monthly Mean, the newsletter that
dares to call itself average at www.pmean.com/news




--------------------
Confidentiality Statement - The contents of this e-mail, including its attachment, are intended for the exclusive use of the recipient and may contain confidential or privileged information.  If you are not the intended recipient, you are strictly prohibited from reading, using, disclosing, copying, or distributing this e-mail or any of its contents.  If you received this e-mail in error, please notify the sender by reply e-mail immediately or the Privacy Office ([log in to unmask] ) and permanently delete this e-mail and its attachments, along with any copies thereof.  Thank you.

Avis de confidentialité – Ce courriel, y compris ses pièces jointes, s’adresse au destinataire uniquement et pourrait contenir des renseignements confidentiels. Si vous n’êtes pas le bon destinataire, il est strictement interdit de lire, d’utiliser, de divulguer, de copier ou de diffuser ce courriel ou son contenu, en partie ou en entier. Si vous avez reçu ce courriel par erreur, veuillez en informer immédiatement l’expéditeur ou le bureau de la Protection des renseignements personnels ([log in to unmask]), puis effacez le courriel ainsi que les pièces jointes et toute autre copie. Merci.
--------------------
<winmail.dat>