 |
 |
Dear Kev and all, Of course additional research questions could control for explanatory aspects of intervention and these are extremely important but are they not a next step? What is wrong with doing a preliminary pragmatic trial to answer the question does this work compared to placebo or standardized regular care? People are going into effects that can be valid like supplements and prior relevant research but they were not part of this trial. Why are we separating medicine from CAM and seemingly judging it by a harsher standard since there are plenty of trials in traditional medicine that are much worse, have big names attached and no one seems to care? None of it is evidence unless it is supported by valid research and just because it is attached to the practice of medicine does it get a free pass and if it is considered CAM and it does make the grade why not accept it as a valid intervention? In medicine a simple pragmatic does it seem to work trial would not be enough to take it from evidence into practice we would require future explanatory rationale and data but we also would not expect this all in one preliminary trial paper. I wonder if there was no name of prominence in the field on this paper would anyone care? Has anyone contacted the authors to let them know their paper is on the chopping block so they can defend it and would you be as harsh face to face as you are in discussion? I agree separating out the important factors and testing them seems the logical next step. "As for pragmatic trials, of course they have a place but there are potential reporting biases in CAM, including nocebo and frustrebo effects" Why single out CAM and not apply to all interventions? We should not lower the standards of proof or they are no longer standards they are bias. The contact idea is interesting and seldom researched well but could you be assuming it is the contact because of your belief system. Is it infallible? What are the standards of proof to serve as a guide for our belief systems? Can you imagine how this is for the public who are with no institutional access, have real and not theoretical needs and are aggressively marketed. They are then told they are expected to be 'shared decision makers?' Patients are often given surface advice that does not answer their questions so yes contact is important but it is context dependent because expertise is context dependent so this is a separate study don't you think? Another consideration is how cost effective is substitution in provider if it does not meet the needs, this may provide a short term reduction but a long term escalation. I agree with your well written paper but my assertion is that it would be best applied to all interventions. CAM should not be singled out. It is important to look realistically at what can be accomplished in one paper. I am a learner here and it could be that I am missing the obvious. The thing is that I am likely not alone. COI: I have unconscious bias, deal with unexpected emotional pushback and on occasion commit logical fallacies, in short I am a human and there are lots like me. Figuring out how to enlighten us as we are and maintain relationship could change the practice of medicine and advance science. Best, Amy From: "k.hopayian" <[log in to unmask]> Reply-To: "k.hopayian" <[log in to unmask]> Date: Saturday, May 18, 2013 8:56 AM To: <[log in to unmask]> Subject: Re: why did CMAJ publish this study? Hi M, First, let me clarify what I said before I examine your opening comment. I did not say that it is wrong to test for an overall effect. I said that if the effect is due to naturopathy then those components that are crucial for naturopathy should be tested against a placebo. You are still testing the overall effect as well as the crucial effect. Now let us look at this opening statement: ³There is nothing wrong in testing for the "overall" effect of an intervention, even if the effect is merely a placebo effect. What matters is that the effect is clinically important.² I wonder what would happen to progress if we did not look at why things have effects. What would happen in oncology? ³It doesn¹t matter which of these anticancer drugs cured the patient, what matters is the overall cocktail². Or in rheumatology? ³It doesn¹t matter what we inject in the joint, just so long as it works². In the case of the naturopathy trial, there was an important bias in favour of naturopathy: increased contact with a therapist. Why should we not control for that with a placebo consisting of contact but no naturopathy?Should we lower the standards of proof for complementary and alternative medicine. Talking seriously, if contact is the important factor, why not employ lesser trained practitioners rather than expensive naturopaths. You would surely expect cost-effective care in orthodox medicine (for example, using podiatrists instead of orthopaedice surgeons where appropriate) why not for CAM? As for pragmatic trials, of course they have a place but there are potential reporting biases in CAM, including nocebo and frustrebo effects (a discussion can be found here Power M, Hopayian K. Exposing the evidence gap for complementary and alternative medicine to be integrated into science-based medicine. J R Soc Med. 2011;104(4):155-161. <https://ueaeprints.uea.ac.uk/34960/1/Power_%26_Hopayian_JRSM_2011.pdf> Kev On 17 May 2013, at 23:16, "Ansari, Mohammed" <[log in to unmask]> wrote: > I have not been following this discussion too closely -- apologies in advance > for redundancy and/or irrelevancy. > > There is nothing wrong in testing for the "overall" effect of an intervention, > even if the effect is merely a placebo effect. What matters is that the effect > is clinically important. Why discard a treatment which effects a meaningful > change by ways other than DIRECT drug-receptor interaction? All we should care > about is that intervention should work. if a placebo effect of naturopathy is > specific to naturopathy and is clinically meaningful then its no less > important than any direct pharmacological effect of it. Having a placebo is a > deliberate attempt to ignore other ways of effecting an outcome because we are > pharmacologically primed from med schools. > > Pragmatic trials, some reason, should not be placebo controlled nor blinded. > > CVS risk score is an outcome that does not invalidate the study ... it just > decreases our confidence for decision-making compared to direct risk scores. > > Attrition bias of large magnitude is a problem -- ITT does not take care of > it, neither does per protocol nor "as treated" analysis. Some recommend using > instrumental variables or g-estimation to adjust for resultant confounding. > Multiple imputations do not give "A" true estimate, at best some idea about > the fragility of results. > > In brief, what I have learnt of it, I don't see the study any more problematic > than the routine drug trials I evaluate from. But then I must admit, I have > not personally read the paper in detail. Will for sure and perhaps have a > Rounds on it in our centre. Must also admit, I am just a student of EBM, may > be even a "newbie" compared to others; and that have previously collaborated > with the lead author on one project a couple of years or so back. > > > m > > (Ottawa) > > ________________________________ > > From: Evidence based health (EBH) on behalf of k.hopayian > Sent: Fri 17/05/2013 4:45 PM > To: [log in to unmask] > Subject: Re: why did CMAJ publish this study? > > > Hi Steve, > Nice to clear the air! However, having worked with Michael Power and having > read his contributions here and elsewhere, I can vouch for the fact that he is > as critical of poor orthodox medical research as he is of alternative. Please > give him credit for that. > > I agree with you that the strongest criticisms of this study are to do with > the design rather than the analysis. Where I disagree is the repeated > statement that the researchers used a surrogate measure. We calculate CVS risk > with models derived from cohort studies. This trial showed that the exposed > group had a lower CVS risk as defined by one such model. (Is this not what > most practitioners in primary prevention do?). Had the authors claimed that > naturopathy reduced CVS rates that would be a different matter. > > Another thing to add to your list is one already mentioned, that the exposure > was in fact not very different in content to that of the usual care so it is > hard to see how the benefits can be attributed to naturopathy. Now a proper > trial of naturopathy would compare usual care + 7 scheduled visits plus > placebo (a practitioner who is not a naturopathy but using similar language) > versus usual care + 7 scheduled visits with a naturopath. Why was there no > such placebo? > > Dr Kev (Kevork) Hopayian, MD FRCGP > General Practitioner, Leiston, Suffolk > Hon Sen Lecturer, Norwich Medical School, University of East Anglia > Primary Care Tutor, East Suffolk > RCGP Clinical Skills Assessment examiner > > http://www.angliangp.org.uk/ > > On 16 May 2013, at 21:49, "Steve Simon, P.Mean Consulting" <[log in to unmask]> > wrote: > > > One problem with critical appraisal is that we are only critical when we > already disagree with the conclusion. If there's a result we don't like, > we proponents of Evidence-Based Medicine become some of the harshest and > hardest to please people. I'm afraid we may have an example here, as evidenced > by Michael Power's comments. > > I am critiquing Dr. Power's comments, not out of any love for > Naturopathy, but rather because it illustrates two important points for > Evidence Based Medicine. > > First, allegations of fraud or even insinuations of fraud have no place > in the critical appraisal of a journal article. Fraud exists, but there > is no tool in the Evidence-Based Medicine tool kit for ferreting it out. > > Second, arguing that a particular data analysis choice was wrong is a poor way > to conduct a critical appraisal. Most medically trained people are unfairly > mistrustful of legitimate statistical methodologies. Also, the emphasis of > critical appraisal should be mostly on how the data was collected. If you > collect the wrong data, it doesn't matter what analysis you choose. > Furthermore, if you collect the data well, almost all reasonable analyses will > tell you pretty much the same thing. > > I also want to highlight Dr. Power's comments because this hypercriticism is a > bigger problem for most proponents of Naturopathy and other forms of > alternative medicine. There's a lot of mistrust of published research that > prevents proponents of alternative medicine from weeding out the worst and > most dangerous aspects of their practices. They take legitimate concerns about > financial conflicts of interest by drug companies as evidence that all studies > of pharmaceutical interventions are invalid. They disregard serious and > carefully run studies that show negative results for alternative medicine by > applying criticisms that are never considered for the positive studies. > > > > Randomization was conducted by the Canadian College of Naturopathic > Medicine. > > > > It's possible that there was some tampering with the randomization list, > but in almost every study I am familiar with, the randomization is > "controlled" by someone who may be tempted to commit fraud. We let drug > companies, for example, control the randomization of their clinical > trials. > > There's a legitimate concern that this comment fails to mention directly. The > authors should have used concealed allocation. So this is a weakness of the > study. > > > > Participants were selected on the basis of higher ratios of total > cholesterol to HDL cholesterol - because cholesterol measurements > are quite variable, and selection seems to have been done on only > one measurement, this would have introduced a risk of bias from > "regression to the mean". > > > > As someone else has already noted, regression to the mean is not a > possible source of bias. > > > > "The naturopathic doctors collected all biometric and validated > questionnaire measures", and were not blinded to study group. > > > > This is an odd comment, because Dr. Power misses the even greater threat to > validity, the failure to blind the patients to treatment status. > > > > Tables 1 (baseline) and 2 (results) are not comparable. First, table > 1 shows data plus/minus standard deviations, while table 2 shows data > plus/minus standard errors of the means. > > > > This is not a source of bias. Should they have been consistent and > always reported a standard deviation? I would say yes, but quite > honestly many papers follow this format. Certainly the tables are > labelled clearly enough. > > My complaint is different. The authors should have computed a Number Needed to > Treat in Table 2. That is far more important than whether a number is a > standard deviation or a standard error. > > > > Secondly, and most problematically, table 1 shows real data, i.e. > data with real numerators and denominators, while table 2 shows > imagined, or at least engineered data - the real numbers have been > adjusted for baseline measure of outcome variables, and missing data > have been created by "a multiple imputation". > > > > "Imagined data"? "Engineered data"? What the authors did was to use > "repeated-measures analysis of covariance in a mixed model by including the > baseline value as a covariate for the continuous data and a generalized > estimating equations approach for the binary data." This is a rather technical > detail, but it falls clearly in the realm of standard statistical practice. If > you do a PubMed search, for example for "generalized estimating equations" you > will find thousands of publications, such as > > Rogatko A, Babb JS, Wang H, Slifker MJ, Hudes GR. Patient > characteristics compete with dose as predictors of acute treatment > toxicity in early phase clinical trials. Clin. Cancer Res. > 2004;10(14):4645-4651. doi:10.1158/1078-0432.CCR-03-0535. > http://clincancerres.aacrjournals.org/content/10/14/4645.long > > Go to Amazon and you will find several books with the title "Generalized > Estimating Equations" written by prominent statisticians. > > Multiple imputation is also a well accepted statistical methodology. It > sounds bad. To impute an unobserved value is surely a bad thing, our > intuition tell us. Well, our intuition is wrong. The imputation is done > using well established statistical methodologies. > > It's easy enough to establish the reasonableness of multiple imputation. Take > a data set that has no dropouts. Randomly introduce some dropouts and then > apply multiple imputation. What you will find is that the multiply imputed > results are consistent with the results with no dropouts. The confidence > intervals are wider, of course, as they should be, but the method introduces > no systematic bias, under most reasonable scenarios. > > There are hundreds of references to imputation on PubMed, such as > > Vinnard C, Wileyto EP, Bisson GP, Winston CA. First Use of Multiple > Imputation with the National Tuberculosis Surveillance System. Epidemiol > Res Int. 2012;2013. doi:10.1155/2013/875234. > http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3645492/ > > Again there are several books at Amazon on this topic published by > prominent statisticians. It's also worth noting that the alternative to > multiple imputation is simply ignoring the missing data and this > effectively imputes a value as well, but a value that presumes that > people with missing values are no different than people without missing > values (technically, the term for this is "missing completely at random"). > > As a statistician, I find it disheartening that use of well-established > statistical methodologies are considered weaknesses of a study. > > > > This data engineering is particularly problematic given the opacity > of the imputation process and the 30% drop out rate. With a 30% drop > out rate the missing data rate would be > 30%. I wonder why a table 3 > with real outcome data was not published? If it had been, we would be > able to see whethere or not the data engineering had created > statistically significant results. > > > > Now, I suspect that most people reading this list consider all advanced > statistical methods as being opaque. That's why I get paid more than the > minimum wage when I consult on statistical issues. But, really, this > comment could be applied to almost any research publication. > > The 30% drop out rate is more troublesome, of course, and does represent > a legitimate criticism of the study. > > As far as publishing a table with "real outcome data" this is not done > often enough, in my opinion. The correct term for this is "crude > estimate" because that's what it is. There is value in seeing this > number. If there is a discrepancy, however, between the crude estimate > and the adjusted estimate, anyone who trusts statistics as an > established and carefully tested methodology would prefer the adjusted > estimate. > > > > This seems to be the first trial of its kind - and we know that > first publications often turn out to have results that are more > extreme than subsequent trials. > > > > This is an excellent comment. I agree 100%. > > > > The response to these issues, especially the data engineering, > should have been to adjust the confidence intervals (i.e. widen > them). But I can only speculate that the reason this was not done is > that all significant differences would have evaporated faster than > n-butyl glycol. > > > > Given that the process is opaque, you have no idea whether the > confidence intervals have been appropriately widened. For all you know, > the confidence intervals might be too wide! So why all this mistrust? Do > you have an equal amount of mistrust when a study in Oncology (see the > first reference above) uses generalized estimating equations? Do you > have an equal amount of mistrust when a study in Tuberculosis uses > multiple imputation? > > Maybe the field of Naturopathy is rife with fraud, but I have seen no > empirical evidence of this. Other fields in alternative medicine do have > problems, of course. > > But an important issue here is that fraud is almost never captured > during a critical appraisal by an outside expert. It is caught by a > whistleblower on the inside or it is caught by a formal audit. For the > average person, there is no way that you can look at a publication and > tell it is fraudulent. > > So, assessment of potential for fraud do not belong in the critical > appraisal step of evidence-based medicine. This is a bitter pill to > swallow (pun intended), as we know of many prominent cases of serious > research fraud. But none of them, as far as I know, were discovered by > someone like Michael Power or me. > > Fraud is perhaps too strong a word. Perhaps Dr. Power was implying > something more subtle, such as running ten different analyses and > reporting the one with the smallest p-value. Or choosing a statistical > approach which is known to exaggerate the differences between groups. > > Given the opacity of most statistical analyses, however, you have no > serious option here. If you mistrust generalized estimating equations > because they could be manipulated to produce bogus findings, well that > would potentially be true for anything more complex than a t-test. And > to be critical would require (if you are consistent) a rejection of > almost all currently published research. > > There's an even more fundamental problem here. Dr. Power makes a mistake > that is fairly common in critical appraisal. He is focusing on how the > data was analyzed more than on how it was collected. Critical appraisal, > must look at research design issues first. Debating the merits of the > log rank test versus Cox regression is rarely helpful in my opinion. > > Furthermore, there is a wealth to criticize in this article, but none of > it relates to the data analysis per se. > > First, there is the issue of blinding. > > Second, the intervention is highly heterogenous and poorly controlled. > > Third, the dropout rate is high. > > Fourth, the authors relied on a surrogate outcome. > > Fifth, the treatment group got more attention than the control group. > > Sixth, there are a large number of exclusions prior to randomization. Of > the 1125 patients screened, only 246 made it to randomization. > > Seventh, there was no concealed allocation (e.g., no sequentially > numbered opaque envelopes). > > Eighth, there was no discussion of whether the changes were clinically > important (e.g., no NNTs). > > The strengths of the study are the use of randomization, intent to treat > analysis, and a reasonably long follow-up time (though two years would > have been better). In spite of Dr. Power's comments, I would argue that > the thoroughness of the statistical analysis, especially the use of > multiple imputation is another strength. > > Although the weaknesses are troublesome, some (e.g., blinding) are a thorn in > the side of many research studies. Almost all device trials, for example, are > unblinded. Others (high dropout rate) are not all that > uncommon in other research areas. A 30% dropout rate is too high, but the > standard that I use (no more than 10% dropouts) is almost never met in any > long term trial. > > All in all, I would call this a good study, but not a definitive study. Eight > weaknesses and three or four strengths is actually better than average, in my > experience. None of the weaknesses is so serious as to invalidate the entire > study. > > I'm not rushing out to visit a Naturopath on the basis of this study, but it > is an interesting finding and indicates that additional research in this area > might be helpful. > > There's a lot more to debate here, but I've gone on for far too long. > > Steve Simon, [log in to unmask], Standard Disclaimer. > Sign up for the Monthly Mean, the newsletter that > dares to call itself average at www.pmean.com/news > > > > > -------------------- > Confidentiality Statement - The contents of this e-mail, including its > attachment, are intended for the exclusive use of the recipient and may > contain confidential or privileged information. If you are not the intended > recipient, you are strictly prohibited from reading, using, disclosing, > copying, or distributing this e-mail or any of its contents. If you received > this e-mail in error, please notify the sender by reply e-mail immediately or > the Privacy Office ([log in to unmask] ) and permanently delete this > e-mail and its attachments, along with any copies thereof. Thank you. > > Avis de confidentialité Ce courriel, y compris ses pièces jointes, s¹adresse > au destinataire uniquement et pourrait contenir des renseignements > confidentiels. Si vous n¹êtes pas le bon destinataire, il est strictement > interdit de lire, d¹utiliser, de divulguer, de copier ou de diffuser ce > courriel ou son contenu, en partie ou en entier. Si vous avez reçu ce courriel > par erreur, veuillez en informer immédiatement l¹expéditeur ou le bureau de la > Protection des renseignements personnels ([log in to unmask]), > puis effacez le courriel ainsi que les pièces jointes et toute autre copie. > Merci. > -------------------- > <winmail.dat>