I agree Huw and thank you Senn for your clarifications. I am keeping this stream of communication, it is important to see things through the eyes of those that write research up for the times when standardization in communication fails us. I have seen areas that have contributed to a less than clear understanding because of my own thinking/background and really appreciate this exchange.

Best

Amy

Perhaps the most interesting thing about this discussion is that the same topic of hypothesis testing regarding RCTs is seen from such very different perspectives by people trained in different disciplines!

Huw.

There seem to be a lot of things being discussed in this thread. I have four comments.

1)      The main purpose of the falsificationism paper was to show that there was a fundamental difference in trying to disprove the hypothesis that the data comes from a single distribution (e.g. the drug is a placebo) and trying to disprove the hypothesis that the data come from two distributions (the interventional drug is not the same as the active comparator). The latter case is what you try to do in active control equivalence studies. If you reject the two distribution theory, then you end up with one distribution and the conclusion that the new treatment is equivalent to the old. (This is seen most clearly in bioequivalence studies but it applies elsewhere also.) The point of that paper was to claim that all the statistical fixes in the world do not deal with the fundamental ‘philosophical’ difference between these two cases. Some of the technical issues are currently being debated on Deborah Mayos’ blog. See

http://errorstatistics.com/2014/06/05/stephen-senn-blood-simple-the-complicated-and-controversial-world-of-bioequivalence-guest-post/

and

http://errorstatistics.com/2014/07/31/roger-berger-on-senns-blood-simple-with-a-response-by-s-senn-guest-posts/

2)      95% of (correctly calculated) 95% confidence intervals will contain the true parameter value. This does not mean (however), for example, that 95% of CIs that exclude 0 do so correctly.  One must beware of invalid inversion.

3)The case series method, as pioneered by Farrington and Whittaker

            Farrington CP, Whitaker HJ. Semiparametric analysis of case series data (with discussion). Journal of the Royal Statistical Society Series C-Applied Statistics 2006; 55: 1-28.

and the earlier case cross-over method of Marshall et al

Marshall RJ, Jackson RT. Analysis of case-crossover designs. Statistics in Medicine 1993; 12: 2333-2341.

can be powerful ways of assessing causality using the timing of events.

Even where we have clinical trials, there are occasions where it is clear that we would not accept the results that the pure randomisation analysis would produce. Such a case is given by the trial of TGN1412 in which 6 out of 6 healthy volunteers given TGN1412 had severe reactions and 2 given placebo did not. A Fisher’s exact test of the result does not begin to describe what everybody. This trial and a much larger one where it would be foolish to depart from classical analysis are discussed here

Senn S. Lessons from TGN1412 and TARGET: implications for observational studies and meta-analysis. Pharmaceutical statistics 2008; 7: 294–301.

However, it is not clear to me from this discussion that case-series methodology is being appropriately applied in the example cited.

4)Picking up on an earlier thread, one has a natural prejudice in favour of one’s own work but I think it would be wise to defer judgement on Tamiflu until (at least) the MUGAS analysis reports http://www.mugas.net/mugas/re-analysis-of-clinical-trials/

.

My declaration of interest is here

http://www.senns.demon.co.uk/Declaration_Interest.htm

Stephen

Agree with Huw.

We have to ask our selves where the laws come from- presumably from testing multiple hypotheses over time resulting in what Quine called "web of knowledge". ( I think we may have been confusing here the difference between hypotheses, theories, and laws - admittedly not easy definition to give in a short reply)

Ben

 

PS. BTW, the 95% CI does not say that the true values are between x and y ; instead, the frequency with which this single computed 95% CI contains the true value is either 100% or 0%. 

Sent from my iPad

( please excuse typos & brevity)

On Jul 31, 2014, at 3:15 AM, "Huw Llewelyn [hul2]" <[log in to unmask]> wrote:

Hi Michael

I agree that the null hypothesis is a device to help estimate the probability of replicating a result using multiple readings eg of an RCT. But that RCT will be based on an underlying hypothesis eg that the treatment molecule is able to compete with an endogenous molecule for a receptor and thus improve a patient's symptoms. This interaction would be based partly on the mathematical model representing a general law called 'the law of mass action'.

If the RCT fails to show a replicable difference in symptoms between this molecule and a placebo then the entire reasoning leading up to the RCT, (including the general 'law of mass action') is called into question. Popper appears to say that the whole theory / hypothesis is 'falsified'. I prefer to say that it is thrown into doubt and that the probability of success of a hypothesis using that rationale in future is lower.

It is possible that some error was made eg that the molecule used in the RCT was not manufactured properly and different to the one used in earlier phases of its development. If this supplementary hypothesis is 'verified', and the correct molecule is used in a second RCT which does show a difference which can probably be replicated, then the probability of the original hypothesis / theory will be restored.

What do you think of this?

Best

Huw

---

From: Michael Power <[log in to unmask]>

Sender: "Evidence based health (EBH)" <[log in to unmask]>

Date: Thu, 31 Jul 2014 06:34:44 +0100

To: <[log in to unmask]>

ReplyTo: Michael Power <[log in to unmask]>

Subject: Re: Since when did case series become acceptable to prove efficacy?

 

 

Hi

 

Apologies about but-ing into this conversation so late. But, sometimes philosophy seems to me to make things so complicated that practical understanding disappears.

 

An RCT comparing the effects of treatments A and B does not test any theory analogous to a general law such as F = MA.

 

A clinical trial is simply a measuring tool, a tool for measuring effects

 

The null hypothesis is a way of framing the statistical theory that underpins the measurement of random variation: If the RCT were to be repeated endlessly, the 95% confidence interval(s) it has measured will “capture” the true measured mean 95% of the time.

 

This is about precision, not accuracy. I.E. the 95% CI captures the true measured mean, NOT the true mean.

 

The difference between the true mean and the true measured mean is the bias, or systematic error of the measurement tool, and is a consequence of the accuracy of the measuring instrument.

 

We can measure precision. But we cannot measure bias, we can only estimate it by critical appraisal (not by deduction as Stephen Senn’s 1991 paper would have us believe).

 

So, there are 3 fundamental problems of induction:

 

1.     Trials measure precision, but our heads see the measurements as accurate – this is a psychological problem, which hopefully can be mitigated by education

2.     Trial design (e.g. control, randomization, blinding) can hopefully mitigate the risks of bias.  And, critical appraisal can guestimate the risks of bias. But, because important bias can arise from sources we can’t control or can’t imagine, our mitigations and guestimates leave an uncertain interval of fuzziness around any measurement. Bias can only be measured against a true reference standard, but we are forced to use an artificial “gold standard”, and we know from experience that the gold can turn out to be fool’s gold. Conclusion: we need outside evidence to calibrate the accuracy of measurements of clinical trials (which will have the Bayesians clapping their hands in joy).

3.     We have evidence of mean (measured) effects and guestimates of risks of bias. How do we apply this evidence to the individual in the consultation? This is nowhere near the problem of betting on whether the sun will rise tomorrow, or if E=MC^2 has to be applied to F = MA when calculating the tides or electron orbitals. (Do I hear the Bayesians again?)

 

I hope I have falsified the hope of falsification in RCTs!

 

Michael

 

 

 

From: Evidence based health (EBH) [mailto:[log in to unmask]] On Behalf Of Djulbegovic, Benjamin
Sent: 31 July 2014 00:31
To: [log in to unmask]
Subject: Re: Since when did case series become acceptable to prove efficacy?

 

Hi Huw,

Your e-mail succinctly summarizes the epistemological problems that have been debated in science for  hundred of years going all the way to Aristotle! Indeed, the problem of application of probability calculus to single events is one of the eternal issues. The usually proposed solution is to accept the premise of exchangeability of past with future events- whether this assumption is always acceptable is another issue, but so far, it has served us pretty well.

I look forward to hearing further thoughts from you and others on these issues that are more relevant to EBM that people may appreciate at first blush..

Best

Ben 

Sent from my iPad

( please excuse typos & brevity)

On Jul 30, 2014, at 6:23 PM, "Huw Llewelyn [hul2]" <[log in to unmask]> wrote:

Dear Amy, Ben, Kev, Stephen and all

Thank you Ben for attaching a link to Stephen Senn's paper.  I would be grateful for your comments on the questions that I pose below to try to clarify my understanding.

If I postulate (or ‘hypothesise’) that it will probably rain tomorrow and it does rain then is my ‘rain hypothesis’ verified and is the alternative ‘no rain hypothesis’ falsified?  Do you think that this use of the term ‘hypothesis’, ‘verified’ and ‘falsified’ for a single event is appropriate?

If I postulate that it will probably rain tomorrow, I may look for more facts which make it more probable or less probable that it will rain.  If a null hypothesis that it will be sunny becomes very improbable, I can decide to ‘reject’ the plan of going to the sea-side tomorrow to ‘sun’ myself.  Note that the null hypothesis does not include all the other possibilities e.g. ‘not sunny but dry’.  Do you agree that we can never verify or falsify a statistical hypothesis about an infinitely large population by direct observation but only estimate its probability?

The problem with a scientific hypothesis, theory or diagnosis is that it is not like a single prediction about a statistical null hypothesis about an unobservable infinite population or an easily accessible falsifiable or verifiable hypothesis about tomorrow’s rain.  It is a title to a group of predictions about past, present and future consequences, many of which are interconnected and the probability of many of which cannot easily be estimated directly by studies.  The result of a single RCT on an infinite number of patients is one of these consequences that we can try to predict from a study sample.  Popper appears to say that if one such a consequence becomes improbable, then the overall hypothesis / theory is 'falsified'. However, to my mind the validity of the overall hypothesis / theory becomes less probable and an alternative hypothesis / theory may become more probable.  A resulting decision to reject one course of scientific investigation and to pursue another is another matter.  Popper points out that even if all the consequences do remain probable after testing, we still cannot assume that the overall hypothesis / theory is verified (as I assume there may be other consequences and alternative hypotheses / theories that we have not yet considered).  It seems to me that our imaginary model is a hypothesis when there is an intention to test it against some alternative but a theory if there is no immediate intention to do so.  Do you agree?

Best

Huw

---

From: Evidence based health (EBH) [[log in to unmask]] on behalf of Amy Price [[log in to unmask]]
Sent: 30 July 2014 20:14
To: [log in to unmask]
Subject: Re: Since when did case series become acceptable to prove efficacy?

Thanks Ben and Kev,

 

I found paragraph 2 of Kev’s explanation helpful. I used to not love theory and think it took too much room in the bathwater. After throwing that baby out I have searched for it. I came to the conclusion that theory became difficult when it was overstated and built on as if it was a validated fact without uncertainties  and so it was not the theory that was the issue but the abuse of it.   I have defined in my own mind the term pragmatic to see if something works and at what dosage/intensity etc and explanatory to define why and to improve the working. To me it is a cycle not an either or superiority. The problem comes with expecting more than the design can support.

 

Best

Amy

 

From: Evidence based health (EBH) [mailto:[log in to unmask]] On Behalf Of Djulbegovic, Benjamin
Sent: 30 July 2014 02:37 PM
To: [log in to unmask]
Subject: Re: Since when did case series become acceptable to prove efficacy?

 

Perhaps, this paper to which I alluded before may help clarify some of the issues we are discussing

 

                    Senn SJ. Falsificationism and clinical trials. Stat Med. 1991;10:1679-1692.

 

Stephen used to be active on this group, perhaps he may wish to comment…

 

Best

ben

 

 

From: Evidence based health (EBH) [mailto:[log in to unmask]] On Behalf Of k.hopayian
Sent: Wednesday, July 30, 2014 5:08 AM
To: [log in to unmask]
Subject: Re: Since when did case series become acceptable to prove efficacy?

 

Dear Ben and all,

To those who dislike theory, at least please read the next paragraph,

 

1 This discussion is important because of its practical consequences. Many people outside EBP mistakenly believe that EBP holds that anything less than a clinical trial is poor evidence. So comparing RCTs to the method of basic science that develops theories can give support to these mistaken beliefs. (That is not to say that you, Ben, are mistaken). Observational studies are no less scientific (in the sense of applying statistics, medical knowledge, pharmacology etc) than RCTs, they both use the tools of epidemiology, it is the risk of bias that differs. 

 

2 I would argue that the RCT process has a superficial resemblance to the Popperian method: Null hypothesis - Experiment - Reject/Do not reject null hypothesis; Theory - Prediction - Experiment -Reject/Do not reject theory.

The difference is that theories attempt to explain observations already made and then predict new observations for testing. The null hypothesis does not explain current observation nor predict new observations. It applies theories that do. The use of the word explanatory in the explanatory vs pragmatic variation of trials only adds to the superficial resemblance, an example maybe of what a contemporary of Popper said, that philosophy is the battle against bewitchment by language.

 

No, it isn't fall yet but the Parrottia persica outside my front door proudly displays leaves of varying shades. It was a bare, straggly, ugly piece of bark when I planted it. Five years later, it is an admirable exhibit. 

 

Good luck with the grant application!

 

Kev

 

On 26 Jul 2014, at 15:38, Djulbegovic, Benjamin <[log in to unmask]> wrote:

 

This is “learning ” weekend for me, Kev

Working on the grant (what else is new?) and my 99% of perspiration is on welcome occasions being alleviated by reflection on thoughtful messages and remarks by people like you…(I confess: when I get tired of writing, I log back on my e-mail, and the messages from the EBM folks never stop inspiring me…)

 

In thinking about your latest example, we may be talking about two different things: evidence vs. decision-making, which in clinical trial design paradigm translates into explanatory trials (whose goal is to provide a scientific answer to a research question, typically focusing on the proof of a concept or mechanism etc ;‘Efficacy” question) vs. pragmatic trials (Which treatment of already proven efficacy is better?” ;“effectiveness question). [Several years ago we used regret approach to tackle these issues; see:Hozo I, Schell MJ, Djulbegovic B. Decision-making when data and inferences are not conclusive: risk-benefit and acceptable regret approach. Semin Hematol. Jul 2008;45(3):150-159.]

In terms of application of Popper’s discourse of scientific method to clinical trials, Jeremy Howick informed me that Fisherian hypothesis testing and Popperian falsificationism is identical. Fisher wrote it first, but Popper had not read Fisher (until much later, I believe) at which point Popper acknowledged the similarity… (I slightly edited Jeremy’s words, and if I am not quoting him accurately, I hope he can clarify)

 

Enjoy auburn leaves (is it already fall in England?)

Best

ben

 

 

 

From: Evidence based health (EBH) [mailto:[log in to unmask]] On Behalf Of k.hopayian
Sent: Saturday, July 26, 2014 7:47 AM
To: [log in to unmask]
Subject: Re: Since when did case series become acceptable to prove efficacy?

 

Hi Ben,

 

Short reply: I respectfully disagree that a trial is an example of Popper's scientific method. Popper was concerned with theory. Whether drug A is better than B is not a theory (although confusingly, the negation of that statement is called the null hypothesis) in the sense that the biochemistry and pharmacology around those drugs have theories (such as drug receptors, enzyme action etc).

 

 

Slightly longer reply: Theories make predictions but not all predictions come from theories.

For example, the laws of physics are applied in engineering. Physics and engineering have theories. Engineers may design cars. Two cars may be compared for performance (acceleration, speed, efficiency etc). In comparing them, a manufacturer may start with an idea (our cars are better than yours) but that would hardly count as a theory. [By the way, have car manufacturers ever ever started with a null hypothesis?:-)] The manufacturer's prediction may be proven wrong (and no doubt get buried in company vaults, car manufacturers are so different to pharma, aren't they?) but no rejection of the theory of engineering is necessarily implied. Now if an engineer designed a car whose performance was beyond the boundaries predicted by the laws of thermodynamics, that WOULD falsify theory.

 

It is a lovely sunny weekend here in Suffolk, with many shades of green and auburn leaves, so out I go. I hope your weekend is a good one too.

 

 

 

Dr Kev (Kevork) Hopayian, 

MD FRCGP
General Practitioner, Leiston, Suffolk,

General Practice Trainer, Leiston

Hon Sen Lecturer, Norwich Medical School, University of East Anglia
Primary Care Tutor, East Suffolk

RCGP Clinical Skills Assessment examiner

NHS Senior Appraiser, East Anglia

http://www.angliangp.org

 

On 24 Jul 2014, at 23:40, Djulbegovic, Benjamin <[log in to unmask]> wrote:

 

Kev,

In fact, clinical trial is a classic example of (Popper's) falsificationism paradigm...and does test hypothesis that one drug is better than the other ( Ho: A=B; Ha:A<>B, as postured by classic frequentist statistical approach). Statistical evidence obtained this way connects one phenomena with others  eventually corroborating ( or, rejecting) theories ( group of related principles and laws that were built by testing a number of hypotheses) such as those if beta-blockers effects are consistent with biochemical drug-receptor theory...

Best

Ben

Ps Steven Senn had a wonderful article some years ago about falsificationism in clinical trials...worth reading...

 

Sent from my iPad

( please excuse typos & brevity)

On Jul 24, 2014, at 5:06 PM, "k.hopayian" <[log in to unmask]> wrote:

"The definition of breakthrough is "it costs a packet" "  Now that, I like.. 

 

But I have to disagree on your method of science. Einstein's general relativity theory remains a theory despite the experimental results that concord with the predictions it makes. What these experiments do is fail to falsify the theory, so we stick with it. There are some things it cannot explain, which quantum theory does better, so we continue with two not compatible but very useful models to explain our world.

 

Such models and experiements should not be confused with the method employed in trials. Trials are not designed to test a theory (for example, a trial of betablockers is not testing the theory that there are biochemical receptors). The trial is there to establish which of one or more interventions is superior, if at all. No theory/modle is falsified by such experiments  - although beliefs (some cherished) can be dispelled. I suppose statisticians/epidemiologists have not helped our understanding by using the term hypothesis testing.

 

Kev

 

 

 

Dr Kev (Kevork) Hopayian, 

MD FRCGP
General Practitioner, Leiston, Suffolk,

General Practice Trainer, Leiston

Hon Sen Lecturer, Norwich Medical School, University of East Anglia
Primary Care Tutor, East Suffolk

RCGP Clinical Skills Assessment examiner

NHS Senior Appraiser, East Anglia

http://www.angliangp.org

 

On 24 Jul 2014, at 20:49, Tom Jefferson <[log in to unmask]> wrote:

 

The definition of breakthrough is "it costs a packet" - and Sovaldi fits the picture.

This kind of bullshit is replicated in the EU with so called early assessment to get better, innovative drugs earlier to patients who desperately need them. So the burden of proof is slowly being pushed back to phase IV or beyond which may be observational, subverting Galileo's methods.

What we should always remember is that Einstein's general relativity theory (1915) was a theory and remained a theory until Eddington's natural experiment during the 1919 solar eclipse confirmed that gravitation could deflect starlight as the theory had put forward.

 

The rise of observational data (even non comparative) is an involution, not an evolution. There are many culprits most of them in my profession (I am a physician) and they will be held to account.

Greed and science do not mix.

Nite from Rome.

Tom.

 

On 24 July 2014 17:10, Poses, Roy <[log in to unmask]> wrote:

Thanks, Tom.  Could not agree more.  But it seems like there is little protest about this paradigm shift, your work, of course, excepted. 

Re: "trials are for regulators" - but in the US, the regulators apparently decided they don't need so many trials.  The FDA designated Sovaldi/ sofosbuvir as a "breakthrough" therapy, which apparently allows approvals based on much more limited evidence, although the evidence behind that "breakthrough" designation itself was not clear.

See: http://www.fda.gov/newsevents/newsroom/pressannouncements/ucm377888.htm

 

On Thu, Jul 24, 2014 at 10:45 AM, Tom Jefferson <[log in to unmask]> wrote:

Roy and all evidencers.

The scientific method of Galileo has been subverted before our very eyes.

Galileo observed, described and then produced a hypothesis or theory which he then proceeded to test with an experiment. This model has served us well in the last 400 years with a few exceptions (the already cited penicillin for example).

What we now witness is a fundamental subversion of the order (I am not going to call it a paradigm shift) of things. Observations are fact, case-controls, case series, cohorts (even retrospective and datalinked ones) are being held out as proof. Trials are for regulators, they say.

The origin of all this is complex and partly known. In the Tamiflu story as we began uncovering the extent of reporting bias affecting the clinical trials that had been used to make policy and justify stockpiling, decision makers turned to observational evidence (of universally recognised poor quality) as props for their unchangeable policies.

It is a sad parable of the world we live in.

Best wishes,

Tom.

 

On 24 July 2014 16:33, Valerie King <[log in to unmask]> wrote:

Agree Roy.

 

Don’t think we can assume, based on these case-series in highly selected populations, that the “eradication” rate is >=90% or that SVR is a good surrogate. Also, although the studies were registered with SVR24 as primary outcome the FDA let them give SVR12 as part of the “breakthrough” designation so we don’t even have a particularly good surrogate.

 

Virtually all of the subjects in published studies had very positive treatment prognosis anyway. Over half of subjects had HCV genotype 2 which is the easiest to treat no matter what drug used.  And there is certainly more than a hint in several of the studies of substantial relapse rates after SVR24 achieved (e.g. ~9% in NUTRINO.) I honestly think that most people are listening to the marketing drumbeat on this drug and not reading the papers for themselves. I’d be happy to have these drugs be the breakthrough that most people seem to think that they are, but in my opinion, currently there are a lack of data to support that position.

 

Cheers,

Valerie

 

Valerie J. King, MD, MPH

Professor of Family Medicine, and

Public Health & Preventive Medicine

Director of Research

Center for Evidence-based Policy

Oregon Health & Science University

Mailstop MDYCEBP

Suite 250

3030 SW Moody Ave.

Portland, OR 97201

Voice: 503-494-8694

Fax: 503-494-3807

[log in to unmask]

www.ohsu.edu/policycenter/

Twitter: @drvalking

 

 

 

From: Evidence based health (EBH) [mailto:[log in to unmask]] On Behalf Of Poses, Roy
Sent: Thursday, July 24, 2014 07:08

To: [log in to unmask]
Subject: Re: Since when did case series become acceptable to prove efficacy?

 

 

But it is not clear that SVR is a good surrogate marker.

See: http://www.thelancet.com/journals/lancet/article/PIIS0140-6736%2814%2961025-4/fulltext 

and the Cochrane review it references:

http://www.ncbi.nlm.nih.gov/pubmed/24585509

 

As an aside, given that hep C infection is a chronic problem and bad effects from it occur long after original infection, it is extremely strange that no one has ever thought to do a good RCT with long term follow up to assess clinical outcomes of ANY hepatitis C treatment.

 

On Wed, Jul 23, 2014 at 4:54 PM, Djulbegovic, Benjamin <[log in to unmask]> wrote:

But, if eradication of viral load is greater than 90% ( and we accept viral load as a good surrogate marker), would then single arm study be justified?

Ben 

Sent from my iPhone

(Please excuse typos & brevity)

On Jul 23, 2014, at 4:50 PM, "Poses, Roy" <[log in to unmask]> wrote:

That seems reasonable, but certainly does not apply to this particular clinical situation and article.  

 

On Wed, Jul 23, 2014 at 4:47 PM, Steve Simon, P.Mean Consulting <[log in to unmask]> wrote:

On 7/23/2014 9:39 AM, Poses, Roy wrote:
> I still don't see why case-series without any control groups are now
> regarded as credible ways to evaluate efficacy of therapy???

I cannot comment on this particular example, but in general you can safely dispense with a control group when there is close to 100% morbidity or mortality in that control group. In such a setting, any improvement is painfully obvious and does not need a rigorous design or fancy statistical analysis. Also, it is pretty difficult and probably unethical to randomly assign half your patients to be in a group that has 100% morbidity or mortality.

Steve Simon, [log in to unmask], Standard Disclaimer.
Sign up for the Monthly Mean, the newsletter that
dares to call itself average at www.pmean.com/news

 

--

Dr Tom Jefferson
Medico Chirurgo
GMC # 2527527
www.attentiallebufale.it

-- 
Roy M. Poses MD FACP
President
Foundation for Integrity and Responsibility in Medicine (FIRM)
[log in to unmask]
Clinical Associate Professor of Medicine
Alpert Medical School, Brown University
[log in to unmask]

-- 
Dr Tom Jefferson
Medico Chirurgo
GMC # 2527527
www.attentiallebufale.it