I see this as being related to internal and external validity. The former concerns the strength of the evidence in addressing the question "was the treatment effective in the patients studied". The latter has to do with the much more difficult question " how will it work in practice?". Although I don't completely agree with their analysis, Deaton and Cartwright discuss the distinction usefully here. https://www.sciencedirect.com/science/article/pii/S0277953617307359
See also Senn SJ. Added Values: Controversies concerning randomization and additivity in clinical trials. Statistics in Medicine. 2004;23(24):3729-3753. Which list the following possible question one might seek to answer from a clinical trial
Q1. Was there an effect of treatment in this trial?
Q2. What was the average effect of treatment in this trial?
Q3. Was the treatment effect identical for all patients in the trial?
Q4. What was the effect of treatment for different subgroups of patients?
Q5. What will be the effect of treatment when used more generally (outside of the trial)?
(See P3738)
The two questions Q2 & Q5 are often confused but can lead to surprisingly different analyses. See for example
http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0167167 for an explanation in the context of n-of-1 trials. The distinction is also related to the difference between fixed effects and random effects meta-analyses
For me, real world data can have a useful role in trying to project the effect from a clinical trial to a target population. For instance, a treatment effect might be fairly stable on the log-odds scale. However for clinical decision making the risk difference scale is useful. The solution is not to abandon the log-odds scale for analysis but to predict how it would translate into risk difference in target populations. This could use real world data on background risk to do so. There is a presentation on SlideShare here https://www.slideshare.net/StephenSenn1/real-world-modified
explaining how.
Stephen Senn
Consultant Statistician, Edinburgh
-----Original Message-----
From: Evidence based health (EBH) [mailto:[log in to unmask]] On Behalf Of EVIDENCE-BASED-HEALTH automatic digest system
Sent: 26 August 2018 00:00
To: [log in to unmask]
Subject: EVIDENCE-BASED-HEALTH Digest - 24 Aug 2018 to 25 Aug 2018 (#2018-159)
There are 5 messages totaling 924 lines in this issue.
Topics of the day:
1. Real World Evidence (4)
2. Ioannidis on Nutritional Epidemology
########################################################################
To unsubscribe from the EVIDENCE-BASED-HEALTH list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=EVIDENCE-BASED-HEALTH&A=1
----------------------------------------------------------------------
Date: Sat, 25 Aug 2018 10:09:43 +0100
From: Jon Brassey <[log in to unmask]>
Subject: Real World Evidence
I am increasingly seeing the term 'real world evidence' and see the FDA are exploring it https://www.fda.gov/scienceresearch/specialtopics/realworldevidence/default.htm.
It also featured at the HTAi conference in Vancouver.
I was prompted to post this here after seeing this news story FDA sets out vision to use real-world data in product evaluations <https://www.medtechdive.com/news/fda-sets-out-vision-to-use-real-world-data-in-product-evaluations/530026/>.
Which led to the recent JAMA article Real-World Evidence and Real-World Data for Evaluating Drug Safety and Effectiveness <https://jamanetwork.com/journals/jama/fullarticle/2697359>.
It got me wondering how does the EBM view this development? How does it fit in with the traditional hierarchies of evidence?
Any thoughts?
Best wishes
jon
--
Jon Brassey
Director, Trip Database <http://www.tripdatabase.com> Honorary Fellow at CEBM <http://www.cebm.net>, University of Oxford Creator, Rapid-Reviews.info
########################################################################
To unsubscribe from the EVIDENCE-BASED-HEALTH list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=EVIDENCE-BASED-HEALTH&A=1
------------------------------
Date: Sat, 25 Aug 2018 09:20:52 -0500
From: Bill Cayley <[log in to unmask]>
Subject: Re: Real World Evidence
At first glance, this simply sounds to me like looking at the difference between effectiveness versus efficacy. In which case, it’s not a matter of where the slides and the evidence hierarchy, but what types of outcomes, what types of patients, and what type of generalizability one is looking at.
Bill Cayley, Jr, MD MDiv
[log in to unmask]
[log in to unmask]
http://twitter.com/bcayley
Work: 715.286.2270
Pager: 715.838.7940
Mobile: 715.828.4636
A cheerful heart is good medicine... (Proverbs 17:22)
On Aug 25, 2018, at 4:09 AM, Jon Brassey <[log in to unmask]> wrote:
> I am increasingly seeing the term 'real world evidence' and see the FDA are exploring it https://www.fda.gov/scienceresearch/specialtopics/realworldevidence/default.htm. It also featured at the HTAi conference in Vancouver.
>
> I was prompted to post this here after seeing this news story FDA sets out vision to use real-world data in product evaluations. Which led to the recent JAMA article Real-World Evidence and Real-World Data for Evaluating Drug Safety and Effectiveness.
>
> It got me wondering how does the EBM view this development? How does it fit in with the traditional hierarchies of evidence?
>
> Any thoughts?
>
> Best wishes
>
> jon
>
>
> --
> Jon Brassey
> Director, Trip Database
> Honorary Fellow at CEBM, University of Oxford Creator,
> Rapid-Reviews.info
>
>
> To unsubscribe from the EVIDENCE-BASED-HEALTH list, click the following link:
> https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=EVIDENCE-BASED-HEAL
> TH&A=1
########################################################################
To unsubscribe from the EVIDENCE-BASED-HEALTH list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=EVIDENCE-BASED-HEALTH&A=1
------------------------------
Date: Sat, 25 Aug 2018 18:42:16 +0200
From: Ernesto Barrera <[log in to unmask]>
Subject: Re: Real World Evidence
I think that the RWE happens like the Health Technology Assessment: the pyramid of evidence has not been clear enough to put them on a level: the HTAs are Syst review?...they are Systematically derived recommendations?.
The RWE would be at the same level as the Studios. There may be more reliable data from RWE than from a randomized clinical trial, provided that the design and evaluation have the possible confounding factors, limitations, biases, etc.
The announcement of the collaboration to update the definition of the HTA opens the opportunity to review this challenging issue https://htai.org/blog/2018/08/13/announcement-international-collaboration-to-update-the-definition-of-hta-now-underway/
A recent article, interesting especially from the regulatory perspective Advancing a Framework for Regulatory Use of Real-World Evidence When Real Is Reliable
http://journals.sagepub.com/doi/10.1177/2168479018763591
Regards,
*Ernesto Barrera Linares*
Family Physician/Médico de Familia
Madrid (Spain)
@ernestob
El sáb., 25 ago. 2018 a las 16:21, Bill Cayley (<
[log in to unmask]>) escribió:
> At first glance, this simply sounds to me like looking at the
> difference between effectiveness versus efficacy. In which case, it’s
> not a matter of where the slides and the evidence hierarchy, but what
> types of outcomes, what types of patients, and what type of generalizability one is looking at.
>
> Bill Cayley, Jr, MD MDiv
> [log in to unmask]
> *[log in to unmask] <[log in to unmask]>*
> *http://twitter.com/bcayley <http://twitter.com/bcayley>*
> Work: 715.286.2270
> Pager: 715.838.7940
> Mobile: 715.828.4636
>
> *A cheerful heart is good medicine... (Proverbs 17:22)*
>
> On Aug 25, 2018, at 4:09 AM, Jon Brassey
> <[log in to unmask]>
> wrote:
>
> I am increasingly seeing the term 'real world evidence' and see the
> FDA are exploring it
> https://www.fda.gov/scienceresearch/specialtopics/realworldevidence/default.htm.
> It also featured at the HTAi conference in Vancouver.
>
> I was prompted to post this here after seeing this news story FDA sets
> out vision to use real-world data in product evaluations
> <https://www.medtechdive.com/news/fda-sets-out-vision-to-use-real-world-data-in-product-evaluations/530026/>.
> Which led to the recent JAMA article Real-World Evidence and
> Real-World Data for Evaluating Drug Safety and Effectiveness
> <https://jamanetwork.com/journals/jama/fullarticle/2697359>.
>
> It got me wondering how does the EBM view this development? How does
> it fit in with the traditional hierarchies of evidence?
>
> Any thoughts?
>
> Best wishes
>
> jon
>
>
> --
> Jon Brassey
> Director, Trip Database <http://www.tripdatabase.com> Honorary Fellow
> at CEBM <http://www.cebm.net>, University of Oxford Creator,
> Rapid-Reviews.info
>
>
> ------------------------------
>
> To unsubscribe from the EVIDENCE-BASED-HEALTH list, click the
> following
> link:
>
> https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=EVIDENCE-BASED-HEAL
> TH&A=1
>
>
> ------------------------------
>
> To unsubscribe from the EVIDENCE-BASED-HEALTH list, click the
> following
> link:
>
> https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=EVIDENCE-BASED-HEAL
> TH&A=1
>
########################################################################
To unsubscribe from the EVIDENCE-BASED-HEALTH list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=EVIDENCE-BASED-HEALTH&A=1
------------------------------
Date: Sat, 25 Aug 2018 23:02:42 +0200
From: Tom Jefferson <[log in to unmask]>
Subject: Re: Real World Evidence
Observational data + spin = RWE.
Professor Tom Jefferson
Senior Associate Tutor
University of Oxford
Oxford OX2 6GG
** Announcing the RIAT Support Center: free support & $150,000 grant competition **
** Details at https://restoringtrials.org/ **
On Sat, 25 Aug 2018 at 18:43, Ernesto Barrera <[log in to unmask]>
wrote:
> I think that the RWE happens like the Health Technology Assessment:
> the pyramid of evidence has not been clear enough to put them on a
> level: the HTAs are Syst review?...they are Systematically derived recommendations?.
>
> The RWE would be at the same level as the Studios. There may be more
> reliable data from RWE than from a randomized clinical trial, provided
> that the design and evaluation have the possible confounding factors,
> limitations, biases, etc.
>
> The announcement of the collaboration to update the definition of the
> HTA opens the opportunity to review this challenging issue
>
> https://htai.org/blog/2018/08/13/announcement-international-collaborat
> ion-to-update-the-definition-of-hta-now-underway/
>
> A recent article, interesting especially from the regulatory
> perspective Advancing a Framework for Regulatory Use of Real-World
> Evidence When Real Is Reliable
> http://journals.sagepub.com/doi/10.1177/2168479018763591
>
> Regards,
>
> *Ernesto Barrera Linares*
>
> Family Physician/Médico de Familia
>
> Madrid (Spain)
>
> @ernestob
>
> El sáb., 25 ago. 2018 a las 16:21, Bill Cayley (<
> [log in to unmask]>) escribió:
>
>> At first glance, this simply sounds to me like looking at the
>> difference between effectiveness versus efficacy. In which case, it’s
>> not a matter of where the slides and the evidence hierarchy, but what
>> types of outcomes, what types of patients, and what type of generalizability one is looking at.
>>
>> Bill Cayley, Jr, MD MDiv
>> [log in to unmask]
>> *[log in to unmask] <[log in to unmask]>*
>> *http://twitter.com/bcayley <http://twitter.com/bcayley>*
>> Work: 715.286.2270
>> Pager: 715.838.7940
>> Mobile: 715.828.4636
>>
>> *A cheerful heart is good medicine... (Proverbs 17:22)*
>>
>> On Aug 25, 2018, at 4:09 AM, Jon Brassey
>> <[log in to unmask]>
>> wrote:
>>
>> I am increasingly seeing the term 'real world evidence' and see the
>> FDA are exploring it
>> https://www.fda.gov/scienceresearch/specialtopics/realworldevidence/default.htm.
>> It also featured at the HTAi conference in Vancouver.
>>
>> I was prompted to post this here after seeing this news story FDA
>> sets out vision to use real-world data in product evaluations
>> <https://www.medtechdive.com/news/fda-sets-out-vision-to-use-real-world-data-in-product-evaluations/530026/>.
>> Which led to the recent JAMA article Real-World Evidence and
>> Real-World Data for Evaluating Drug Safety and Effectiveness
>> <https://jamanetwork.com/journals/jama/fullarticle/2697359>.
>>
>> It got me wondering how does the EBM view this development? How does
>> it fit in with the traditional hierarchies of evidence?
>>
>> Any thoughts?
>>
>> Best wishes
>>
>> jon
>>
>>
>> --
>> Jon Brassey
>> Director, Trip Database <http://www.tripdatabase.com> Honorary Fellow
>> at CEBM <http://www.cebm.net>, University of Oxford Creator,
>> Rapid-Reviews.info
>>
>>
>> ------------------------------
>>
>> To unsubscribe from the EVIDENCE-BASED-HEALTH list, click the
>> following
>> link:
>>
>> https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=EVIDENCE-BASED-HEA
>> LTH&A=1
>>
>>
>> ------------------------------
>>
>> To unsubscribe from the EVIDENCE-BASED-HEALTH list, click the
>> following
>> link:
>>
>> https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=EVIDENCE-BASED-HEA
>> LTH&A=1
>>
>
> ------------------------------
>
> To unsubscribe from the EVIDENCE-BASED-HEALTH list, click the
> following
> link:
>
> https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=EVIDENCE-BASED-HEAL
> TH&A=1
>
########################################################################
To unsubscribe from the EVIDENCE-BASED-HEALTH list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=EVIDENCE-BASED-HEALTH&A=1
|
