Your questions are of interest to us because we are presently involved in
technology assessments of several biofeedback topics.  You have asked
several rather controversial questions.  Some of them apply to the HCFA
process (which may or may not be evidenced based), and some of them are
challenges for evidence based medicine in general.  I can only reply to them
briefly here in general terms.

The first questions involve the HCFA process.  You should be aware that
their present procedures are a work in progress.  They have received much
criticism other than yours, and they expect consumers and interested parties
to continue to give them feedback as they work on the process.  There is
some history to this.  While medical devices and pharmaceuticals in the US
are regulated by the FDA, medical procedures are not regulated by any
government agency.  The only regulation that occurs for procedures is in
terms of policy on coverage.  Several years ago the Office of Technology
Assessment did some assessing of procedures, and this information was made
available to the public and to coverage decision makers.  A powerful lobby
of clinicians who did not want their procedures officially assessed
influenced congress to do away with the OTA.  Technology assessment is now
farmed out by the Agency for Healthcare Research and Quality (AHRQ) to the
private sector.  HCFA can tap into this information, or must develop the
needed information on its own.

Previously, HCFA's coverage decisions were made by unknown processes behind
closed doors.  Amid much consumer pressure congress recently required HCFA
to develop a defined procedure to arrive at its coverage decisions at least
partially in public view, with input from interested parties.  You are
seeing the first attempts at implementing this new policy.  While these
first attempts may not be pretty, hopefully with some more experience and
continued constructive input from the public and interested parties, the
process will improve and become acceptable to all involved.  But this
process will always be the center of a tug-of-war between consumers and
clinicians who want their favorite procedures covered, and taxpayers and
their representatives who want reasonable limits.

Evidenced based medicine was originally developed by physicians as a set of
principles to systematically determine the level of evidence that supports
an intervention of interest.  But "who decides the questions" addressed by
EBM principles can be anyone from a consumer to policy makers.  When the EBM
approach identifies high quality evidence in the form of one or more well
designed, large randomized controlled trials, this is very helpful in
decision making.  At the present, EBM groups, such as the Cochran group,
have concentrated on searching for such high quality evidence and placing
this evidence in accessible databases.  Unfortunately, there are not now,
nor are there likely to ever be, large RCTs on every possible intervention
and every new refinement of a previous intervention.  But there has been
very little effort addressed toward how to use "the available evidence," of
whatever quality it might be, to make necessary decisions.  When clinicians
encounter situations for which there is not high quality evidence, judgement
and personal experience must be used.  But this situation takes on other
dimensions for third party payers, whether governemment or private.  There
are unavoidable incentives to develop a knee-jerk attitude that nothing will
be covered unless there is good large RCT data.  This perfectionist approach
may not be sustainable over all decisions that must be made.  This will
always be a major difficulty for policy makers and is a serious challenge to
the proponents of EBM.  Nevertheless, it will always be the responsibility
of the promoters of new technologies to provide solid evidence for
interventions before asking consumers to foot the bill, whether as tax
payers or as workers whose pay is funneled into insurance companies or HMOs.

Another question you ask is whether an intervention should be compared to
placebo or to another available intervention.  The answer is both.  The
policy decision must hinge on how the new intervention compares, not to
placebo, but to other available interventions.  It is the best intervention
that we are after, not just anything that works better than nothing.
However, without a placebo control, there is no way to know if the competing
intervention has been carried out effectively.  It could be an ineffective
"strawman" that makes the new intervention look better than it really is (as
happened recently in a South African trial of high dose chemotherapy for
metastatic breast cancer).  If a placebo control is prohibitive ethically or
practically, then the competing intervention results must be compared
critically to historical controls.  By the same token, as you correctly
point out, there must be some effort to assure that the new intervention is
carried out with reasonable competence.  This is a particular problem with
biofeedback.  This would all seem to be common sense research, whether it is
called EBM or not.  If researchers and policy makers use unreliable methods
of research and research analysis, this reflects on the individuals and
organizations involved, and should not be blamed on EBM principles that may
have been violated.

I can only urge you (1) to stay active in a constructive way as HCFA works
out its new methods for determining coverage.  And (2) as a proponent of
biofeedback to be active in developing reliable evidence that will be useful
to technology assessors in and out of government.  Call it evidence-based
medicine if you want to, but we are really only talking about good
scientific research, meaning well described studies with good positive and
negative controls.  

David L. Doggett, Ph.D.
Senior Medical Research Analyst
Technology Assessment Group
ECRI, a non-profit health services research organization
5200 Butler Pike
Plymouth Meeting, PA 19462-1298, USA
Phone: +1 (610) 825-6000 ext.5509
Fax: +1(610) 834-1275
E-mail: [log in to unmask]


> -----Original Message-----
> From:	[log in to unmask] [SMTP:[log in to unmask]]
> Sent:	Tuesday, May 02, 2000 10:59 PM
> To:	EBHC
> Subject:	Some very naive questions
> 
> Dear List:
> Please forgive the intrusion of a retired psychologist from the 'States,
> but I urgently need suggestions from members of this list.  I have learned
> everything I know about "evidence-based-medicine" in the past six months,
> and now barely know enough to ask, hopefully, the right questions.
> 
> THE BACKGROUND
> As you may know, in the USA a federal agency called the "Health Care
> Financing Administration" (HCFA) administers the national health insurance
> program, "Medicare".  Although this program pertains only to the elderly,
> the majority of private health insurance programs look to Medicare for
> leadership in evaluating technology and coverage issues, so their
> influence extends far beyond their mandate.
> 
> Recently a new generation of staff members at HCFA decided to use an
> "evidence-based-medicine" approach to reach coverage decisions.   The new
> approach was applied first to the use of "biofeedback" and "electrical
> stimulation" to the treatment of urinary incontinence.  
> 
> (Biofeedback is much more highly developed in the United States than it is
> in Europe, where Electrical Stimulation has been widely used for many
> years.  But, as a reasonable estimate, 90% of the commercial support for
> biofeedback comes from very small businesses (less than 20 employees), who
> are not in a position to mount a serious challenge to government policy.)
> 
> Last fall HCFA invited professional organizations to submit written
> testimony by early December in preparation for public hearings in January,
> 2000.  At the last minute HCFA postponed the hearings and announced that
> they would be held on April 12-13.  Still, all professional organizations
> were under the assumption that their prepared testimony had been submitted
> to the "Medical - Surgical Panel" of the
> "Medicare Coverage Advisory Committee" for consideration.
> 
> Less than a month before the hearings professional organizations were
> informed that none of their prepared testimony had been distributed, but
> instead, an "Evidence-based" evaluation of these technologies, prepared by
> the "Technology Evaluation Center" of the "Blue Cross Blue Shield
> Association", would be the sole evidence debated at the Panel meeting.
> The professional organizations, which included the American Urological
> Association, American College of Gynecologists, the American
> UroGynecological Society, as well as the Association for Applied
> Psychophysiology and Biofeedback, the Agency for Health Care Policy and
> Research, as well as the national physical therapists' and nurses'
> organizations -- all scrambled to revise their testimony in two short
> weeks to accomodate the new rules for debate.  [BTW, NO professional
> organization desented on the value of biofeedback, and only one on
> electrical stimulation.]
> 
> Two days before the public hearings, HCFA posted to its website a list of
> "official" questions which would be addressed to the panel.  And it was
> only at the hearing itself that these questions were formulated into a
> formal "flow chart", such that if the panel could not conclude that there
> was sufficient "evidence-based" research PROVING the effectiveness of
> these technologies, they were not allowed to even address the issue of
> whether or not there was sufficient evidence, none the less, to justify
> coverage of them in the absence of conclusive evidence.   
> 
> THE VOTE ON SCIENCE
> After reviewing the TEC report, the physician-members of the Panel voted 9
> to 2 that the evidence was not conclusive for biofeedback.  All of the
> males voted against biofeedback, while the two females voted in favor of
> biofeedback.   Several male panel members remarked later that they agreed
> with the principle voiced by one of them, that "the level of evidence in
> science is 0.05, but the level in policy decisions is 0.50".  But the
> panel was NOT allowed to discuss policy.  
> 
> As one might imagine, the process used by HCFA is now the subject of a
> popular uprising that may or may not generate sufficient political
> pressure to reverse the results of the HCFA hearings.  That is another
> topic in its own right.  I have joined this list to pursue a different
> tact.  
> 
> MY NAIVE QUESTIONS
> 1. Who decides which questions are addressed in "evidence-based-medicine?"
> The TEC report investigated only those RCTs which compared "Biofeedback"
> with "Pelvic Muscle Exercises Alone".  In contrast, in the companion
> report, they investigated RCTs that (1) compared Stim to placebo (sham),
> as well as (2) Stim compared with alternative treatments.  
> 
> In the first instance, the Burgio et al 1999 JAMA study (which was given
> the highest methodological rating in Berghmans' recent review) was NOT
> considered at all because Burgio compared biofeedback to drugs and
> placebo, but NOT to PMEs alone.   [If the same criteria that was used for
> stim was used for biofeedback, Burgio would have been included, with very
> different results.]   Is this kind of manipulation common in
> "evidence-based-medicine"?
> 
> 2. By what standards are RCTs dismissed from consideration because they do
> not present sufficient evidence of not guarding against selection bias,
> etc.?   The TEC report(s) never established that several studies WERE
> defective in these areas, but merely intimated that they MIGHT be.
> 
> 3. To what extent has "evidence-based-medicine" investigated behavioral
> therapies such as biofeedback and pelvic muscle exercise?  Clearly there
> are many areas of medicine that lend themselves to clear and simple
> analysis.  In pharmacological studies, for example, it is fairly simple to
> ask how many milligrams of the active ingredient were used?   There is
> some controversy in the incontinence field, however.   In some of the
> studies that were classified as "Pelvic Muscle Exercises alone", subjects
> actually received repeated vaginal evaluations by a trained
> physiotherapist who coached subjects on how to contract their pelvic
> muscles.  These studies were then compared with studies where the
> "feedback" was through a machine only.   In the two largest studies
> (Burns, 1993;  Berghmans 1996) verbal feedback was not statistically
> different, in outcome, from machine-feedback.  (In the largest study of
> all, Burgio 1999, machine feedback was dramatically superior to drugs or
> placebo, !
>  bu!
> t that comparision wasn't included.)
> 
> 4. How do you experts account for the problems that may be created by
> uneven or incompetant treatments being tested?   In drug studies, for
> instance, its seems warranted to assume that the subjects who were given
> "50 mg of drug X" really GOT "50 mg of drug X", and those who got placebos
> got placebos.  Drug manufacturers are required to employ chemists who
> certify the integrity of their products.  
> 
> But in behavioral therapies, who is to decide if the subjects who got six
> sessions of biofeedback training REALLY GOT six session of competent
> biofeedback training?  The therapist in Burns' study was never formally
> trained in biofeedback techniques. At the time, her biofeedback subjects
> set a new record LOW for treatment success (61% Sx reduction).  The
> subjects in Berghmans' study used an electrical stimulation electrode
> (circular electrodes) instead of an EMG sensor (longitudinal electrodes);
> they then set a new record for poor results from biofeedback (55% Sx
> reduction).   Would it be considered appropriate to compare surgeries
> performed by first year medical students with surgeons who are board
> certified?
> 
> So my question is, who decides if the "treatment" meets minimal
> professional standards? Or isn't that a consideration in
> "evidence-based-medicine"?  If so, isn't that a serious limitation of
> evidence-based-medicine?  Or are these factors normally considered?
> 
> Well, I have more quesitons, but that's enough for now.   You can find the
> relevant documents at the HCFA website, http://www.hcfa.gov/quality/8b.htm
> and follow the links to biofeedback and Incontinence.  The documents from
> our professional societies are available on my website,
> http://www.incontinet.com; there are detailed descriptions of available
> testimony in the first three paragraphs in the text of the home page.
> 
> On behalf of tens of thousands of urologists, nurses, physicians, physical
> therapists, and their many patients, I hope that some of the members of
> this list will look into this subject and provide guidance as to the
> adaquacy of the "evidence-based-medicine" being promoted by our
> government.  It is even a possibility that this recent activity will
> reflect badly upon the "evidence-based" movement, at least in the United
> States.
> 
> Thank you in advance.
> 
> John D. Perry, PhD, MDiv, BCIA, FAACS
> Committee on HCFA of the Association for
> Applied Psychophysiology and Biofeedback
> 
> 1192 Lakeville Circle
> Petaluma, CA 94954 USA
> 


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%