Steve Simon brings up an interesting point, maybe inadvertently, with his
asking if two particular numbers are "comparable".

Over the last decade and a half or so, the concepts of evidence based health
have helped, in my opinion, improve how we use and review the medical
literature for the purpose of helping patients. With terms like NNT and
absolute risk reduction we are becoming more cognizant of the magnitude of
the risk and benefit of therapy. And, with each metanalysis or Cochrane
review we get a more precise estimate of the potential benefit of therapy
etc. This is all good.

However, for some reason, we continue to translate this medical information
by using terms so vague they often render the "precision" of evidence based
health virtually meaningless. Authors of medical articles continue to use
terms like high risk, low risk, comparable, significant, clinically
important, greater, safer, use with caution, monitor closely etc. without
defining what they mean when they use these terms.

I believe this is an issue that needs to be addressed and is as important as
getting clinicians to use absolute instead of relative numbers.

I have interviewed 100's of patients about the concepts of risk. If you ask
patients "if you were told you were at high risk for heart disease what does
that mean your chance of a heart attack is over the next five years?". The
median response is between a 40 and 50% chance. Almost no one, even if they
had every risk factor for heart disease, is at that high a risk for heart
disease over a five year period.

As a further example, I recently had a long discussion with an editor of the
BMJ about the use of the word "significant". Below is a letter I sent to BMJ
around 6 months ago (it was rejected unfortunately) but I thought readers of
this group might find it interesting.  I am not suggesting that we write and
speak without adjectives and adverbs but we should quantify/qualify,
whenever possible, what we mean when we use these words. Any comments would
be more than welcome.

MY LETTER TO THE BMJ

A significant significance problem

I recently co-wrote an article that was published in the BMJ. Throughout the
initial draft of the article, I repeatedly used the phrase “statistically
significant” to indicate when the results presented were shown to be
associated with a p value of less than 0.05. When I received the galley
proofs for the article the editors had changed all my “statistically
significant” phrases to the single word phrase, “significant”. The following
e-mails transpired and I thought BMJ readers might find the discussion
interesting and thought provoking.

I e-mailed the technical editor with the following:

Dear Editor;

I have always had a problem with the word significant standing on its own -
it is a relative term and is virtually meaningless, at least in my opinion,
without a defining word in front.

It could mean the writer thought something was significant, it could mean
that the results were statistically significantly different or the results
could be clinically significant. Most importantly, just because results are
statistically significantly different does not necessarily mean the results
are clinically significant - a very important difference and the reader
needs to be aware to which type of significant the author refers.

The editor responded with the following comments:

Dear Author;

I know that many people are quite free with their use of significant, but
our house style is to use significant only when it means statistically
significant. In other cases we would use words such as important or
substantial. We never say statistically significant as statistically is
redundant. In addition, we don't allow authors to make claims that something
is significant without the results to support it so it should usually be
obvious that it is "statistically" significant. I hope this is acceptable to
you.

Dear Editor;

I looked up the word significant in a dictionary and it confirmed what you
are saying; that the word "significant" on its own, when used in the context
of statistics, does mean "statistically significant" so it is redundant to
add the word statistical. Always great to learn something new and am happy
to follow the BMJ's guidelines.

Nonetheless, in talking to a number of my colleagues, they did not realise
this strict, so to speak, definition.

However, just for fun - I have an odd sense of fun I suppose - I went
through an article from last weeks BMJ - BMJ 2000;320: 1297-1303 - to see
how the word significant was being used.

The word significant or significantly was used 10 times in the body of the
paper and as a reader I tried to think how would I interpret the word
significantly used in the sentences - did the authors mean clinically
significant or statistically significant. In my opinion, five times the word
significantly was used in the clinical sense, three times in the statistical
sense, and in two cases the meaning could have gone either way.

Here are a few suggestions you may wish to consider as to how to deal with
this significant significance problem.

1) Allow authors to use the word significant only in the results section of
the paper.

2) Every time the word significant is used, there should be a p value placed
in the same sentence or directly after the word significant.

3) In the advice to authors section a sentence or two on the "correct" use
of the word significant may help potential authors and readers with this
issue.

Dear Author;

I suspected that if you looked at the BMJ you might find a misuse of the
word significant (there is always one that gets away). However, I didn't
expect it to be that easy. I will forward your comments for discussion at
our next technical editors meeting. It seems we all have something to learn
here.

Dear Editor;

I am truly impressed that you took my constructive criticism in the manner
it was meant, and decided to review the situation. I would be very
interested to hear the results of the discussion. It appears the use of the
word significant is, dare I say, a "significant" problem.


James McCormack, Pharm.D.
Associate Professor
Faculty of Pharmaceutical Sciences
University of British Columbia
Vancouver, B.C.
Canada
604 822-1710