I am grateful to Doug Altman for responding on behalf of the CONSORT group. For clarification of my motives in raising the issue, see the
messages archived on www.jiscmail.ac.uk : allstat.
The defence of any misunderstandings regarding the guidelines is that they
arose from approaching the documents as a fresh eye. So while I accept
that multiple observations (repeated measures) were intended by the
authors, the words used did not clearly rule out multivariate responses.
The text seems capable of improvement if it is to have sufficient impact
on readers not already familiar with the concepts. Sod's law says that
anything that can be misread will.
The word 'validity' does occur once in the checklist, but on the
penultimate item and relating to generalizability. The implication of
this position is that it is a retrospective check, though logic dictates
that it should have been built into the planning. That is my excuse for
overlooked the mention. The guidelines set out to advise on the
reporting of studies, rather than spelling out the steps in the planning
and design.
The notes supporting the guidelines give brief definitions of internal and
external validity. I agree that the word is used with several distinct
meanings. The meaning used by CONSORT, of unbiasness, is also the one
given in Kendall and Buckland. To my mind this is of limited use and
reinforces the outsiders' impression that statistics is about mathematics
and 'doing the calculations right'. I would be more concerned to stress
the 'construct or content validity', an explicit check that what was
measured was *what was intended* and what was *the quantity of interest.*
This idea is hinted at in several places, but for example the
recommendation always to use previously-tested instruments may discourage
researchers from triangulating their own study. In educational and social
research, formal tests of 'reliability' are easily overused and
misapplied. The simplistic application of statistics in (UK) health
management should be a warning to any medical researcher!
Checking on what I meant has made me realise that the textbooks I
consulted mainly give sketchy definitions of the term, with the
honourable exception of Mick Youngman's "Analysing Social and Educational
Research Data". What did not come out of *any* book was the idea that
validity applies to *both* variables and sampling. Good soundbites are
"make the important measurable, don't make the measurable important" and
"an approximate answer to the right question is better than an exact
answer to the wrong question". Indeed, unbiasness is not an unmixed
blessing.
One easily available reference, relevant to the medical context, is a
paper by Camilleri-Brennan & Steele "Measurement of quality of life in
surgery", available as http://www.rcsed.ac.uk/journal/vol44_4/4440010.htm
A number of other references found turned out to be students' writings,
suggesting that this concept is deemed to be something needing discussion,
not a definition to be memorized.
As noted before, I am strongly supportive of the CONSORT recommendations
and would be pleased to see similar initiatives covering other types of
analyses, and the rationale for choosing between procedures.
R. Allan Reese Email: [log in to unmask]
|