Dear Allstat,

My query regarding change scores for ordinal variables produced some
very interesting responses. My thanks to Stephen Senn, Paul Swank,
Adrian Lambourne, Philip McShane, Ruby Chang, Val Gebski, Shakir
Hussain, John Whittington, Blaise Egan and Michael Dewey for their
informative replies. I will attempt a broad summary of the responses
here, and if anyone would like to read all of the replies I received, I
will forward them on request. 

Generally there seems to be a consensus that it is not valid to
calculate change scores for ordinal variables. In the literature, the
Wilcoxon signed rank test is frequently used to determine whether
patients in open-label studies (as opposed to RCTs) have made
significant improvements during treatment, and is often applied to
qualitative questionnaire data, because questionnaires (such as the HAQ,
RAQoL etc) generally produce summated scores that are ordinal in nature.
The Wilcoxon test subtracts the second value in each pair from the first
(essentially creating a change score), before ranking these differences.
I was interested to discover that the test makes an assumption that is
rarely mentioned in general statistical textbooks - namely that in
addition to the requirement that the values in a data set be rankable,
differences between values must also be rankable. This requirement
essentially means that the Wilcoxon signed ranks test is only really
valid for testing ordinal data that approach interval level scaling.
This assumption is almost never formally tested in practice, probably
because most textbooks just introduce rank-based non-parametric tests by
stating that they make fewer assumptions about the distribution of the
data than parametric tests, then go on to discuss the penalties one pays
in terms of power when relaxing these assumptions. I have yet to find a
book in our collection here, other than Siegel, that explicitly points
out the assumptions of the Wilcoxon signed rank test. Do any of the
commonly available statistics packages provide means by which one can
test the assumption of rankability of differences between values? I get
the impression that a lot of effort goes into testing whether or not
data violate the assumptions underlying parametric tests, and if the
data do violate these assumptions then non-parametric methods are
applied as a matter of course, with no further assessments being made to
ensure the assumptions of those tests are met.

Several of those who responded recommended that ordinal regression be
used in circumstances where one wishes to compare the change in ordinal
scores between two treatment groups, taking baseline scores as a
covariate or factor (rather than using a Wilcoxon-Mann-Whitney test on
change scores). Paul Swank also suggested using Rasch techniques to make
ordinal measures more interval in nature before creating a nonlinear
mixed model. These are both excellent suggestions, but in practice may
not always be feasible. The accuracies of both regression and Rasch
techniques are to an extent dependent on the number of cases available
for analysis. Where one is analysing small data sets (total N<=40) these
techniques may not provide results that are substantially more accurate
than the (albeit invalid) change score technique. Since non-parametric
techniques are often recommended for the analysis of small samples this
could be problematic. 

I am slowly working my way through a paper on the stratified Wilcoxon
rank sum test (below) which was highlighted by Val Gebski. Val mentioned
ACCoRD could perform this test: are there any other commonly available
packages that offer it? The only mention I've found online so far is to
time-to-event stratified Wilcoxon analysis in Stata and SAS, which I
don't think is quite the same thing.

JASA  Sept 1999 v94 i447 p970(9) Nonparametric two-sample comparisons of
changes on ordinal responses. Peter Bajorski; John Petkau.  

Regarding the unresolved issue of how to assess change within one group
on an ordinal variable that violates the assumptions of the Wilcoxon
signed ranks test; Stephen Senn pointed me towards a paper by Diane
Kornbrot: The rank difference test - a new and meaningful alternative to
the Wilcoxon signed ranks test for ordinal data. Kornbrot D.E.. BRITISH
JOURNAL OF MATHEMATICAL & STATISTICAL PSYCHOLOGY 43: 241-264 Part 2 NOV
1990.

This test is essentially the same as the Wilcoxon signed ranks test,
except that it first ranks the raw values, then calculates the
difference between the ranks of values in each pair. I'm glad to have
this reference as a colleague and I had the same basic idea ourselves
whilst discussing this problem, but had not got as far as figuring out
how to derive significance values for the test, a process which turns
out to be relatively complex. Does anyone have any opinions regarding
the use of this test? Should more attention be given in the literature
and in the statistical packages to testing the assumptions of
non-parametric tests? It can't be enough to simply assume the data meet
the conditions required, any more than it would be acceptable to assume
the data were normally distributed and therefore suitable for parametric
analysis.

Many thanks again to all those who replied,

Liz Hensor

 

 Dr Elizabeth M A Hensor PhD

Data Analyst

Academic Unit of Musculoskeletal and Rehabilitation Medicine

36 Clarendon Road

Leeds 

West Yorkshire

LS2 9NZ

Tel: +44 (0) 113 3434944

Fax: +44 (0) 113 2430366

[log in to unmask]