Dear Allstat members,
My query regarding the proper assessment of reliability when the outcome
is a count has produced many interesting replies, which I have
summarised below. Many people suggested collapsing the counts into
categories and even reducing them to binary 'abnormality present/absent'
assessments. Some suggested first assessing agreement at this binary
level, then looking at the non-zero counts in more detail. Just looking
the non-zero counts would be difficult because there are not many of
them in the data set. I was not able to collapse the counts because I
needed to keep in mind what the counts were going to be used for once
reliability was assessed. In arthritis research, damage progression is
an important outcome, and in sites of the body where large bone lesions
develop it is interesting to record how many of them a patient starts
with, then to record how many of them remain following treatment, how
many have resolved, and how many new ones have appeared. If we were to
merely state that abnormalities were present/absent before and after
treatment we would lose a lot of information, and we might heal 4 out of
5 lesions in a patient without being able to demonstrate change
following treatment. The lesions in the current analysis are very small
and difficult to count, and this analysis is the first step towards the
possible creation of a new method of assessment for these small lesions
that might or might not involve counts, depending on what the agreement
is like when counts are used. So the researchers may end up creating an
ordinal or even binary scale, but first of all they need to know whether
this loss of data is necessary - how good or bad is the agreement when
we just count these small lesions?
Happily, despite my concerns that the patients had not, strictly
speaking, been allocated into different categories by each reader, John
Hughes pointed out that an example in Altman's Practical Statistics in
Medical Research uses count data in Kappa analysis. The counts in my
data set do not exceed 5 - I will need to double check with my
colleagues as to what the theoretical limit is in terms of the number of
abnormalities that could be present in the particular anatomical regions
of interest in this study. If it turns out that Kappa statistics are
suitable for this data then I will also try weighted kappa, given that
there seems to be good agreement within a margin of 1 in my data set. I
have never been exactly sure how to interpret weighted kappa results, if
one is permitted to use whatever weights seem appropriate. If one person
applies weights of 0.5 to the cells immediately off the diagonal in a 5
by 5 table, whereas the next applies weights of 0.75 to the off
diagonals and 0.1 to the cells further removed, and they both obtain a
weighted kappa of 0.7, they cannot have measured the same level of
agreement, so what does the weighted kappa statistic tell us? Generally
I am not sure how to qualify the level of agreement once a test
statistic has been produced: one cannot reasonably expect that raters
who have been looking at the same patients will only agree as much would
be seen by chance, so the significance levels provided with kappa and
ICC results by most stats packages are largely meaningless. I have
references that detail how to assess whether agreement is significantly
greater than some prescribed minimum (rather than significantly greater
than chance); most of these are by Allan Donner and Michael Eliasziw,
and are very useful when powering reliability studies. However, given
the sensitivity of both Kappas and ICCs to the distribution of the data
in question, I find it hard to apply the qualitative agreement values
suggested by Landis & Koch (1977) to the actual agreement statistics
produced: two reliability studies in which raters disagree by exactly
the same margin (say 0.01 mm) over the same variable will produce
different ICC values if the spread of the data between patients is
different. The agreement in one study might therefore only be 'poor',
whilst the other has found 'good' agreement, when the disparity between
raters' assessments is exactly the same, it's just the spread of values
within each dataset that differ.
If it is possible for a patient in our study to develop more than 10 or
so of these small lesions, I think will follow Jay Warner's advice and
apply some small value to my 'zero' counts (0.1 for example) then try
transforming the data before going on to more complex model-based
analysis. I can then compare the results of the two methods of assessing
agreement (Kappa/non-kappa). As Jay points out, a zero count may in some
cases not reflect 'no abnormality present', instead there may be some
low level damage that has not yet developed sufficiently to be deemed a
distinct lesion, so the allocation of some small but non-zero figure to
these cases may be justified.
Many thanks to all of you who replied.
Liz Hensor
Dr Elizabeth M A Hensor PhD
Data Analyst
Academic Unit of Musculoskeletal and Rehabilitation Medicine
36 Clarendon Road
Leeds
West Yorkshire
LS2 9NZ
Tel: +44 (0) 113 3434944
Fax: +44 (0) 113 2430366
[log in to unmask] <mailto:[log in to unmask]>
.........................................
Original query:
My colleagues are interested in assessing the agreement between two
readers who each assessed patients' MRI scans for the number of certain
abnormalities present. For each patient I therefore have two abnormality
counts, one from each reader. Because these are counts, Kappa-like
statistics are not applicable since there are no defined categories into
which each reader places each patient. I was initially drawn towards
Intraclass Correlation Coefficients, but strictly speaking I don't think
ICCs are suitable for count data either since they are ANOVA-based
(please correct me if I'm wrong in assuming this). The majority of
patients have no abnormalities, which serves the dual purpose of skewing
the data into significant non-normality, whilst making it difficult to
perform transformations (given that most of the data points are zero).
Any hints on how best to proceed here would be greatly appreciated.
.........................................
Response from Robert Newcombe:
My instinct is to examine two parts of the issue separately: any
abnormality vs. none, which is binary, leading to the simplest
application of kappa; and, when both readers detect abnormality, some
sort of comparison of multiplicity of abnormalities detected. Also,
regard different kinds of abnormalities as giving rise to different
present/absent binary variables - this may be more informative than
dealing with the actual number of abnormalities.
Hope this helps.
.......................................
Response from Paul Swank:
A nonlinear mixed model should allow you to estimate the random
variances given the distribution you describe. If the zeros are numerous
enough, you may need to go with a zero-inflated Poisson (ZIP) model.
......................................
Response from John McKellar:
Can I make an assumption about your objective? - which changes the way
the problem presents itself:
If you want to be good at distinguishing "no abnormalities" from "with
at least one abnormality" - then the issue is binary. I suggest that
this is very important to get correct.
Then if you need to determine that the "with abnormalities" assessments
are "of the same order of magnitude", you need to continue with the
modelling issue you discuss.
By this management of the issue, you can separate out the problem you
were hitting and come to a conclusion.
Now, if the assessments from the binary part are poorly correlated, then
you have a serious problem - more serious than if the latter part is
poorly correlated. So, you need a method of discussion the comparison
between assessors which allows you to weight the two measures together
.. And that, I fear, is arbitrary. My hope would be that the binary
part is very consistent between assessors - and then much of the issue
fades away.
Also, ask yourself (or client) what you need. Is it assurance that
there is a degree of agreement, or is it the need for a high degree of
agreement?
......................................
Response from Zoann Nugent:
Have you considered two analyses?
1 A simple kappa with two classes - no abnormality(0) & something(1+)
2 An examination of the major axis of correlation for instances where
both examiners give a score above 0. In this case, perfect agreement
would lie on a line with intercept 0 and slope 1, r squared=100%.
(Just a high r squared is insufficent).
...............................
Response from Nick Longford:
I have experience of a similar setting: two surveyors assess the level
of disrepair of a residential property. The assessment is on a scale
1-10 for a large number of elements, such as windows, roof, floor,
heating, insulation, wall decoration, gutters, etc. This is in a large
scale survey. A random subsample of dwellings is assessed twice,
independently. The task is not only to make inferences about the
consistency of the assessments, but to take account of the inconsistency
in the estimates of ...
Solution: Each subject/property is associated with an ideal assessment.
For each possible assessment, neighbourhoods are define that delineate
the level of discrepancy (error) in the case when the ideal assessment
is A, and the assessment made is B. From pairs of assessments, the
probabilities of discrepancies/errors are estimated -- nothing to do
with kappa.
..........................................
Response from John Hughes:
I don't understand why you have ruled out the Kappa statistic. I presume
that the number of abnormalities observed come from a Poisson
distribution. You say that the majority of counts are zero which
suggests that patients only have a few abnormalities at most. I think
that it would be quite reasonable for you to use these counts as
categories, the example in Altman's Practical Statistics in Medical
Research p. 403 has four. If there are too many count categories I don't
see why you can't categorise these counts into a smaller number or even
just two for patients with either none or some.
....................................
Response from Jay Warner:
1) Have you considered that this is a problem in measurement &
reliability, which the mfg. folks have long since addressed? I would
suggest, if you have the readers examine (electronically) identical MRI
scans, that you try an R & R - Repeatability & Reproducibility study, if
I recall the words correctly.
You have a number of 'samples' - MRI scans of different patients. You
have 2 readers. You may have more than one MRI for each patient, taken
back to back or with different set-ups. You may have more than 1 display
machine or reading condition setting in which they work. That's an R&R.
The question you wish to answer is, how much of the variation in
measurement (count in your case) is due to the readers, and how much to
the changes in the MRI machine, run to run or setting to setting? If
you pick your test samples properly, you can also back out the variation
due to the typical count as well. I expect that there will be less
variation when the typical count is 2 than when the count is 25 or 100.
One word of warning: So long as you expect the two readers to exactly
agree, you will miss the boat. Even the determination that a 'count' -
event - exists at a location can be seen as the conversion of a
continuous observation of something, into a dichotomous
'event/non-event.' If this warning makes absolutely _no_ sense at all,
we need to talk. If it is unclear, I'm sorry - I'm in a hurry. I can
try again later.
2) If your counts run around 25 or more, in those cases where there
is a count, then I would suggest you check if the distribution is
log-normal. Also, you might be able to treat the counts as ratio,
continuous values and go from there.
3) Where the count is 0, we are really saying that we did not detect
the signs of an 'event.;' In chemical analysis, we do not say 0 ppm of
anything, we say "not detected" or "less than 5 ppm." One way to treat
those 0's is to call them 0.1 or 0.5, and then run whatever
transformations are involved.
....................................
Response from Peyman Jafari:
I am a Ph.D student on biostatistics. I think a tranformation on data
would be helpful. Square root of data is a good transformation for count
data. Based on this transformation, the distribution of your data will
change to normal.
.....................................
Response from Irene Stratton:
I have looked at data like this - from microaneurysm counting on retinal
photographs. Most people (type 2 diabetes) had none, then 1, then 2, and
we grouped together 7 or more. We then used kappa statistics using a
weighting value of 0.5 for the cells one off the diagonals. This might
work for your data. It certainly seemed to give us answers that made
sense (which is the main object of these sorts of analyses). If you are
trying to work out a count of DIFFERENT sorts of abnormalities you are
into a whole new ball-park - I don't think that this is very easy - I
tended to look at microaneurysms, then cotton wool spots, then hard
exudates..... we did use a US derived scheme that allocated grades for
different combinations of things on the photos, but this was insensitive
and when we had a serious problem with one grader it turned out to be
cotton wool spot grading, which we would have detected earlier and more
easily if we had been doing the lesions one by one instead of reporting
the US derived grading.
Does this help at all?
|
