We ran into a problem recently with using a Kappa statistic, and would like some feedback. We had 2 reviewers looking at forms to determine if particular items were included. The prevalence of items on the forms was anywhere from 25% to 100%. We found that, although agreement was high, the Kappa statistic for items with 100% prevalence was lousy.
In reading about Kappa, it is robust for moderate prevalence items, but much less robust for items of very low or very high prevalence.
We found a correction for the low prevalence problem, but have not found a correction for the high prevalence problem. Any ideas?
Brent
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|