Brent Beasley wrote:
>
> We ran into a problem recently with using a Kappa statistic, and would like some feedback. We had 2 reviewers looking at forms to determine if particular items were included. The prevalence of items on the forms was anywhere from 25% to 100%. We found that, although agreement was high, the Kappa statistic for items with 100% prevalence was lousy.
You might use the raked kappa, see:
Agresti, A. Ghosh, G.Bini, M.
Raking Kappa: Describing Potential Impact of Marginal
Distributions on Measures of Agreement.
Biometrical journal. 1995 v 37 n 7 p.811
It is similar to Cohen's kappa, but before doing the calculations you
transform your tables to fit a pre-selected prevalence (the same for all
your comparisons - 50% is a common choice), while *mantaining the odds
ratio of the actual data* of each table.
You can then compare the raked kappa between different pairs of
observers, or you can compare the raked kappa with the Cohen's kappa of
the same pair, to judge how much the latter is influenced by the
prevance.
Keep in mind that this technique can adjust for differences in
prevalence between populations (like the ones that you describe) *OR*
between observers (when one observer of a pair gives a higher prevalence
estimate than the other) but not both! So you may want to use it only
after a negative McNemar test (i.e. a paired chi-square test).
cheers,
Piersante Sestini
--
======================
dr. Piersante Sestini
Istituto di Malattie dell'Apparato Respiratorio
Universita' degli Studi di Siena
Viale Bracci 3, 53100 Siena, Italia
tel (39)0577-586794
fax (39)0577-280744
mailto:[log in to unmask]
http://respir.med.unisi.it
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|