Dear Michel,

as far as I know the ICC *is* the standard for reporting reliability - 
especially inter-rater reliability (see Cronbach et al. 1976, key word 
generalizability theory). One problem of the ICC is that there are numerous 
ICCs that can be calculated (even for the same design) depending on the so 
called universe to which you want to generalize your results: should the 
raters / subjects / items etc. be regarded as fix or random, i.e. do you want 
to generalize to the specific instances that you applied in your study or do 
you want to generalize to the universe ("population") of all possible rates / 
subjects / items ...?

Of course the ICC itself does not reduce variance, which is done by an ANOVA 
(or variance componetns estimation), where you have to speciy the factors. I 
absolutely do not agree with the reviewer with respect to data exclusion, 
since to my opinion you cannot exclude any data you have measured (provided 
they were collected "technically correct"). If you do so, you have to have 
good arguments to justify any exclusion.

With respect to your actual question, I am sorry that I cannot help, because I 
do not know which method he or she is referring to, but I think that it is 
not adequate to mention that there are any better methods without specifying 
one them...

Good luck,
Thilo


On Wednesday 05 August 2009 11:41, Michel Grothe wrote:
> Dear SPMers,
>
> sorry for the question not directly related to SPM. In a recent Paper we
> estimated inter-rater reliability of a HC and AMG manual delineation
> protocol by calculating the relative error for each pairwise comparison
> between the four raters for all volumetric measures and further calculated
> the intra-class correlation coefficient. This was criticized by a reviewer
> saying that "it does not exclude extremes and does not reduce inter-rater
> variance. There are better methods."  For most Papers that I´ve read the
> ICC seemed to be an accepted standard. Does anybody know what kind of
> alternative method he referred to?
>
> Any hints will be highly appreciated.
>
> Thanks in advance,
> Michel

-- 
Thilo Kellermann
RWTH Aachen University
Department of Psychiatry and Psychotherapy
JARA Translational Brain Medicine
Pauwelsstr. 30
52074 Aachen
Germany
Tel.: +49 (0)241 / 8089977
Fax.: +49 (0)241 / 8082401
E-Mail: [log in to unmask]