hi ufuk,
I am on the road and can't give you references but I urge you to read the chapter on recording in my "content analysis, an introduction to its methodology.
re q1, of course you need to give your coders all of the categories you wish them to apply and you need to be very specific. after all reliability measures the degree to which coding instructions are unambiguous and coders can apply them faithfully.
re q 2, you should use as many coders as you can make available. this provides you not only with a more general assessment of the reliability of your data, but also who is up to the task.
re q 3, this possibility needs to be part of your coding instructions, provided to the coder as a code, and be reliably used as well.
finally, cohen's kappa is severely limited to two coders, perfect agreement on the margins of a contingency table, and large sample sizes. 14 interviews is not a large number
Sent from my iPhone
> On Aug 8, 2014, at 6:54, "ufuk ulusan" <[log in to unmask]> wrote:
>
> Dear all,
>
> I am in the coding process of some data transcribed from 14 semi-structured interviews with design students. I have 30-40 codes for each one with repetitions between them and 6-8 themes for the final categorization. I plan to use Cohen's kappa with the help of some other coders to ensure the reliability of the themes but i have three questions to be answered:
>
> 1) I believe i should give the coders the themes and their detailed explanations so that they can agree or disagree about the case to be in that theme, but should i also give them the codes or will marking the segments that are coded in the transcripts be enough?
> 2) Should i pick up two coders to agree or disagree between each other or should i pick several coders to measure the agreement between me and the coders?
>
> 3) What happens if a coder thinks that some segments in the transcriptions do not belong to neither of the themes in the categorization system?
>
>
> I would appreciate any help about these questions. Thanks in advance.
>
> Best regards,
> Ufuk Ulusan
>
> Department of Industrial Design
>
> Mimar Sinan Fine Arts University
> Šstanbul, TR
>
>
> -----------------------------------------------------------------
> PhD-Design mailing list <[log in to unmask]>
> Discussion of PhD studies and related research in Design
> Subscribe or Unsubscribe at https://www.jiscmail.ac.uk/phd-design
> -----------------------------------------------------------------
-----------------------------------------------------------------
PhD-Design mailing list <[log in to unmask]>
Discussion of PhD studies and related research in Design
Subscribe or Unsubscribe at https://www.jiscmail.ac.uk/phd-design
-----------------------------------------------------------------
|