Hi
I downloaded Jon's document this morning and had a quick scan before
I sent the brief note about how we calculate facility and
discrimination.

What struck me, looking at the document again just now, was how much
of what we do at the end of the day is the same.  I absolutely
agree that the statistical analysis is a waste of time if no-one
looks at the results and attempts to interpret them in a meaningful
way.

In haste, here are a few notes and thoughts about how I look at
facility and discrimination data, for ABCD MCQ exams.

1. Only the values from the main diet exam mean anything. Here
we would expect N>250, so results are almost certainly significant,
though we don't have a formula for it.

2. Resit values are expected to be different, since the ability of
candidates does not represent the general ability of the class.
Also, N is much lower.

3. Negative discrimination usually means that the wrong answer has
been entered.  Occasionally, it is a really poor
question/answer, in which the better students are disadvantaged
because they have more knowledge (or whatever is being tested).
I would delete such a question from the scoring.

4. Questions with a high facility value usually have lower
discrimination then those with low facility value.

5. A question with low facility and high discrimination is a good
question that only the best students are managing to answer.

6. A question with low facility and low discrimination is a poor
question - or one that indicates that students have misunderstood the
material taught, or the question.

7. A good exam paper will be designed to contain questions with
differing values of facility and discrimination.

Mary Masson

On Fri, 23 Mar 2001 13:58:11 +0000 Jon Maber <[log in to unmask]>
wrote:
> (part omitted)

> Apart from any of that the statistical analysis is a complete waste of
> time if noone intends to take action as a result and for that to
> happen they have to understand how to interpret the result.  A common
> problem is that a simplistic explanation can become current - e.g. if
> you get a negative discrimination index on your question you are a
> cr*p teacher and if all your questions have a positive discrimination
> index you are a  brilliant teacher.
>
> The document I referenced before attempts to deal with this issue by
> presenting various scenarios.
>
> Jon Maber

----------------------
Dr Mary Masson
Department of Chemistry
University of Aberdeen
Meston Walk, Aberdeen AB24 3UE

Tel 01224 272931   Fax 01224 272921
email [log in to unmask]