I'm not sure exactly when AnoVa was invented but its use has only
become common (I mean in general not just for MCQs) in recent decades
so I'd probably go along with the spirit of the opinion given below
but point out the AnoVa is generally the accepted way to compare two
or more means with each other. As mentioned before we have to be
careful about terminology. The traditional maths that goes with the
term "discrimination index" at least in the UK is to place the students
in two groups according to performance on the test, ignoring the
mediocre third of the students as a first step. Then for
each question you simply calculate the mean score of the "top" and
"bottom" students on a scale 0.0 to 1.0 and find the difference.
There are a number of problems with this.
It treats performance on the test as a whole as a logical selection
into two groups when clearly it is a score on a sliding scale with a
roughly normal distribution. It makes the opposite mistake on the
score for the question when it is a true MCQ. It also gives no guidance
on how significant the result is. A "craft" advice is often given
that if the number of students falls below "n" you should bump up
the numbers by not discarding the middle third. There are no real
statistical justifications for the validity of any of this and my
experience is that you have the double problem with academics that
either a) they don't understand the purpose or value of the analysis
or b) they understand the statistics well enough to argue that it
shouldn't be done. The upshot is they get away with never examining
the performance of the questions.
If you do the stats right you still need to convince half your
audience but at least it can't be shot down on the basis of unsound
methodology.
Apart from any of that the statistical analysis is a complete waste of
time if noone intends to take action as a result and for that to
happen they have to understand how to interpret the result. A common
problem is that a simplistic explanation can become current - e.g. if
you get a negative discrimination index on your question you are a
cr*p teacher and if all your questions have a positive discrimination
index you are a brilliant teacher.
The document I referenced before attempts to deal with this issue by
presenting various scenarios.
Jon Maber
|