David,

Unfortunately when it comes to setting MCQs panels of
experts dont know how difficult a question is. The only way
to find out is to try the questions on students. Worse than
that the same question is not equally difficult for
students on appently not very different courses. To give
you an example there is a factor of 1.5 between the times
given for the same question to different groups of students
studying essentially the same material.

> Implicit in what you said is that you don't need a panel of experts to judge
> the difficulty of questions in an exam set. In fact you were a little
> provocative to suggest they wouldn't know. Presumably someone knows how
> suitable a question is, otherwise it wouldn't be used?
>
> Surely someone's had to ratify the questions before they got into your exam
> set? OF COURSE BUT IT DOESNT MEAN THEY KNOW WHAT THEY ARE
DOING.

 Or does anyone just chuck 'em in and wait for the analysis after the
> exam to check how good they were? Does your external examiner have any say?
>
> Without wanting to labour the point, if someone, even an single individual,
> perhaps the module coordinator, decides what questions go into an exam, this
> is made no more or less of a test if there are 50 discreet questions or 50
> stems with hundreds of interchangeable branches. In fact I'd argue that the
> task was made just a little bit easier if the computer can assist in the
> compilation of questions.
I AGREE BUT THE SET OF QUESTIONS THAT EACH STUDENT GETS ARE
NOT EQUIVALENT - FORTUNATELY THEY DONT COMPLAIN AND NOONE
REALLY  KNOWS WHICH IS EASIER AND WHICH MORE DIFFICULT
WITHOUT COLLECTING DATA ON EACH SEPARATE COMBINATION. WE
COLLECT DATA ON ALL OUR QUESTIONS AND ARE BUILDING A TOOL
WHICH WILL ALLOW THE TEACHER OF EACH UNIT THAT USES EACH
QUESTION TO HAVE THEIR OWN SET OF MEASURES FOR THE
QUESTIONS DIFFICULTY AND TIME-TO-DO.

STUDENTS STILL COMPLAIN THAT OUR TESTS ARE NOT EQUALLY
DIFFICULT BUT HOPEFULLY IT WILL GET BETTER WHENT HE TOOL IS
AVAILABLE.

----------------------
Jon Sims Williams
Dept. Engineering Maths,
University of Bristol
Bristol BS8 1TR
Email: [log in to unmask]
Tel; 0117-928 7757, Fax: 0117-925 1154