Dear all,

I was wondering if anyone could give me some advice.

I am looking at a study where participants were asked to estimate (as a
percentage) the risk of several events. For simplicity lets say the events
were; Getting run over by 1) A bus, 2) A car, 3) A cyclist, and 4) A lorry.

Lets suppose we know the actual risks. For simplicity, lets exaggerate the
risks. So the risk of ever getting run over by 1) a bus is 1%, 2) a car is
2%, 3) A cyclist is 5%, and 4) a lorry is 10%.

There are three separate conditions (independent groups). Lets say these
conditions are age groups; 1) Children 2) Adults or 3) Pensioners.

Suppose I want to compare the accuracy of these three groups risk perceptions.

One way I thought of doing this was to subtract the real risk from the
estimate. So if a participant estimated the risk of getting run over by a
car as 4%, then I could compute an ‘accuracy score’ by subtracting the
‘actual risk’ from this estimate. 4 – 2 = 2. Scores closer to zero are more
accurate.

If someone else estimated that the risk of getting run over by a car was 1,
their accuracy score would be 1 – 2 = -1.

At first this seems appealing, because if someone over-estimates the risk
they get a positive score, and if someone under estimates the risk they get
a negative score.

But my problem comes when I want to compare mean ‘accuracy scores’ across
the three groups.

Supposing Children have a mean accuracy score of -1.6, Adults have a mean
accuracy score of 1.5, and pensioners have a mean accuracy score of 3.

If I run an ANOVA and find significant differences between all three groups,
what can I conclude? It seems reasonable to conclude that Pensioners are
less accurate than Adults, because their score is significantly different,
and further from zero.

But… Children’s and Adults’s scores are also significantly different from
each other. Yet it does not seem reasonable to conclude that one is more
accurate than the other. They are slightly different distances from zero,
but a larger difference from each other.

This made me wonder if it made more sense to compute absolute differences
when computing the ‘accuracy’ scores. You can’t have a negative accuracy
after all (I think – surely accuracy ranges from 0% to 100%). When I do this
it has a big impact on the results. 

Yet I feel uneasy about using absolute differences and ignoring the sign. It
feels like I’m loosing some important information along the way or maybe
inflating the degree of inaccuracy displayed by participants. 
Another way to look at accuracy would be to take each condition (i.e. age)
separately, and each risk estimate (i.e. vehicle that might knock them down)
separately and conduct a series of one sample t-tests. This would tell me if
each groups mean estimate was significantly different from a specified test
value (the actual risk).

This also seems like an unsatisfactory solution. Is there some sort of
multivariate equivalent of a one sample t-test or is this a contradiction in
terms?

I would really appreciate any advice or thought on the type of analyses I
could run on this data that would let me compare accuracy between the three
groups.

Thanks in advance,
Brian