Mattias, Fil, Karel and list
In general, any evaluation depends on the intention of the evaluator.
Criteria are always specified by intention but vary with context and
the needs, desires, and goals that spring from the circumstances of
the situation or product being evaluated. Surveys usually short this
issue by assuming that groups pursuing the same activity are "like
minded" in the criteria they apply. That is clearly not so as people
apply preferences that have developed over time through their
experience. This occurs even when a jury of evaluators are asked to
apply the same criteria as anyone who has served on a jury
understands. In my view, what is needed is not a survey but an
heuristic process that can be applied in any situation where an
evaluation is being made. The data it generates would constitute the
"survey" results. The evaluations of different individuals would
provide some measure of variability in applying criteria and of the
similarity of what is being evaluated in different applications of the
heuristic (assuming adequate discrimination within the heuristic.)
One needs better tools to evaluate evaluations.
Chuck
On Jun 5, 2010, at 5:22 PM, Mattias Arvola wrote:
> Fil and Karel,
>
> Such a survey would be interesting. And it would actually test the
> generalizability of my results. We are in our project also looking
> at professionals and amateurs in computer game design and digital art.
>
> There are however a few problems you would need to handle in such a
> survey. My impression after a few years of researching this, is that
> you get a rather idealistic or simple answer when you ask people
> what good design is in general. Such unprecise questions give rise
> to platitude answers. I suspect that when you ask people to tell you
> what was good and bad about one of their own actual projects you get
> a more nuanced answer. Finally, if you look at what people actually
> do consider during a design task you will get yet another answer. I
> will investigate this further in the analysis of my data during the
> autumn. As always, what people say they do is not what they actually
> do. However, what people say they do is also interesting. I will
> write three or four more articles on these issues in interaction
> design during next year..
>
> If you set up a survey study you would need to address these rather
> tricky problems. There might however be some smart questionnaire
> design that I haven't thought about.
>
> Cheers,
> // Mattias
> --
> MATTIAS ARVOLA, Ph.D.
> Sr. lecturer in Interaction Design.
> Linköping University and Södertörn University.
> www.arvola.se
|