Forwarded to list for those who may be interested (Bart :-)

Alex

-------- Original Message --------
Subject: 	Re: Assessing the learning of undergraduate design students, 
'wing it'
Date: 	Tue, 25 Jul 2006 07:33:25 +1000
From: 	teena clerke <[log in to unmask]>
To: 	Alex Velasco <[log in to unmask]>
References: 	<a04320417c0e4c704b7fc@[192.168.0.2]> 
<[log in to unmask]>



Hi Alex,

Good to hear from you - I must confess, when you first asked for 
suggestions as to how to construct the program's assessment I 
hesitated before responding, as I was aware that what I was going to 
write would go against the grain, and where I was coming from had 
been so influenced by that one session with David Boud's ideas about 
sustainable assessment.

I am aware that assessment as an activity is a dry topic (I rolled my 
eyes when contemplating the prospect of yet another assessment 
subject in my masters of education!), especially in the context of 
design discussions when the lively conversation is about creativity, 
reflection, practice, theory, what is research, etc., but hardly ever 
about learning (especially at undergrad level).

Yet, in my experience, it takes up a large amount of time, 
discussion, thinking and agonising in the practice of teaching. As a 
casual lecturer at 3 different Sydney universities over the last ten 
years, I observed it is the activity that takes the most time, is 
always conducted externally (without student involvement), is treated 
as a secret activity (we are not allowed to divulge students' grades 
to others) and though we are encouraged to provide written formative 
feedback, it doesn't seem to engage students as they seem only to 
notice the grades (marking outcomes at the end). In this sense, and 
from the student's perspective, assessment is a highly emotional 
experience as it seems to cast judgement on their individual worth, 
rather than the work or progress in learning. My observations of how 
they assess their progress across subjects is always based on a 
comparison of grades achieved, not about what they learn, what they 
love, what they still need to learn - their grades influence their 
perceptions about their aptitude in different subjects, even as 
assessment processes are very different from subject to subject in 
the same degree, school, institution. I think that, whilst aware that 
assessment is traumatic and emotional, most lecturers are unaware of 
what it is that they are actually measuring and how to engage 
students in the process of measurement.

And succinctly, learning in an institution is always about 
measurement, for awarding grades + certification, for subject 
cost-effectiveness, for student satisfaction, for teacher 
effectiveness, etc. But it is hardly ever applied as a measure of 
learning and a tool for identifying future learning needs, and it 
almost always precludes student engagement in the process. Assessment 
as an idea is what interests me.

I am heartened that you have picked up on this as I continue to 
struggle with this conflict - between what Glenn expressed as being 
the essence of art and design, or 'winging it' (always highly valued 
and often well rewarded in the institution and practice - seems to 
keep the 'mystery of creation' myth alive), and trying to ascertain 
exactly what it is that I measure when I award grades - sadly, I 
suspect it is akin to 'winging it' as well. However, if we are to be 
responsible educators, we need to be able to validate/support and 
make transparent the construction of this process for the benefit of 
the student.

In questioning the difference between freshmen and graduates, I guess 
it is not dissimilar to the thinking in phd candidature (as I am a 
new phd candidate in transdisciplinary research - with supervisors in 
education and design) - that we are apprentices in learning how to do 
research. So, what are design undergrads? Apprentices in learning how 
to do design? This is where it is tricky - how we define the activity 
of design, and how we assess progress in its apprenticeship (using 
the old binary term). And this is where the topic becomes bogged down 
in the old 'content' debate. And whilst the institutional trend is to 
insist 'student outcomes' aim for achievement of 'generic 
attributes', just how does this get measured, by whom, and in whose 
interests? And is it legitimate criteria for this apprenticeship when 
it seems to be imposed by the institution as a 'one-size-fits-all' 
kind of wish list? Sorry if I am raving - it is 5 in the morning and 
I can't sleep - it's the flu season in Sydney's winter.

As to your question, about the correlation between evaluating 
'criteria' and the subsequent transformation from a student into a 
professional practitioner, then no, I don't think it amounts to the 
same thing. I think it is not dissimilar to the idea of pitting the 
measurement of process against outcome, and assessment being 
'imposed' rather than internalised. If the teacher awards marks for 
criteria such as "Robustness in design process", then this is imposed 
on the student and is difficult to clearly substantiate - exactly 
what does this mean, what does this look like, what evidence can be 
produced to support this, can it be rated on a scale of one to ten? 
The teacher would have to provide explicit examples of what a 
distinction looks like, what a pass looks like, etc. in order for 
students to be able to internalise the marking standards for this 
criteria. And anyway, these are artificial standards (grades 1-10) 
and are not supported in practice - ie. do clients 'judge' your 
presentation by robustness in design process (on a scale of 1-10), or 
is this implied in the outcome, or employed in the rhetoric of 
presentation persuasion? So you can see the purpose for setting this 
as an assessment criteria is solely for the institution. Is it 
measured in the same way in professional practice?

As to evaluating the transformation process, since I have been 
employing Boud's thinking, I have found that by the end of an 
undergrad course, before I give students grades, I ask them how they 
think they went in terms of what they have learned, what they think 
they still need to learn, and what they would do differently if they 
did it again - their answers almost always correlate to my written 
feedback (I don't ask them about grades).

Recently I supervised 2 postgrad students in independent design study 
projects, where I tested Boud's ideas. I set up a learning contract 
which asked them to identify their learning needs in the project, 
learning resources, learning outcomes and to construct their own 
assessment criteria as well as weighting it. This process was 
negotiated and signed off by both of us. The purpose of meeting 
through the duration is to discuss progress, and possibly amend the 
contract (as they clarify what it is that they are learning). Then on 
submission, I ask them to self-assess the project. I mark their work 
according to the criteria and weighting they have set. In each case, 
the final mark varied only by 2%. I found this meant that both of us 
were satisfied that their learning goals had been achieved, that they 
had internalised the standards for achieving them, and that they each 
commented that they were able to continue with this process in other 
subject areas. This is an outstanding outcome for me as an assessor. 
A small sample, I know, but encouraging.

And it seems to support Boud's ideas - if the learning process can be 
self-sustaining, and the student produces evidence of their 
capability of identifying what they learned, what they still need to 
learn, (eventually conducted on their own), and according to what was 
explicitly described as the acceptable standards within a degree 
program (ie. what are the levels you expect them to reach at each 
stage of the degree), then you/they are able to measure the 
transformation in an engaged way. Does this make sense?

I hope this is not too muddled. kind regards, teena

>Glenn,
>
>If I understand you correctly... In my experience, "winging-it" IS 
>the formulaic tradition in art and design education. I don't have 
>any nostalgia for unstructured design (I'm not suggesting that you 
>do.) Consider this: In my ID studies, we were given design projects, 
>without instruction on how to proceed. For example, we were led to 
>believe that research was beneficial, without being told how to do 
>it. Ergonomics was emphasised without any formal training in the 
>science. I could go on.
>
>
>Teena,
>
>I certainly agree with you about radically re-thinking assessment 
>processes in my case. Your message made me realise that my original 
>urge when asking about assessment was slightly different to what 
>came out when I typed it up.
>
>I was curious about the difference we should expect between a new 
>freshman and a graduated student. In what ways should a student be 
>transformed between their entry into university and their 
>graduation? That's why my criteria had points about personal 
>attributes of the student: "Robustness in design process", 
>"identification with a global professional community", "developing 
>strategy for professional career trajectory", "positive bullish 
>outlook for career" etc.
>
>Then, I wanted to know, how to evaluate that transformation that 
>occurs, from a (normally) young adult into a professional 
>practitioner.
>
>Thoughts or contributions? Or, do you think it amounts to 
>more-or-less the same thing?
>
>Regards,
>Alex

-- 
Teena Clerke
PO Box 1090
Strawberry Hills NSW 2012
0414 502 648