Timothy,
I concur with John Peter's sentiments, but would like to take the
conversation a little further. I think this is a valuable topic to
discuss because of what it reveals about view of teaching. Incidentally
the latest issue of Exchange (published by the HE Academy in the UK)
reveals just how contested notions of excellence are.

First I have some questions. How do you expect the two protocols to be
used? Who will be making the judgement about whether the criteria have
been met and at which 'level'? What evidence will be used to support
claims being made by the staff member?

It seems to me that what appears at first sight to be a rational model
with some aspirations to 'objectivity' begins to look much less secure
once we consider how in detail the protocols might be used.

Let's look at one of the criteria suggested as an example. 
The lecturer actively incorporates his/her research programme into the
teaching of the module
The lecturer incorporates ..... to a certain extent 
There is very little evidence of the lecturer incorporating....

The suggestion is that actively incorporating his/her research programme
into the teaching of this module is amenable to measurement and that it
is possible to distinguish between an absolute state of 'active
incorporation' and varying degrees of partial 'active incorporation'. 
Who is to set the standard of active incorporation? How will this
standard be interpreted across vastly different subjects e.g. in
engineering and in fine art? How will consistency of interpretation be
moderated, and by whom and with what authority? How will the reliability
and validity of these judgements be established? How will appeals
against these judgements be considered?

Once you begin to ask these questions and imagine how the protocol will
actually be used by staff, either as applicants or as 'judges' it is
clear that the appearance of objectivity is bogus. At every level, and
in respect of every criteria, personal judgements will have to be made
which as John has said will be coloured by values, ideologies and
beliefs.

This is even more obviously the case when you suggested that judgements
be made about the perceptions of lecturers by students as 'Competent,
Trustworthy, Students Feel Included/Respected, Stimulating,
Challenging'. 

That is not to say that such a scheme could not be operated (though I
think it would run into difficulties about the comparability of the
judgements being made by different people at different times about
different portfolios, and the time and expense that would be involved in
trying to provide appropriate evidence against so many criteria), but it
could not claim to be objective. Better to accept that we are dealing
with a complex area of social life about which objective measures are
not possible. 

In my view, the general rule for criteria is the simpler they are the
better (and don't try to have different levels of achievement).
Secondly, allow for some variability between different subjects. Allow
staff to determine what they think is relevant evidence derived from a
variety of sources and give them the opportunity to justify the evidence
they have chosen. Have a training programme of your staff making the
judgements in order to achieve some degree of consistency and try to
ensure all staff understand the criteria. 

Incidentally much the same difficulties occur with judgements in other
areas of professional life - social work, nursing, health related
disciplines, medicine, law etc. Judging teaching quality is not in
principle different from making judgement in these other professions. 

Peter Seldin has written interestingly about this topic in 'Evaluating
faculty performance' (Anker 2006), though I don't always agree with what
he says. Like you he wants to establish a more objective basis for
making judgements about staff, and he quite rightly objects to 'casual
bias, hearsay evidence, and gossip' as the basis for promotions and
fulfilment of probation (tenure) requirements. We are all against
prejudice, bias and bigotry in personnel decisions, so the challenge to
find a fair and consistent way of making judgements is a real one. Most
PgCerts in the UK do try to make such judgements, though in some cases
the judgement is more about the academic skills of the staff member
writing the portfolio than their qualities as a teacher.

As in so many other areas of professional life, there is no perfect
answer, but we can try to operate a fair and consistent system (but not
one that will ever be entirely objective).

David Gosling
Higher Education Consultant
Visiting Research Fellow
University of Plymouth 
tel/fax: 0161 456 6148
mobile: 0784 1647275


-----Original Message-----
From: Online forum for SEDA, the Staff & Educational Development
Association [mailto:[log in to unmask]] On Behalf Of John Peters
Sent: 09 January 2008 12:56
To: [log in to unmask]
Subject: Re: Summative Assessment of Teaching and Learning Protocol


Hi Timothy,

This is a very interesting piece of work and I have to admire your
positive and scholarly collation of possible measures.  I think this is
a really useful document - but not necessarily in the way it is
currently presented!

I have some serious concerns about the assumptions in it.  I'd strongly
recommend you read A. Skelton (2005) _Understanding Teaching Excellence
in HE_.  His persuasive observation, based on research into reward
schemes for teaching, is that 'teaching excellence is a contested,
value-laden concept which is historically and situationally contingent'.
He suggests there are many varied models of what constitutes excellent
teaching and that we need to expose and discuss their underlying
assumptions before even seeking to define criteria.  He offers a
convincing critique of the performative, managerial model of teaching
excellence, a less convincing critique of the psychologized 'approaches
to learning' model, but a compelling argument for a critical,
emancipatory model of teaching excellence.  

I wouldn't want to speak for Skelton but I think he'd share my concern
about whether 'demonstrably objective measures' of teaching quality are
possible or even desirable.  The underlying assumptions of such measures
often emphasis performative, managerial models of good teaching, measure
those things that are most easily measured, and ignore both the wider
structural determinants and the critically-aware human beings at the
heart of it all.  Perhaps, rather than attempting to define measures for
staff, it would be better to invite staff to present their own
individual cases for their excellent teaching [based on their own
explicit selection from the variety of models and measures] and to
establish that these cases will be judged, not on whether they comply
with management objectives but on their academic merits?!


cheers
John

Dr John Peters
Learning and Teaching Centre
University of Worcester
01905 855506

-----Original Message-----
From: Online forum for SEDA, the Staff & Educational Development
Association [mailto:[log in to unmask]] On Behalf Of Timothy Murphy
Sent: 08 January 2008 16:31
To: [log in to unmask]
Subject: Summative Assessment of Teaching and Learning Protocol

Dear Colleagues,

As a postdoctoral researcher at the National
University of Galway Ireland I have constructed draft
protocols (SALT 1 and 2) for the Summative Assessment
of Teaching and Learning in Higher Education.  It is
proposed that SALT 1 will be completed during the
academic's probationary period, i.e. first 3 years. 
SALT 2 is a further development of SALT 1 and has a
particular focus on what is referred to as
research-informed teaching.  It recognises the
centrality that is accorded to research in higher
education today.

As this is a preliminary draft in an effort to develop
a Summative Assessment of Teaching and Learning
Approach for Higher Education, any thoughts or
comments or indeed recommendations that you may have
would be very much appreciated.

Thanking you sincerely for your time and
consideration.

Respectfully,

Timothy Murphy
Postdoctoral Researcher
National University of Ireland 
Galway
    

No virus found in this outgoing message.
Checked by AVG Free Edition. 
Version: 7.5.516 / Virus Database: 269.17.13/1214 - Release Date:
08/01/2008 13:38
 


This message has been scanned by MailController -
www.MailController.altohiway.com