Good questions, Ian, and an interesting discussion. 

The unit of assessment will surely be (in the final analysis) institutional? Everything said so far seems to indicate that. This means that any 'index' would need to operate at institutional level, and translate across very different kinds of institution. Given the vastly different types of roles, 'job families' and organisational structures, coming up with some kind of scholarship index which makes sense to all institutions is going to be very tricky. I can certainly see the merit of such an index for internal institutional purposes, but finding something which works across such diverse institutions, with quite different roles and missions, could be next to impossible. (I write from an institution which, for example, will just not submit applications for NTFS because of the seriously adverse effects of doing so in the past - although we may re-open the discussion on that one in time.)

My understanding is that the 'basket of metrics' will predominantly include data already available, such as NSS, staff qualifications/professional recognition, student retention and achievement, and so on. It would be good to see some recognition of teaching-related 'scholarship' (defined broadly), but is it logically possible to do this at institutional level right across the sector? 

It will certainly be interesting to see how this develops.

All the best

Dilly

Dr Dilly Fung PFHEA
Director, Centre for the Advancement of Learning and Teaching (CALT)
UCL
020 7679 5939 (Internal ext. 45939)
1-19 Torrington Place
London 
WC1E 7HB
 
@DevonDilly
@UCLConnectedC

On 23 Jul 2015, at 09:58, Ian Scott wrote:

I am enjoying this conversation, particularly as I organising an event for NTFs on 30th of July at Brookes to discuss 'TEF'  Sorry at moment for funding issues only open to NTFs; but I will open it up wider if I have places to fill. I think the purpose of TEF as a manifesto item is two fold, 1) to answer the value for money question and 2) to ensure that teaching quality and UK institutes remains world class. I think two significant issues to be resolved are a) at what level the measures will be applied i.e exactly what is the unit of assessment (individual, programme, department, university or all of them). and 2) how do we measure distinguish TEF measures from those of the emerging new system for quality assurance.  A third important issue is that , now the government has linked TEF to funding, any outcome of TEF has to be able to withstand judicial review, - this tends to foce the process down the metrics route. I have been reliably informed that adding 'Impact' to the REF cost 60 million, mostly cost associated with making it sufficiently robust to withstand judicial review.   So apart form the simple question of what teaching  excellence  really means ( and I do wonder if it is about excellence or being good enough) there are some other interesting questions to be resolved.

Regards

Ian 

On 22 July 2015 at 21:24, Joelle Fanghanel <[log in to unmask]> wrote:
Dear all

Tried to join the conversation at an earlier stage but technology did not let me. In the HEA project on SoTL (not sure this concept appeared in our conversations as I am following this remotely) we are presently doing (with Bristol, Keele and Brighton) one of our recommendations is (in accord perhaps with Chris and Gwen?) that we consider something like the Sydney Index? Qualitative types of indicators alongside the ‘hard’ data which no doubt will be used. It would be good to hear the views of SEDA colleagues. 
Joelle
Professor Joëlle Fanghanel
Associate Pro Vice-Chancellor (Academic Development and Scholarship)
Head of the Graduate School
 
University of West London |Room A400|St Mary’s Road | Ealing|W5 5RF |UK
 

Editor of New Vistas

Past President |International Society for the Scholarship of Teaching and Learning | http://www.issotl.com
Co-Chair Heads of Educational Development Group UK
My book:
 
 HEA Project Defining and Supporting SOTL

From: "[log in to unmask]" <[log in to unmask]> on behalf of "Lea, John ([log in to unmask])" <[log in to unmask]>
Reply-To: "Lea, John ([log in to unmask])" <[log in to unmask]>
Date: Wednesday, 22 July 2015 15:46
To: "[log in to unmask]" <[log in to unmask]>
Subject: Re: TEF thoughts

That’s a great list, Chris, but I fear we will need to be on guard all the way through the consultation:

 

On 1, I suspect that it will not be your list here, but the current HESA list, which includes qualifications and awards which are not HE focused – big deal; not sure?

 

On 2, this will be tested by how much the REF continues to pull people in the direction of REFable pedagogic publications, which could still exclude lots of the scholarship associated with L and T.  Although anything that might bring L and T committees closer to research committees would be no bad thing in my book.

 

On 3, I was involved in teasing out the usefulness of some of the NSSE questions in the UK, so naturally I would like to see this, but this really would need lobbying for because the questions are more focused on student performance than teacher performance – which is very HE as I see it, but perhaps not as our current audit culture in the UK tends to see it.  And, we would probably have to have this sit next to NSS given how much has been invested in that – unless we can get a two part NSS – where the second part focuses on these engagement matters.

 

On 4, I like this idea, and it will be intriguing if it throws up the irony that Arum and Roska noted in their use of these tests – that some of the high fee elite universities in the US seem to be producing a kind of unwritten `dis-engagement contract’ (Kuh), where in exchange for leaving faculty alone to get on with research, the students are not challenged too much, knowing that they will get elite jobs because they went to elite schools.  Sorry, that’s a very crude summary of only part of a whole book.  But I do think this has a lot of mileage if it could somehow be linked to a value-added index – i.e. what students came in with, compared with what they exited with, and even more intriguing if it could be linked with the achievements of so-called non-traditional learners.

 

On the UKPSF – please, let’s not let that get hi-jacked into becoming part of a competency approach to anything.

 

Cheers

 


John


John Lea

From: Online forum for SEDA, the Staff & Educational Development Association <[log in to unmask]> on behalf of Chris Rust <[log in to unmask]>
Sent: 22 July 2015 14:59
To: [log in to unmask]
Subject: Re: TEF thoughts
 
I don't think it is actually that difficult George to come up with possible metrics that, while blunt and imperfect, would be meaningful (and valid), e.g.:
- the number of staff with PG Certs, HEA Fellowships, NTFs, etc., 
- number of pedagogic papers published
- student engagement, measured by the UK version of the NSSE, currently being developed and piloted
- before and after measures of say, critical thinking skills using the CLA possibly, or use of AHELO

And however imperfect, I find it hard to believe that it would be a bad thing if institutions were trying harder to score well on such measures!

best wishes
Chris

Chris Rust
Professor Emeritus 


On 22 Jul 2015, at 14:04, George Roberts wrote:

I am not against thought experiments and agree with Chris Rust that proactive engagement is better than grumbling after the fact.
That said, I find it very hard to come up with metrics that might be meaningful (or "valid"). But I do not think that is the intent, however much we might wish it were.
We in educational development at Oxford Brookes have long taken a phenomenographic, reflective and developmental approach to teaching improvement, very much not a competency-based approach. We believe that it leads to better teaching and our NSS and NESE scores might just possibly provide some slight evidence that at least we are not doing too much harm to too many learners. Our BME attainment figures however do call some certainties into doubt.
One could reasonably easily come up with a competency model of teaching - not unlike that used in ITE in Schools of Education - which could be used both individually as well as being aggregated across the institution and link it to the UKPSF. This could be used ("administered": mandated nationally and implemented locally) and correlated with other measures of "educational added value" such as entry tariff and exit scores. Whether this is self-inflicted or whether an OFSTED-like inspectorate of HE Teaching is established is an open question. But, imagine the VC whose HEI were put into "special measures".
I could see using an instrument of some sort along side our phenomenographic approach to professional development. If it helped our attainment figures improve that couldn't be a bad thing.
But would it?

George

Dr George Roberts
Principal Lecturer, Educational Development
Oxford Centre for Staff and Learning Development
Oxford Brookes University
[log in to unmask]

+44 7711 698465
@georgeroberts

Have you seen:
Brookes Open Online Courses: http://www.moodle.openbrookes.net/

On 22 July 2015 at 11:20, Ridall, Claire <[log in to unmask]> wrote:

There’s a group discussion board on the SEDA LinkedIn page that everyone could contribute to?  https://www.linkedin.com/grp/home?gid=8346473

 

I’m also on the Services and Enterprise Committee so would be happy to discuss with the group and liaise with the Exec chairs as appropriate?

 

Kind regards,

Claire

 

From: Online forum for SEDA, the Staff & Educational Development Association [mailto:[log in to unmask]] On Behalf Of Lea, John ([log in to unmask])
Sent: 22 July 2015 11:13
To: [log in to unmask]
Subject: Re: TEF thoughts

 

What do I know about social media?  But I do like open forums (fora? – damn, I knew I should have listened more in Latin class) so it would be good if everyone on this list had a chance to contribute.  What about some kind of drop box – open to everyone – where we can put our ideas for workable/acceptable metrics – which someone (or a group) at SEDA then works up?

 

Again, what do I know?  Is there a 6 year old on the list who could help here?

 

Best

 

John

 

From: Online forum for SEDA, the Staff & Educational Development Association <[log in to unmask]> on behalf of Burden, Penny <[log in to unmask]>
Sent: 22 July 2015 10:19
To: [log in to unmask]
Subject: Re: TEF thoughts

 

Just to let everyone know, the SEDA Exec has been discussing this and are planning to make a response.   I have been lurking with interest because of my role on the Exec – I think we will be responding in the light of this thread so watch this space.   We have scheduled time at the end of July (i.e. from now) to consider this and I would expect the Chairs, Pam and Yaz will co-ordinate on behalf of the SEDA community – and communicate accordingly.

 

Hope this helps for now.

 

Penny

 

Penny Burden, SFHEA

Associate Professor and KAPS Director

Centre for HE Research and Practice

Kenry House

Kingston Hill Campus

Kingston University

 

Tel:  0208 417 5589

Ext:  65589

 

https://staffspace.kingston.ac.uk/dep/cherp/Pages/default.aspx

 

 

From: Online forum for SEDA, the Staff & Educational Development Association [mailto:[log in to unmask]] On Behalf Of Chris Rust
Sent: 22 July 2015 10:11
To: [log in to unmask]
Subject: Re: TEF thoughts

 

I think the SEDA Exex should set up a working group - they've done it before on important national issues

Chris

 

Chris Rust

Professor Emeritus 


 

On 22 Jul 2015, at 10:08, Jason Davies wrote:

 

Dear Chris, Dilly & everyone

a) as a community we know a lot and should collectively be clever enough to develop a set of metrics that would have at least some validity


Shall we do it then? Who's in? Do we want to set up some kind of list or shared resource to use? I suspect this list is not the right place...

--

Dr Jason Davies
http://www.ucl.ac.uk/~ucgajpd/Academic/

 






--
Dr Ian Scott, Associate Dean Student Experience, Faculty of Health and Life Sciences
Oxford Brookes University
Marston Road
Oxford
OX3 OFL
0044 1865 48 2638



Have you seen

Scott, I. and Mazhindu, D. (2014) Statistics for Health Care Professionals: An Introduction (2nd Edition). Sage 

Scott, I. and Spouse, J. (2013) Practice based Learning in Nursing, Health and Social Care; Mentorship, Facilitation and Supervision, Wiley

Ely, C and Scott, I. (2007) Essential study skills for Nursing, Elsevier