Dear Katya,



One of my colleagues is an expert in this area. Please contact me off line if we can be helpful.



Best wishes,



Fiona

---

Sent via BlackBerry



-----Original Message-----

From: Katya Hosking <[log in to unmask]>

Date:         Wed, 27 Jun 2007 09:57:46 

To:[log in to unmask]

Subject: Data analysis for impact assessment: summary of responses



Dear all,



Responses to my question about data analysis lead me to ask another: In preparing for impact assessment do you have, or have access to, someone with expertise in statistics or data analysis?



Here is a summary of responses (my original email is at the bottom).  Progress ranges from not-yet-started to finished-and-published, but at some universities people are definitely finding it difficult to extract the relevant data from their internal systems.  From informal exchanges, one problem seems to be that the section or department within the university which manages the database sometimes sees itself as a gatekeeper, and unless that section is convinced of the need for E&D monitoring it will not facilitate it.  Another problem seems to be lack of knowledge about databases and statistics resulting in a lot of false starts, or discussion about the relevant fields, before the right data emerges.  



1. Two universities had not yet begun tackling student data.  One added that they had had no problems extracting data from the staff system.



2. For UG admissions, one university purchases the UCAS conversion data set which can be downloaded periodically.



3. "At xxxx impact assessment is carried out by the Department that holds central responsibility for the policy (as opposed to those responsible for implementing it), following training and with support from the Staff and/or Student Equality and Diversity Adviser.  The lead assessor for the Assessment Policy is the Senior Assistant Registrar (Recruitment and Admissions). 



"I have just received a first draft IA of our Admissions Policy.  The data used for this was applications data held on the university admissions database.  This predominantly relates to undergraduate & PGCE applications only, but does also include some postgrad applications info.  With the help of a statistician the assessor has looked at applications to offers (conditional & unconditional) by gender, age, disability and ethnicity. The statistician carried out a chi square test of association on the faculty data to identify whether or not there was an association between offer rates and demographic characteristic. Consideration was also given to looking at who has applied to study at the university, but it was decided that this was more relevant to the Widening Participation Policy than the Admissions Policy.



"As far as I'm aware the raw data wasn't difficult to get hold of (I deal with staff data only, but I gather that our student data system is fairly good).   However, there is the issue that no info was available

on most postgraduate applications meaning that there's a potential gap in the draft IA.  



"The IA has not yet been finalised, but the limits of statistical analysis are acknowledged within the draft, particularly in relation to difficulties with small variances and to statistical significance findings within a large sample. Attention is also drawn to the need to compare university data with national benchmarks of educational attainment/HE participation, in order to put any findings into the national context."



4. "I have also started to ask some questions around the information that we have on the students that apply to us and have also been told that the data may not be readily available.  I deal mainly with staff issues but also increasingly with students so it something that I will need to pick up on.  So no real answer to your questions but just to let you know you are not alone."  



5. "In undertaking our initial data collection work on this we have used both internal data (essentially the enrolment data for first year students as of 1.12.07) which is generated from our internal student record system, but a more substantial analysis of the admissions process was undertaken from data from UCAS (I understand that this is supplied on CD to all university admissions officers annually in about February, for the previous years admissions process year).  This included data on applications, offers and acceptances (but not on all equality variables)."  This university has published the relevant reports and also the initial screenings.



6. "We have had no difficulties in getting the relevant data from our student records system. We have looked at ethnicity, gender, disability and age.  Age was the most difficult area because of very small numbers in the older age groups.



"A number of people have been involved in analysing the data including the Head of our disabled student support services and our Impact Assessment Co-ordinator. We have been able to identify areas where there are statistically significant differences and this information will be used when addressing policy and practice."



Best wishes,

Katya



>>> Katya Hosking <[log in to unmask]> 13/06/2007 11:29:14 >>>

Dear all,



We're currently conducting an impact assessment of admissions, and it has proved surprisingly difficult to extract and analyse the relevant data from the student information management system.  There seem to be various reasons for this, some of them technical ones which I don't pretend to understand.  I'm wondering whether this is a common experience across the sector, or whether it is something to do with our particular information management systems.



Questions:

What (quantitative) data have you been able to analyse for use in impact assessments?

Has it been difficult to get hold of the raw data, that is, to extract it from the system?

Who has been responsible for analysing it, and have they encountered any major problems?



I'd be very interested to hear your answers, and will compile a summary for the list if anything useful emerges!



Best wishes,

Katya



_______________________________________________

Katya Hosking  (029) 2087 9218

Accessible Curriculum Officer

Registry, Cardiff University

30-36 Newport Road, Cardiff CF24 0DE

www.cardiff.ac.uk/learning/themes/access/index.html