Dear all,
Below is the post requested by a considerable number of you. Am
grateful for the interest.
Bill C.
A note on correlation between career success and prior academic grade
=====================================================================
(by Bill Cranston, Emeritus Prof, University of Paisley - placed in
public domain, November 1999 - Comment and discussion welcome to
<[log in to unmask]> (note that 0 is the numeral zero))
1. INTRODUCTION
===============
About seven years ago, I attended a meeting of Heads of Polytechnic
Engineering Departments in the UK. A representative of our Engineering
Council was attempting to defend their proposed (and now established)
policy of only approving courses taking in students with high (upper
quartile) 'A' level scores. One of the key grounds for this was that
the Polys were 'obviously' producing inferior graduates since they were
taking in students with low grades. I tentatively suggested that before
pressing this policy, the Engineering Council should do a random survey
of their Chartered Members to establish:
a) 'A' level points score achieved prior to University entrance.
b) University (or Polytechnic) attended.
c) Grade of degree obtained.
d) Career progress at 5, 10 and 15 years after graduation.
The speaker thought this would be a good idea. But nothing came of it.
It never occurred to me at the time (at that stage only three years back
into academia after a 26 year 'break' in industrial research) that any
work might actually have been done in this area - and none of the 50 or
so UK heads of department present mentioned anything to me afterwards.
So I was surprised some 4 years later (1996) to come across work on the
correlation between engineering competence of American trainee engineers
assessed in the workplace and their prior academic grade in a Carnegie
Foundation Bulletin of 1917. A year further on, I discovered, browsing
on the library shelves at Paisley University, a 1991 paper by Professor
Goldschmid of Lausanne on the linkage between training and professional
success of engineers. (It is a pity that I had not discovered it before
going to that HOD meeting!)
Then in December last year, on a trip to Wales to attend my 65th
birthday celebration, hosted by our eldest son and his wife, I stopped
off at Hay-on-Wye specifically to buy second-hand books by and about
Father Pierre Teilhard de Chardin. Having overspent my budget, I was
reduced to looking at the throwaway (i.e. 50p each!) volumes held in
leaky cabinets outside the former Hay-on-Wye cinema. Here I came across
Professor Valentine's book on 'The Reliability of Examinations',
published by University of London Press in 1932 - a bit mildewed, but
still legible.
And just a week ago (October 1999), I visited Glasgow University
Library, to browse their holdings of journals on medical education. My
objective was to assess the development of problem-based learning in
medical education world-wide. There I came across a 1980s paper in an
American Medical Association journal describing work similar to that
presented in the Carnegie Bulletin of 1917; medical students in
residence (i.e. under training in hospitals) had been assessed for
competence, and their ratings correlated with academic grades.
Why is trivial detail about the discovery of these references being
given, you may ask? I wanted to underline just that fact - they were
discoveries. So they represent a random sample of what is available out
there.
Let's see what they say.
2. THE MAIN REFERENCES (2.1 to 2.4)
===================================
Each reference is listed. The key features of the work are indicated.
The specific evidence on correlation between career success and prior
academic grade is then summarised. Finally some personal remarks ('BC
Comment') are added.
2.1. REFERENCE 1 - CARNEGIE 1917
--------------------------------
A STUDY OF ENGINEERING EDUCATION, prepared for the Joint Committee of
Engineering Education of the National Engineering Societies, Bulletin
Number Eleven of the Carnegie Foundation for the Advancement of
Education, New York City, 1917, pp 139.
2.1.1. Key features - Carnegie
-------------------------------
The study was carried out by Professor Charles R Mann of Chicago
University. It arose out of action by a joint committee on engineering
education, representing the principal engineering societies in America
at that time (Civil, Mechanical, Electrical, Chemical, Mining and
Engineering Education).
The key concern underlying the study was that the 4 year curriculum of
engineering study, consisting of two years mainly in the fundamental
sciences, and then two years mainly in applications of these sciences
(copied from the French almost a century earlier) had now become heavily
overloaded. As a result (quoting):
'There is a widespread feeling that under this pressure the great body
of students fail to gain, on the one hand, a satisfactory grounding in
the fundamental sciences; and on the other hand, do not fulfil the
expectations of engineers and manufacturers in dealing with the
practical problems with which they are confronted on leaving the
engineering schools.'
The major recommendation was that theory and practice should be taught
together - a reform that is only now being implemented in the USA, some
80 years later!
2.1.2 Career/grade correlation - Carnegie
------------------------------------------
First of all in regard to grades it was established that about half the
graduates of engineering schools received low grades in physics,
calculus and mechanics. The question was - were graduates with these
low grades also low grade engineers?
This was assessed by looking at the records of 168 graduates taken on in
1913 by the General Electric Company of Schenectady. On these company
records each man (no women engineers in those days!) was rated by each
of the foremen under whom he worked as A, B, or C in each of five
qualities, as follows:
Technical Ability
Accuracy
Industry
Ability to push things
Personality
The date of the rating is not given but would be 1916 at the latest,
i.e. when the graduates were still under training. Through the courtesy
of the engineering schools they had attended, copies of the full college
record of these same men were secured. An extended study of these two
sets of records was made by Professor E L Thorndike of Columbia
University. This showed that the correlation between the two was very
slight; that is, that ability to score high grades in college was no
indication of ability to meet the requirements of the General Electric
Company. On the other hand, the college grades signified something,
since the grades for the senior (4th) year correlate closely with the
average grade for the entire course, showing that ability to score high
grades in college is generally a stable and permanent characteristic of
an individual. A similar study was made in the Westinghouse Electrical
and Manufacturing Company of Pittsburgh of a group of 40 college
graduates. The results were practically the same.
2.1.3 BC Comment - Carnegie
----------------------------
Since 'Ability to push things' and 'Personality' are unlikely to
correlate with performance in physics, calculus and mechanics, some loss
in correlation is to be expected. But one would expect 'Technical
Ability', 'Accuracy' and 'Industry' to be cultivated in college.
Something more than a 'slight correlation' would be expected.
That cohort of American engineers, starting work in 1913, would be in
mid career in the early 1930s. It is perhaps worth noting that the
Spitfire, although a brilliant UK design, had to be equipped with
American Browning guns, and only got into the air in quantity courtesy
of machine tools also imported from America in the 1930s. While this
badly dents a boyhood myth of mine about British engineering, it does
indicate that American engineering education must have been doing some
things right! Something to keep in perspective.
2.2 REFERENCE 2 - GOLDSCHMID 1991
---------------------------------
Goldschmid, M L, 'The Training and Professional Success of Engineers: An
Empirical Study', International Journal of Applied Engineering
Education, Vol 7, No 6, pp 440-443, 1991.
2.2.1 Key features - Goldschmid
-------------------------------
Over 800 engineers and architects who graduated from the Swiss Federal
Institute of Technology in Lausanne between 1946 and 1987 completed an
extensive questionnaire. It focussed on the following variables:
* university education (type of study, reasons for choice of discipline,
satisfaction, etc)
* post-diploma training (duration, form, content, number of courses,
etc)
* professional activity (tasks, choice of employment, work conditions,
problems at work, etc)
* personal qualities (interpersonal, leadership, ability to communicate,
energy, etc0
* life events (marriage, children, friendships, etc)
* satisfaction with career ('subjective' index of professional success)
* demographic variables such as age, gender, etc
* 'objective' indices of professional success (income, number of
subordinates, responsibilities, and status)
2.2.2 Correlation career/grade - Goldschmid
-------------------------------------------
None of the objective indicators of professional success were found to
be significantly correlated with the overall grade point averages
recorded during studies at university. Specifically:
Number of subordinates (r = 0.06)
Income (r = 0.09)
Decision making power (r = -0.02)
Responsibilities (r = -0.03)
Satisfaction with work (r = 0.05)
Importance of success (r = -0.04)
By far the most important reasons given by respondents to explain their
professional success were personal qualities (18% of respondents chose
it) and perseverance at work (15% of respondents).
Particular 'nuances' found were that there was a significant correlation
between high grades and appointment as a university professor. Also,
independent people, i.e. people who owned their firms or businesses, had
significantly *lower* grades.
A general comment is made in the conclusions:
'The criteria for evaluating performance in academia obviously do not
correspond to those used in industry'.
2.2.3 BC Comment - Goldschmid
-----------------------------
Goldschmid subsequently extended his work to cover doctors,
psychologists, lawyers and economists - in total over 2600 graduates
from French-speaking Swiss Universities.
In this later investigation, again supported by the Swiss National
Foundation for Research, a personality test (the California
Psychological Inventory) was used with a subsample distributed between
the four professional groups, enabling the particular personality traits
of the different professional groups to be studied.
A paper on this work ('Academic Training and Professional Success of
University Graduates: An Empirical Study') was presented to the 17th
International Conference on Improving University Teaching in Glasgow in
July 1991. (Despite my strong interest in learning and teaching I (and
how many other local academics?? . .) was quite unaware of this event.)
As for engineering, no correlation between career success and grade was
found for these other professionals, whether assessed on an objective or
subjective index.
In regard to the personality tests it emerged that the profiles
associated with high academic performance were *very different* from
those correlated with high professional success . . .)
2.3 REFERENCE 3 - VALENTINE - 1932
----------------------------------
Valentine, C W, 'The Reliability of Examinations', An Enquiry with
Special Reference to the Entrance Examinations to Secondary Schools, the
School Certificate Examination, and the award of Scholarships to
Universities, University of London Press, 1932, pp 196.
2.3.1 Key features - Valentine (and Schuster - 1907)
----------------------------------------------------
The book deals predominantly with secondary school and university
performance, as that relates to prior performance in entry examinations.
A low correlation is observed. This is also observed with present day
correlations between 'A' level results and degree grade. (Two of us at
Paisley University are currently updating a note I sent two years ago to
the Engineering Council, with a view to presentation at a conference in
2000.
But for our purposes there is a reference on p.35 of the book to work by
a Dr Edgar Schuster, who is quoted as having shown:
'that in the professions of the Law, of the Church, and of Teaching, a
decided superiority in professional status is observed for those who had
obtained first-class honours at universities as compared with those who
had only obtained third-class honours (1)'
The footnote reads:
(1) See 'Eugenics Laboratory Memoirs', 1-5 (London, 1907). Dr Schuster
blended his figures for Teaching and the Church, and found that
'distinguished positions' (Bishoprics, Deaneries, Professorships,
Headships of Public and First-class Grammar Schools, etc) were obtained
by 68% of the first-class honours men within the whole group, but by
only 32% of the third-class honours men. One would, however, in any
case expect a close relation between high honours and such posts in the
teaching profession. In the Law the figures were less striking, 46% of
the first-class graduates gaining positions of distinction, against 22%
of the third-class honours men. In all, 3,508 men were traced under
'Church and Teaching', and 634 under 'Law'.
2.3.2 Correlation career/grade - Valentine/Schuster
---------------------------------------------------
Third-class honours men still seem to me to be doing pretty well -
since, as I understand it, a third-class award in those days gave no
indication of academic performance - it was really no more than an
attendance certificate. So the first-class honours men are perhaps not
getting such a good return on their efforts as all that!
But some evidence of correlation has been established - that must be
accepted - but since it must refer mainly to graduates from the 1890s
and before, its relevance to today must be questionable. What is also
interesting that if any work had been carried on on this topic in the
period 1907 to 1932 in the UK, Valentine at least was not aware of it.
2.3.3 BC Comment - Valentine
----------------------------
As an aside, I found his explanation of numerical correlation very
helpful (statistics is not my strong point). His footnote on p.57
states:
'Briefly we may say that if two orders A and B, resemble one another
completely (the pupils being in the same order in B as they are in A)
then their correlation is 1.0. If the orders are exact opposites, so
that the top boy in A is bottom in B, the second in A bottom but one in
B and so on, the correlation is -1. If there is no more resemblance
between the two orders that would be given by mere chance, the
correlation is 0.'
On page 74 an example is given of a class of 21 boys where their order
in the entry exam is compared with their order in the School Certificate
examination four years later. The correlation is 0.46. The difference
in the orders are compared graphically in a strikingly clear way. He
states:
'that even the correlation of 0.46 is consistent with a few striking
changes in order. Note especially:
(a) That of the two boys who were 14th equal on entry, one rises to
3rd and the other drops to 20th, and
(b) One boy who was 8th equal at entrance drops to the bottom of
the class, while the 18th rises to 8th.'
Put another way, 4 boys in that group of 21 (20% of the total) performed
very much lower or higher than their entry scores would have suggested.
2.4 REFERENCE 4 - MEDICAL RESIDENTS - 1980'S
--------------------------------------------
I discover from my notes that my reference notes are incomplete. (I
will be returning soon to Glasgow University Library, and will be noting
precise details at that time.)
Journal - Journal of the American Medical Association (AMA), 1980s
Title - 'Medical School Achievements as Predictors of Residency
Performance'
Authors - Kathleen J Yisha (?), Paul S Rosenfeld M.D., Michael B
Donnelly, Ph. D.
I recall that the work on which the paper was based was done because of
the growing competition between US medical school graduates to get a
hospital residency, because of restrictions in places. The practice was
apparently also growing of choosing candidates for residency purely or
dominantly on the basis of grades achieved in medical school.
I have copied the conclusion of the authors that:
'The use of medical school achievement as a single selection factor or
as a method of screening out applicants for further consideration is, at
best, justified only in the case of extremely high or low medical school
achievement'
3. CONCLUSION AND DISCUSSION
The most thorough work described above is that of Goldschmid. It
indicates no correlation between career success and academic grade. One
of the two older references (Carnegie) backs this up, while the other
does indicate some correlation. Finally a modern study of American
doctors in residence (i.e. at the start of their career) shows some
correlation.
Some further work I am currently looking at into career progression of
engineers in Germany does not consider academic grades at all,
indicating again that it is not of much relevance. A key factor is the
level of opportunity and encouragement offered in initial employment
(backing up a finding by Goldschmid).
While the data is limited, I think it virtually certain that in at least
80% of our academic courses, the correlation between career success and
academic grade is small or non-existent. What also seems clear is that
there is currently no data which demonstrates a *strong* correlation.
The key questions therefore to be asked are:
a) What factors do then govern success in a career?
b) How far can these factors be brought into academic courses?
In regard to a), Goldschmid's work gives some strong clues. Already
mentioned above are personal qualities and perseverance. He also
mentions luck - there's not much any of us can do about that! Events
occurring during a professional's career are also important. In all
careers studied contact with and assistance from an 'influential person'
in the profession is an important ingredient of success. Some
differences emerge, in that a stay abroad is likely to be helpful to
economists, and interestingly (for me as a Christian) it seems that a
change in interior or spiritual life is helpful for psychologists.
As far as b) and academic courses are concerned, it is a very wide
question. But it would appear to be essential to have some elements of
group work in all courses, accompanied by some instruction in group
dynamics, with reflection required by students on the dynamics in their
particular groups. And that almost certainly means group dynamics and
group teaching by staff. It is interesting that this is being
introduced in some courses as part of the reform of engineering
education in the USA. To encourage perseverance, it would seem
essential to require a fair amount of student work to be submitted in
draft form, and to insist on several submissions until the standard is
acceptable. And it would be nice if some way be found to reward
students who persist conscientiously with subjects they find really
difficult, or even boring.
5 CONCLUSION
============
I cannot do better than quote Marcel Goldschmid's final remark in his
conclusions to Reference 3:
'Further research is called for, especially to better elucidate the
transition from academia to professions and to determine the long term
effects of university instruction.'
None would appear to be in hand at this time.
6. REFERENCES
=============
These are given in the text
--------------------o0o-------------------------
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
