A draft of the following was mailed to those who replied to my earlier
ALLSTAT message on the revised training guidelines for students funded by
the UK Economic and Social Research Council (cf www.esrc.ac.uk).  As the
timescale for comments appears very short (they expect to draft their
definitive document in December), the amended text is circulated here. 

Unless a substantial number of people contact me with objections, I
propose to submit this document to the ESRC as representing "a substantial
body of professional statistical opinion", with myself as the conduit
rather than the instigator. I am unable as yet to report whether the RSS
has been consulted, but request that comments on the following be sent to
me before 13 November. 

--------------------------------------------------------------------
ESRC Postgraduate Training Guidelines: a response to the section on 
quantitative data analysis


The following comments were initiated when one of us was contacted by an
academic department to suggest an appropriate response to the proposed
Framework for Research Methods Training, in particular to the section on
quantitative analysis and statistical methods.  The comments were
circulated to the ALLSTAT email list, which is run from MAILBASE and has
over 2500 members, and were refined with responses from list members.  As
such, the following represents a consensus from a large group of qualified
and practicing statisticians, in the UK but also from many other countries. 

As a preface to the specific remarks, we welcome warmly the 
requirement that students should be able "to understand the significance 
of alternative epistemological positions that provide the context for 
theory construction, research design, and the selection of appropriate 
analytical techniques."  By contrast, the sections on statistical methods 
then seem prescriptive and out-of-date.  That paragraph was quoted in its 
entirety in the first circulation on ALLSTAT.

The major problem is that the prescribed syllabus lists topics in ways
that would have been found in most introductory books of mathematical
statistics of the 1960s and 70s.  The unfortunate effect of such teaching
has always been to convince most students that statistics is (a) an
obscure branch of mathematics, (b) based on a set of tests expressed in an
indecypherable language, and hence (c) best approached by blindly
following a cookbook formula.  Such students, for example, present the
statistician with questions of the type, "My data are not normal, so I
guess I have to use Kruskal-Wallis anova." 

Alternative paradigms of data analysis, seeing it as a detective exercise
in teasing out features and as an exercise is matching theory with
observations, have been proposed and widely used since the 1970s.  These
include Exploratory Data Analysis (EDA: Tukey), Generalized Linear
Modelling (GLM: Nelder), Multilevel Modelling (MLM: Goldstein), and
Structural Equation Modelling (SEM: Joreskog).  The Council may argue that
such techniques are appropriate for the "more advanced levels of
competence" referred to in general terms.  In our experience, the damage
wrought by many introductory courses has the effect of deterring most
students (and staff) from tackling 'advanced' topics. 

The factor that has most changed perceptions of data analysis since the
1970s has been the emergence of powerful computers as everyday and instant
tools.  It is therefore essential in training to focus on the appropriate
and safe use of the computer, and on understanding the assumptions and
limitations of analyses, rather than the internal details that were
relevant to hand calculations. It is *especially* important to stress the
continued relevance of numerical analysis, to counter the growing menace
of researchers who assume they can "program the whole analysis in a
spreadsheet and not bother with statistics packages." 

Some words and phrases in the proposed framework caused particular alarm.
Taking them in the order they appear, they were:

* "Frequency distributions ... and measures of asymmetry" - this seems a
direct throwback to the mathematical syllabus. While all statisticians are
familiar with ideas of asymmetry, the specific measures are of little
practical use.  Other terms such as bi-modality, mixture distributions,
etc would have the same (limited) relevance to introductory training.  On
the other hand the detection of outliers would be a topic of very high
importance in teaching the practical use of distributions; data censoring
is another useful concept not currently taught.  The useful learning
outcome should be that the student understand the concept of variation,
and not equate "error term" with faulty procedures or experimental
incompetence.  Sampling needs to be taught in a context of answering 
research questions, not as a Platonic exercise with urns. 

* "Graphical methods" - while welcomed, this term is so vague as to be 
meaningless within the syllabus.  While all educational bodies pay lip 
service to the use of graphs and graphics, the level of graphical 
literacy among students and academics (as defined, for example, in books 
by Tufte and by Cleveland) remains woefully low.  A module in Graphical 
Interpretation of Data is taught at Hull and draws attention to features 
of graphics use that are ignored or misunderstood by almost everyone.

* "Conditional probabilities ... and Bayesian theory" - again, the first
term smacks of the formal mathematical syllabus while the last is too
vague to be interpretable.  Are students to apply formal Bayes' analysis,
or simply to be made aware that there exist statisticians who would not
accept their P values?  Should students be briefed on the Appeal Court 
ruling that it was improper for Bayes' Theorem to be adduced in a court of 
law?

* "Methods of statistical inference including the normal distribution" -
the normal distribution is not a method of inference, but that seems just 
a lapse of punctuation.  The phrase that follows does, however, invite 
the trainer again to focus on the mathematics of "statistical tests, 
confidence levels, and statistical power."  In our view, this emphasis 
should be completely reversed, with the syllabus couched in terms of 
'formulation and testing of appropriate models.'  Such models (eg GLMs) 
explicitly require consideration of error distributions other than normal.

* "Two-way contingency tables" - these may be a useful presentation of 
final results, but in real data analysis are invariably hopelessly 
inadequate.  The syllabus might, however, suggest log-linear modelling 
with some guidance on the sophistication that should be reached.

* "Trend tests" - ditto.  One wonderful example was in a Lotus 1-2-3 
manual, with data showing retail sales for Mon, Tues, Wed, Thurs, Fri and 
extrapolated to Sat and Sun.  At this level, it is probably enough to 
make students aware that special considerations attach to time-dependent 
and other particular types of data.

* "Correlation coefficients" - ditto, unless the subtext of the syllabus 
is the assumption that any competent trainer *will* warn students against 
quoting any correlation without examining the scatterplot.  (cf graphical 
methods)

* "Multivariate regression" - ALLSTAT respondents assumed that this was 
intended to be "multiple regression", but worried that the committee was 
unaware of the distinction.  We note, however, that multiple (linear) 
regression is a tiny subset of the range of fittable models, and achieved 
prominence in the 1960s mainly because of its mathematical tractability 
and ease of computation.  

Conclusion

The proposed framework appears curiously dated and restrictive when
compared, for example, with the content of Scarbrough and Tanenbaum's
"Research Strategies in the Social Sciences", though that book is arguably
more appropriate at the higher-level modules in subsequent years. It
might, however, be used at an introductory level for its worked examples
which show domain-interpretation; this would indicate to students the
appropriate depth of analysis and motivate their study of underlying
principles.  The case for moving "From statistics to statistical science" 
was spelled out in Professor Nelder's paper of that title in the Journal 
of the Royal Statistical Society, 1999, Vol 48 part 2.  Full references 
to the other authors named above can be supplied. 

We conclude that the recommended academic content for research training in 
this area must be drastically revised; without such revision it will not 
be recognised as relevant by the target community. 

Collated by:
R. Allan Reese MA MSc CStat          Email: [log in to unmask]  
Associate Manager                    Direct voice:   +44 1482 466845
Graduate Research Institute          Voice messages: +44 1482 466844
Hull University, Hull HU6 7RX, UK.   Fax:            +44 1482 466846
====================================================================
The views expressed were approved by a wide variety of statisticians
and other researchers though an email consultation, but should not be
assumed to represent an official view of any part of the University of
Hull.








%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%