I agree with Mike (and Alan) that there are many examples (at least in the
transport sector) where policy forecasts have been demonstrated to agree
fairly well with reality ... but there are many more where they have not
... and this issue relates to the “length” (horizon?) of the forecasting
period. We normally teach our students that the worst errors in forecasts
are NOT due to the models being totally off the mark, but because the
planning variables used in the models for the design year are.

For example, it was shown many years ago that if the forecasts of
population, car ownership and income were replaced by their true values in
the design year (20 years after), transport model forecasts made in the
base year had very reasonable errors (i.e. less than 5%) ... whence the
forecasts made 20 years previously had errors well over 100%. Now we have
more confidence because our models are much better than 30 years ago.

Finally, Mike’s example has one flaw in my opinion. All the models tested
in the ISGLUTI study suffered from being inconsistent (in the sense of not
having a proved convergence), therefore it was actually too much to ask of
them that they would perform well in the tests.

If you do not believe that a model will produce sensible forecast it would
be rather uncomfortable to use it, but even if the forecasts have a
certain degree of error (and they must, always), models may be useful to
compare alternative policies ... I am assuming that the mental models we
all have will perform worse than a sensible model based on a reasonable
theory, but I am sure this was not the question.

Juan de Dios Ortuzar

De: News and discussion about computer simulation in the social sciences
[mailto:[log in to unmask]] En nombre de Michael Batty
Enviado el: miércoles, 29 de abril de 2009 3:14
Para: [log in to unmask]
Asunto: Re: [SIMSOC] any correct policy impact forecasts?-- a clarification

I was at the meeting where Scott raised this issue. Alan Wilson said that
his company GMAP was built on developing spatial interaction models for
predicting short term shifts in retailing activity which routinely
produced predictions that were close to the mark. There are no better
examples than the large retail units that routinely - every week - run
their models to make predictions in retail markets and reportedly they
produce good predictions. These are outfits like Tesco, Asda, M and S and
so on. I cant give you chapter and verse of where these predictions have
been verified and documented because I am an academic and dont have access
to this sort of material. The kinds of models that I am referring to are
essentially land use transport models which began in the 1960s and are
still widely used today. Those people reading this post who arent familiar
with these models because they are not agent based models can get a quick
view by looking at my early book which is downloadable from our web site
www.casa.ucl.ac.uk or directly from
http://www.casa.ucl.ac.uk/urbanmodelling/.

I think that the problem with this debate is that it is focussed on
academia and academics don't traditionally revisit their models to see if
longer term predictions work out. In fact for the reasons Alan says one
would probably not expect them to work out as we cant know the future.
However there is loads of evidence about how well some models such as the
ones I have referred to can fit existing data - ie in terms of their
calibration. My book and lots of other work with these models shows that
can predict the baseline rather well. In fact too well and the problem has
been that although they predict the baseline well, they can often be quite
deficient at predicting short term change well and often this arises from
their cross sectional static nature and a million other problems that have
been raised over the last 30 or more years.

There was a study done in the late 1980s by the then Transport and Road
Research Laboratory comparing land use transport models built by various
groups around the world all with the same structure - models built by
Putnam in the US, Mackett then at Leeds now UCL, Echenique at Cambridge,
Brotchie and Sharpe at CSIRO, and Wegener at Dortmund . This study showed
that models with a similar structure when they were transferred to
different places and rebuilt along side the models of those places led to
massive difference simply in terms of the way data was define, programs
written and so on - ie the issue was that the personal knowledge and
idiosyncracies of the model builders got in the way of any true
comparisons and also the way the data came, differences in official data
definition and so on made comparisons almost impossible. This is why in
the land use transport model field it is much rarer to find generic
software as the local situation is always so unique.

I wont go on but the book in question that compares these models is

Urban land-use and transport interaction : policies and models / report of
the International Study Group on Land-use / Transport Interaction
(ISGLUTI) / edited by F.V. Webster, P.H. Bly and N.J. Paulley / authors,
J.F. Brotchie ... [et al.]. Aldershot : Avebury , 1988. xiv,520p :   ill ;
  22cm


Mike Batty

At 22:46 28/04/2009, Alan Penn wrote:
Scott,

two issues come to mind - neither is a direct answer as I also cant
think of the case you ask for.

First, I suspect that getting published in advance of verification may
be part of the problem. This is why the published cases you find are
all post hoc.

Second, the best description I have heard of 'policy' in the sense you
are using was by Peter Allen who described it "at best policy is a
perturbation on the fitness landscape". Making predictions of the
outcome of any policy intervention therefore requires a detailed
understanding of the shape of the mophogenetic landscape. Most often a
perturbation will just nudge the system up a wall of the valley it is
in, only for it to return back into the same valley and no significant
lasting effect will be seen. On occasion a perturbation will nudge the
trajectory over a pass into a neighbouring valley and some kind of
change will result, but unless you have a proper understanding of the
shape of this landscape you wont necessarily be able to say in advance
what the new trajectory will be.

What this way of thinking about things implies is that what we need to
understand is the shape of the fitness landscape. With that
understanding we would be able to say how much of a nudge is needed
(say the size of a tax incentive) to get over a pass. We would also
know what direction the neighbouring 'valleys' might take the system,
and this would allow predictions of the kind you want.

Now what this means for me is that understanding the shape of the
landscape is mainly an analytic task - in my field analysis of spatial
morphology - and the representation and analysis of the system of
interest allows predictions to be made of the effects of design
interventions. I am not clear on how agen simulation fits into this
way of thinking.

Alan


On 28 Apr 2009, at 21:46, Scott Moss wrote:

I have now had half a dozen responses to my question about policy
forecasts but none seem to me to provide examples of what I had
_intended_ as the issue.  I must have been unclear.  Forgive me if I
now
expatiate on the question.

The motivation for the question:  In relation to policy, it is common
for social scientists (including but not exclusively economists) to
use
some a priori reasoning (frequently driven by a theory) to propose
specific policies or to evaluate the benefits of alternative policies.
In either case, the presumption must be that the benefits or relative
benefits of the specified policies can be forecast.  I am not aware of
any successful tests of this presumption and none of my colleagues at
the meeting of UK agent-based modelling experts could point me to a
successful test in the sense of a well documented correct forecast of
any policy benefit.

The importance of the question:  If there is no history or, more
weakly,
no systematic history of successful forecasts of policy impacts,
then is
the standard approach to theory-driven policy advice defensible?  If
so,
on what grounds?  If not, then is an alternative approach to policy
analysis and an alternative role for policy modelling indicated?

What constitutes a successful forecast of policy impact?  I suggest
the
minimal criteria to be a correct forecast of the direction of change
in
the magnitude of specified social indicators together with a
forecast of
the time lags between policy action and social response.  These seem
to me to be such weak criteria that nobody could claim that social
policy modelling has been useful and relevant if they could not be
satisfied.

There are many cases of correct captures of policy indicators
calculated
from past data.  What we cannot identify are cases where a policy
forecast has been published and then the policy implemented and
found to
have the forecast impact on social (including economic) indicators
in a
forecast time frame.

I hope you find the question interesting.

Scott
________________________________________
Michael Batty
Centre for Advanced Spatial Analysis (CASA)
University College London
1-19 Torrington Place
LONDON WC1E 6BT, UK

Tel  44 (0) 207 679 1782/1781 Mobile 44 (0) 7768 423 656
Email [log in to unmask] Web http://www.casa.ucl.ac.uk/
________________________________________