This is an important question, but hard to answer in a live policy
environment for a several reasons. First, the policy change modeled is
often slightly different from what is finally enacted and/or
implemented. Second, single/simple changes are very rarely enacted, but
rather a set of changes are enacted and/or implemented, making it hard
to isolate the effect/impact of the target change. Third, other
variables that the model assumed, e.g. the economy, may widely diverge
from what was assumed, making it hard to disentangle the effect of the
policy change from the effect of these other variables. Finally, many of
us who work in a government policy environment cannot find the resources
for such research/validation. (I'm probably forgetting other reasons,
but you get the idea of the difficulty.)

That said, my office in the U.S. Department of Health & Human Services
some years ago did try to track the effect of one policy change (one
among a large set of changes) that seemed particularly amenable to such
an effort. Given that we chose the policy change, it isn't surprising
that the predicted effect was within about 10% of the dollar prediction
and slightly worse at the "people" impact. These analyses were all
internal in-house efforts and were never published -- governments are
notoriously leery of exposing the errors of prediction in their models.

The model we used is fairly simple in concept, but has lots of detail.
It models the income and benefits of individual people in the U.S. It is
a large, (static) micro-simulation model based on a nationally
representative survey of the country. We correct for some under- and
non-reporting of income and benefits, we impute some additional
variables, and we align to administrative totals for some of these
variables. The model then uses a large database of program rules and
parameters to simulate a program or other change. We essentially run the
model twice: once with current law and again with the proposed law
change. We then compare the results.

This type of model is well suited for comparing the short-term impact of
alternative policy changes, but not well suited for comparing long-term
impacts or making predictions beyond a few years, nor is it well suited
for dramatic changes in policy that are expected to change the
underlying human behavior in major ways.

Reuben Snipper
Personal views

 
-----Original Message-----
From: News and discussion about computer simulation in the social
sciences [mailto:[log in to unmask]] On Behalf Of Scott Moss
Sent: Tuesday, April 28, 2009 9:47 AM
To: [log in to unmask]
Subject: [SIMSOC] any correct policy impact forecasts?

In a recent discussion among UK agent-based social modellers, I asked
whether there has ever been a correct, publicly documented, model-based
forecast of a policy impact. There are, of course, many such forecasts
tested successfully on past data but none of us were able to produce a
single example of a correct forecast of policy impact reported or
documented prior to the implementation of the policy.

Does anyone know of a correct, real-time, model-based, policy-impact
forecast? Please include references to publications of the forecast
prior to policy implementation or to some other evidence that the
forecast preceded the impact.

Many thanks in advance,
Scott