---------- Forwarded message ----------
From: Lucas <[log in to unmask]>
To: [log in to unmask]
Date: Thu, 11 Jun 2009 11:18:36 +0100
Subject: Re: [SIMSOC] any correct policy impact forecasts?

Hi,

I've been discussing with many social simulation researchers the issues
surrounding usefulness of models since May 2008 and, so far, answers
have been similar in terms of what can be done with results beyond theory.

Prediction is often cited by stakeholders or policy-makers as what would be
most useful to them. Such possibility seems tied to the type and time-scale
of the scenario in question and examples that can't overcome these abound.

Finding a *social* simulation model, driven by evidence of some dynamic social
phenomenon, which results are consistently regarded as useful (in any sense)
as fieldwork findings by non-academics is another story. I've presented and
discussed this matter first in MX (May.08), UK (Nov.08), then in NL and DE
(Jan.09), and UK again (April.09).

This isn't necessarily linked to validation, as the point is to
discuss where and
when social simulation models are pragmatically used, outside the academic
world. Snipper mentioned important  aspects of how policies are discussed
and implemented, as those seems recurrent in different environments, such
as the micro-finance case-study I work with.

Regards, P.
--
kaiowas.no-ip.org


2009/4/29 Snipper, Reuben (HHS/ASPE) <[log in to unmask]>:
> This is an important question, but hard to answer in a live policy
> environment for a several reasons. First, the policy change modeled is
> often slightly different from what is finally enacted and/or
> implemented. Second, single/simple changes are very rarely enacted, but
> rather a set of changes are enacted and/or implemented, making it hard
> to isolate the effect/impact of the target change. Third, other
> variables that the model assumed, e.g. the economy, may widely diverge
> from what was assumed, making it hard to disentangle the effect of the
> policy change from the effect of these other variables. Finally, many of
> us who work in a government policy environment cannot find the resources
> for such research/validation. (I'm probably forgetting other reasons,
> but you get the idea of the difficulty.)
>
> That said, my office in the U.S. Department of Health & Human Services
> some years ago did try to track the effect of one policy change (one
> among a large set of changes) that seemed particularly amenable to such
> an effort. Given that we chose the policy change, it isn't surprising
> that the predicted effect was within about 10% of the dollar prediction
> and slightly worse at the "people" impact. These analyses were all
> internal in-house efforts and were never published -- governments are
> notoriously leery of exposing the errors of prediction in their models.
>
> The model we used is fairly simple in concept, but has lots of detail.
> It models the income and benefits of individual people in the U.S. It is
> a large, (static) micro-simulation model based on a nationally
> representative survey of the country. We correct for some under- and
> non-reporting of income and benefits, we impute some additional
> variables, and we align to administrative totals for some of these
> variables.  The model then uses a large database of program rules and
> parameters to simulate a program or other change. We essentially run the
> model twice: once with current law and again with the proposed law
> change. We then compare the results.
>
> This type of model is well suited for comparing the short-term impact of
> alternative policy changes, but not well suited for comparing long-term
> impacts or making predictions beyond a few years, nor is it well suited
> for dramatic changes in policy that are expected to change the
> underlying human behavior in major ways.
>
> Reuben Snipper
> Personal views
>
>
> -----Original Message-----
> From: News and discussion about computer simulation in the social
> sciences [mailto:[log in to unmask]] On Behalf Of Scott Moss
> Sent: Tuesday, April 28, 2009 9:47 AM
> To: [log in to unmask]
> Subject: [SIMSOC] any correct policy impact forecasts?
>
> In a recent discussion among UK agent-based social modellers, I asked
> whether there has ever been a correct, publicly documented, model-based
> forecast of a policy impact.  There are, of course, many such forecasts
> tested successfully on past data but none of us were able to produce a
> single example of a correct forecast of policy impact reported or
> documented prior to the implementation of the policy.
>
> Does anyone know of a correct, real-time, model-based, policy-impact
> forecast?  Please include references to publications of the forecast
> prior to policy implementation or to some other evidence that the
> forecast preceded the impact.
>
> Many thanks in advance,
> Scott
>