Hi all,
Interesting how we keep coming back to these difficult issues :-)

I sympathize with both views here, but isn't one problem here acknowledging
that there is no such thing as theory-free observation?  (This is an
observation arising out of social science that is certainly useful).


The problem with the view that simulations are tests of theories is that
there is no way other than argument to 'prove' that a simulation represents
a theory -- Oreskes et al. nailed this problem some time ago.  Once we
accept that problem, then simulation becomes yet another mode of discursive
argument, one surrounded with complex and 'cool' techie stuff, but
nevertheless, just another mode of argumentation.  If my simulation
'represents' my theory, and yours represents yours, and they both are
capable of replicating empirical phenomena, then there is no way to choose
between them, other than old-fashioned argumentation.

I'm open to argument on how much a simulation is bolstered by establishing
the robustness of its behavior to choice of parameter values etc., although
I'm not wholly convinced on that point: surely we know enough about the
world to realize that it is robust to variation in some parameters and
highly sensitive to variation in some others.  The robustness argument
ultimately rests on statistical inference: if something is unlikely to have
occurred by chance (because it needs a very specific set of parameter
choices) then it can't be right.  Well that would be fine if the real world
was a sample from an infinite population.  Without getting to metaphysical
about it, it isn't.

[I guess we could argue about this one forever, and in some other universe,
we probably are ;-)]


On the other hand, building simulations populated by agents whose behaviors
are derived from observation, (i) assumes that we can make theory-free
observations, and (ii) still runs into the difficulty of establishing that
your simulation is somehow a valid representation of your observations.


I'm not intending to dismiss either of these lines of argument, I just
think that we may have to own up to a couple of hard truths:

1. Simulations are simulations, and the world is the world.  Assessing the
representational 'truth' of a simulation is IMPOSSIBLE.  The best we can
hope for is practical adequacy, that is, the simulation is useful until we
discover that it is no longer useful (and we won't know that until it's too
late...).

2. Observation of the world is always theory-laden.  So, when we make a
claim that the world is self-evidently composed only of individuals (say),
so it's 'obvious' that we should model on the individual level, we are
taking a theoretical position, not simply observing the world.


None of this is necessarily as bad as it sounds: a good simulation can be
practically adequate for a long time, hence useful and informative, a good
part of the time (unfortunately, it is likely to let us down, just when we
need it most, i.e., when everything is changing...).

Also, simulations or models are great media for thought experimentation and
education.



David



At 11:38 AM 11/14/2003 -0300, you wrote:
>what an incredible remark, if I understand it correctly
>
>in my opinion, one should make observations to accept or reject theories
>(and for that measurements have to be very accurate)
>
>the world is full of predictions based on observations with no theoretical
>back up that led to horrendous failures an  huge amounts of money spent in
>terrible projects
>
>of course is theory does not validate it is of not much use ... but without
>theory observations are pretty useless
>
>J de D Ortuzar
>
>At 02:13 PM 11/14/2003 +0000, Scott Moss wrote:
>>I guess the views of Penn, Tesfatsion and Markovsky (excerpted below)
>>expressed in the simulation and explanation (now theory) thread are
>>fairly typical and widespread.  They are consistent with conventional
>>approaches to social science.  Theory predominates over observation.  I
>>have been arguing (e.g. in my Presidential Address to ESSA) that if
>>social simulation with agents is to be anything other than another in
>>the long line of failed approaches to social science, it will be a
>>positive departure only because it facilitates the dominance of
>>observation over theory.
>>
>>The great successful scientists -- Copernicus, Kepler, Galileo, Newton,
>>Darwin, Planck, Faraday, Einstein, Watson & Crick (maybe not quite in
>>the class of the others) -- built their conceptual structures and
>>generalisations around observation.  Theory always gave way to
>>evidence.  Newton and Darwin in particular kept their theories to
>>themselves for decades before being convinced that they were supported
>>by a sufficuently wide range of evidence.  Only when these theoretical
>>structures were well validated did they come into general use for
>>guiding new observation, identifying new problems and, to solve those
>>problems, developing new theoretical structures based on and validated
>>by new evidence.
>>
>>Consider this tradition in relation to the following excerpts:
>>
>>
>>Alan Penn:
>>
>>>can anyone think of excellent examples of
>>>simulation providing explanation in any field (let alone sociology).
>>>It seems to me that simulations (at their best) are built on top of
>>>pre-existing explanatory theory, they may act as tests of those
>>>theories, but in a true Popperian sense, can do little to confirm only
>>>perhaps help falsify them.
>>
>>
>>
>>If you start from a social theory that does not itself specify agent
>>behaviour and interaction so that those specifications can be validated
>>by, for example, the individuals the agents represent, then the only
>>link with evidence is the predictions generated by the theory.  If there
>>are no correct predictions, then there is no link at all between theory
>>and the world we observe.  There has never in the history of Economics
>>and Management Science been a correct forecast of macroeconomic or
>>financial market turning points or turning points in retail market sales
>>(by brand or SKU).  I know less about sociology, but my reading of the
>>journals in that field suggests that no sociological theory offers
>>systematically well validated predictions, either.  Presumably, top-down
>>sociological theory does not offer well validated propositions about
>>individuals and their interactions.  Does anyone have any
>>counterexamples to these statements?
>>
>>Leigh Tesfatsion (paper cited in her email):
>>
>>   Thus, as implemented for this study, the labor market framework
>>comprises an
>>    equal number of work suppliers and employers. These work suppliers
>>and employers
>>    repeatedly participate in costly searches for worksite partners on
>>the basis of
>>   continually updated expected utility, engage in efficiency-wage
>>worksite interactions
>>   modelled as prisoner's dilemma games, and evolve their worksite
>>strategies over time
>>   on the basis of the earnings secured by these strategies in past
>>worksite interactions.
>>
>>This excerpt from Leigh's paper is perhaps a case in point.  I would be
>>astonished if anyone could provide evidence that employers and employees
>>(== work suppliers, presumably) would claim to "participate in costly
>>searches for worksite partners on the basis of continually updated
>>expected utility."  Is there any independent evidence that prisoners'
>>dilemma games are good representations of "worksite interactions"?  This
>>is a pretty good example of starting from theory (which has been shown
>>to be invalid by experimental economists repeatedly over the past 50
>>years) in order to draw conclusions about the world we observe.  Why is
>>it better to restate the problem in terms of such a theory than to get
>>evidence about actual worksite behaviour and design agents to describe
>>that behaviour?
>>
>>
>>Barry Markovsky
>>
>>>>I want to expand on Alan Penn's comments (below) because I think there
>>>>are some common misunderstandings and misuses of simulations in
>>>>sociology that we need to battle.
>>>>
>>>>Say you start with a typical discursive sociological theory, and you
>>>>want to write a simulation that is in some sense designed to reflect
>>>>the claims and explanatory mechanisms of that theory.
>>
>>>By this view, every simulation provides an explanation: Their
>>>statements and functional relationships constitute the explanations
>>>for the
>>>patterns of output they generate. Certainly Sugarscape accomplishes
>>>this (as Mike points out below), and so do countless others. A reader may
>>>feel that such an explanation flawed or insufficient, but then the
>>>burden is on the reader either to show how the simulation fails to
>>>account for the phenomenon it intended to explain, or to provide an
>>>alternative theory or simulation that provides a superior account.
>>
>>
>>I don't understand why the burden is on the reader "either to show how
>>the simulation fails to account for the phenomenon it intended to
>>explain, or to provide an alternative theory or simulation."  An
>>alternative scientific approach that has worked well in the physical and
>>biological sciences is to place the burden on the theorist/modeller to
>>demonstrate that the theory or model is descriptively accurate or, where
>>that demonstration cannot be made, that the theory/model is robust with
>>respect to the specifications that cannot be validated.
>>
>>As Rosaria said, simulation and theory should not be confused.  The
>>virtue of agent based simulation is that the agents can be descriptors
>>of observed behaviour.  Where different observers of behaviour have
>>different descriptions of that behaviour, then those different
>>descriptions can be modelled using agents.  Personally, I find it easier
>>to do that in declarative languages, but there is no insuperable
>>difficult about doing it in Java or C/C++ (hence, RePast or Swarm).
>>
>>This is what makes agent based social simulation different from the long
>>line of empirically failed social theories and modelling techniques such
>>as  utility theory and game theory.  It enables us to engage with
>>observation and evidence without the constraints of unvalidated theory.
>>If good social science and social theory can be produced, then the
>>experience of the natural sciences is that it will be produced on the
>>basis of good observation and evidence.  Agent based modelling enables
>>us to formalise such observations without spurious generalisation.  As
>>such, it might be a means of developing good social theory.  But I
>>wouldn't expect to see that theory any time soon.
>>
>>
>>Scott Moss
>>Professor of Social Simulation
>>Centre for Policy Modelling
>>Manchester
>
>
>
>
>Prof. Juan de Dios Ortuzar        e-mail: [log in to unmask]
>Departamento de Ingenieria de Transporte
>Pontificia Universidad Catolica de Chile
>Casilla 306, Cod. 105, Santiago 22, Chile
>Tel: 56-2-686 4822  Fax: 56-2-553 0281