Dear Tim,
 
the questions you raise are very worthy ones indeed.
 
In my own case I mapped my own theory against an alterative account that could be seen as a limiting case, by which I mean, a case that could be seen as a polar opposite (though in truth in the area of my theory (literary affects) there are no polar opposites but there are significantly different positions that some might see as being radically different).
 
Being able to use my theory map in predictive ways in this alterative terrain showed at least how to apply my theory (procedures) as well as confirming the potential for my theory to elegantly include within its compass, existing accounts that might, on first glance, have seemed to be disconfirming.
 
One of my external examiners attempted to "prove" my theory wrong by asserting my theory didn't suit his theory  of theories which was that all theory is wrong because no theory can account for everything.
 
Could one prove my theory to be wrong? Certainly! The same data might be accounted for differently.
 
At a minimum, my theory could be shown to be faulty which means it is open to alteration, development, revision etc.
 
I am happy to treat my theory in pragmatic ways. Does it offer more useful accounts of existing data than other available accounts? If yes, then it would seem useful.
 
I am still finding new uses for my theory and I am still open to more serious neurological testing to provide more concrete evidence of my key assertions. Perhaps one day I will meet up with someone who can set up the tests.
 
hope this helps
 
keith
 

>>> Tim Smithers <[log in to unmask]> 18/12/2012 9:17 pm >>>
Dear Don and Keith,

Thank you both for your supportive remarks!

When I received the comment that a theory author couldn't
properly test his or her theory, I too thought it stupid, but
the situation demanded a degree of politeness that disallowed
saying so: it came from "an established and well respected
researcher," as they say ...  However, I did say I didn't
think anybody else would do this, and that I saw testing as a
normal part of theory development.  This brought the further
comment that I clearly didn't know much about theory
development :-) ...  Which I may not.

So perhaps I could raise the following issue: a more serious
one, I think.

As I say in the paper, I didn't try to do a conclusive test of
the theory.  It was more an attempt to see if this theory
development work was "on the right track."  In this sense, the
testing worked, I think: it found supporting evidence and
identified some needed refinements.  But, as you point out,
Don, this did not confirm this theory, not in any big way, at
least.  For this more testing would be needed.

My question is what should we be trying to do here: devise and
execute studies and/or experiments that look for confirming
evidence of a theory, or should we devise and execute studies
and/or experiments that seek to disconfirm (disprove) the
theory, a la Popper?  The two, as I see it, are not the same
kind of work.  The first--studies and experiments that look
for confirming evidence--can result in disconfirming evidence,
but if we only seek disconfirming evidence, how do we ever
have much reason to have any confidence in the theory?  A test
designed to look for disconfirming evidence that fails does
not provide confirming evidence, not in tests of design
theories, at least.

For me, this is not so much a philosophical issue, nor even a
scientific method issue, it is more a practical one.  Seeing
what evidence for a theory of designing looks like is easier
to do than seeing what evidence against such a theory might
look like.  Devising studies and tests that look for
supporting evidence is easier to do that devising tests that
look for disconfirming evidence.  So, as I report in the
paper, I did some designing and recorded this in a way that
was intended to generate useful data that could be used to
compare with what the theory said.  But, the theory informed
the design of this observation and data recording.  So, the
theory was implicated in the test design, which I know some
people think is at least poor if not wrong experiment design.
But, if you don't do this, how can you be sure you have
designed an experiment that really does test your theory?  You
can't use just any kind of observational data to test a theory
of designing.

This is, I think, to do with the nature of theories of
designing.  Other kinds of theories don't necessarily suffer
this difficulty.  Examples of these latter kinds can be found
in the physical sciences.  Theoretical work in the social
sciences, and, I would say, in design research, do not result
in theories that can be treated in the same way.  A different
practice of theory development and testing is needed, and, to
put it bluntly, Popper doesn't show us a useful way to go on
this.  So, who or what does?

Keith, how was the theory testing you did for your thesis?
Did you test for supporting evidence, or disconfirming
evidence?  Or did you do something different?  And, Don, would
you say successful use of a design theory can count as
supportive evidence, albeit, only a little bit each time, and,
that unsuccessful use counts as some disconfirming evidence?

Best regards,

Tim

===============================================================

On Dec 17, 2012, at 23:12 , Keith Russell wrote:

> Dear Don,
> 
> yep, I agree - if we want to generate theories then we should be part of the testing process.
> 
> The Appendix to my PhD (which offers a theory) consists of a testing/application of the theory. 
> 
> cheers
> 
> keith
> 
>>>> Don Norman <[log in to unmask]> 12/18/12 5:15 AM >>> 
> One of the stupidest comments I have seen recently (that an author of a
> theory cannot test it. Just the opposite is true: the author has an
> obligation to test it.). In the hard core sciences, authors of theories
> often test their own theories. Hell, who else would do it? If people doubt
> the validity of the results, they are encouraged to try to replicate the
> results. That's how science progresses. Note, a single test never confirms
> a theory -- it takes numerous ones. (Actually, theories can only be
> disproved -- no amount of tests can validate a theory).
> 
> (Hey folks: attachements work!)
> 
> Don
> 
> On Mon, Dec 17, 2012 at 9:33 AM, Tim Smithers <[log in to unmask]>wrote:
> 
>> and so not a theory.  After publishing this work, the same
>> person said that the author of a theory could not properly
>> test his or her own theory, so my test was invalid.
>> 
> 
> Don Norman
> Nielsen Norman Group, IDEO Fellow
> [log in to unmask]   www.jnd.org http://www.core77.com/blog/columns/ 
> Latest book: "Living with Complexity <http://www.jnd.org/books.html#608>"
> Revision of "Design of Everyday Things" completed. Pub date Q3 2013
> 


-----------------------------------------------------------------
PhD-Design mailing list  <[log in to unmask]>
Discussion of PhD studies and related research in Design
Subscribe or Unsubscribe at https://www.jiscmail.ac.uk/phd-design 
-----------------------------------------------------------------


-----------------------------------------------------------------
PhD-Design mailing list  <[log in to unmask]>
Discussion of PhD studies and related research in Design
Subscribe or Unsubscribe at https://www.jiscmail.ac.uk/phd-design
-----------------------------------------------------------------