Dear Don and Keith,
Thank you both for your supportive remarks!
When I received the comment that a theory author couldn't
properly test his or her theory, I too thought it stupid, but
the situation demanded a degree of politeness that disallowed
saying so: it came from "an established and well respected
researcher," as they say ... However, I did say I didn't
think anybody else would do this, and that I saw testing as a
normal part of theory development. This brought the further
comment that I clearly didn't know much about theory
development :-) ... Which I may not.
So perhaps I could raise the following issue: a more serious
one, I think.
As I say in the paper, I didn't try to do a conclusive test of
the theory. It was more an attempt to see if this theory
development work was "on the right track." In this sense, the
testing worked, I think: it found supporting evidence and
identified some needed refinements. But, as you point out,
Don, this did not confirm this theory, not in any big way, at
least. For this more testing would be needed.
My question is what should we be trying to do here: devise and
execute studies and/or experiments that look for confirming
evidence of a theory, or should we devise and execute studies
and/or experiments that seek to disconfirm (disprove) the
theory, a la Popper? The two, as I see it, are not the same
kind of work. The first--studies and experiments that look
for confirming evidence--can result in disconfirming evidence,
but if we only seek disconfirming evidence, how do we ever
have much reason to have any confidence in the theory? A test
designed to look for disconfirming evidence that fails does
not provide confirming evidence, not in tests of design
theories, at least.
For me, this is not so much a philosophical issue, nor even a
scientific method issue, it is more a practical one. Seeing
what evidence for a theory of designing looks like is easier
to do than seeing what evidence against such a theory might
look like. Devising studies and tests that look for
supporting evidence is easier to do that devising tests that
look for disconfirming evidence. So, as I report in the
paper, I did some designing and recorded this in a way that
was intended to generate useful data that could be used to
compare with what the theory said. But, the theory informed
the design of this observation and data recording. So, the
theory was implicated in the test design, which I know some
people think is at least poor if not wrong experiment design.
But, if you don't do this, how can you be sure you have
designed an experiment that really does test your theory? You
can't use just any kind of observational data to test a theory
of designing.
This is, I think, to do with the nature of theories of
designing. Other kinds of theories don't necessarily suffer
this difficulty. Examples of these latter kinds can be found
in the physical sciences. Theoretical work in the social
sciences, and, I would say, in design research, do not result
in theories that can be treated in the same way. A different
practice of theory development and testing is needed, and, to
put it bluntly, Popper doesn't show us a useful way to go on
this. So, who or what does?
Keith, how was the theory testing you did for your thesis?
Did you test for supporting evidence, or disconfirming
evidence? Or did you do something different? And, Don, would
you say successful use of a design theory can count as
supportive evidence, albeit, only a little bit each time, and,
that unsuccessful use counts as some disconfirming evidence?
Best regards,
Tim
===============================================================
On Dec 17, 2012, at 23:12 , Keith Russell wrote:
> Dear Don,
>
> yep, I agree - if we want to generate theories then we should be part of the testing process.
>
> The Appendix to my PhD (which offers a theory) consists of a testing/application of the theory.
>
> cheers
>
> keith
>
>>>> Don Norman <[log in to unmask]> 12/18/12 5:15 AM >>>
> One of the stupidest comments I have seen recently (that an author of a
> theory cannot test it. Just the opposite is true: the author has an
> obligation to test it.). In the hard core sciences, authors of theories
> often test their own theories. Hell, who else would do it? If people doubt
> the validity of the results, they are encouraged to try to replicate the
> results. That's how science progresses. Note, a single test never confirms
> a theory -- it takes numerous ones. (Actually, theories can only be
> disproved -- no amount of tests can validate a theory).
>
> (Hey folks: attachements work!)
>
> Don
>
> On Mon, Dec 17, 2012 at 9:33 AM, Tim Smithers <[log in to unmask]>wrote:
>
>> and so not a theory. After publishing this work, the same
>> person said that the author of a theory could not properly
>> test his or her own theory, so my test was invalid.
>>
>
> Don Norman
> Nielsen Norman Group, IDEO Fellow
> [log in to unmask] www.jnd.org http://www.core77.com/blog/columns/
> Latest book: "Living with Complexity <http://www.jnd.org/books.html#608>"
> Revision of "Design of Everyday Things" completed. Pub date Q3 2013
>
-----------------------------------------------------------------
PhD-Design mailing list <[log in to unmask]>
Discussion of PhD studies and related research in Design
Subscribe or Unsubscribe at https://www.jiscmail.ac.uk/phd-design
-----------------------------------------------------------------
|