On 22/03/07, Kathryn Jane Gardner <[log in to unmask]> wrote:
> Hi Jeremy,
>
Hi Kathryn
> I am also interested in your position on 1 and 2 tailed tests. Someone
> asked you: " Are you saying that you should always make a two-tailed
> prediction and perform a two-tailed test? Or are you saying that even
> if you have made a one-tailed prediction, you should still perform a
> two-tailed test?"
>
Yes.
> And you replied..."The latter. You can make a prediction that it will
> go in a particular direction, but you still should use a 2 tailed test.
> Note that lots of tests don't exist in one tailed forms - ANOVA,
> chi-square, for example."
>
> I follow your latter point that not all tests have 1 tailed forms, but
> I have never heard of running a 2 tailed test with a 1 tailed prediction
> before and I can't see members in my dept. using such an approach
> either. Would you mind elaborating on your reasons for this?
Because if you would report and interpret a significant result in the
opposite direction to that you expect, you are actually doing a two
tailed test. If you really, really wouldn't have done that, and can
prove to me that you made that decision before you even began
collecting the data, I *might* accept it.
Psychologists are a little notorious for being able to make up a
theoretical explanation for any result. And then you find that you
coded a variable backwards, and the results are the opposite.
> Also, in a
> related vein, I was wondering what your thoughts are on making
> directional predictions when the theory states that the results should
> go a certain way, but there is not any research to suggest that they
> will e.g., we would expect 2 tests of the same construct to converge
> with high correlations (if they are measuring the same underlying
> construct), but there is no research correlating these particular tests.
> Is this grounds for a directional hypothesis? I have heard some
> researchers say that we can make directional predictions based on theory
> alone, whilst others prefer theory and pervasive trends in the
> literature.
>
You can make a directional prediction based on anything. But if you
then use that directional prediction to argue that you can do a one
tailed test, then that's (in my opinion) naughty.
Have you come across the precocity-longevity hypothesis? I've
forgotten who wrote the articles, but it was published in Personality
and Social Psychology Bulletin. it's the idea that if you achieve
greatness earlier, you die earlier. The author used a one tailed
test, which effectively halves your p-value. Then said that because
the sample was small, he would use 0.10. This means that his type I
error rate, if he had used 2-tailed, was actually 0.20. Now, if you
want to use 0.20, that's fine with me, but SAY THAT YOU ARE USING
0.20, don't try to hide behind reasons.
For a test of two constructs that measure the same thing, we're not
really interested in whether the correlation is significant, we're
interested in whether it's large, and so we should give a correlation
and the confidence intervals of that correlation, which give us an
idea of the likely size in the population.
I review a lot of articles for journals, I'm a statistical editor of
the BJ Clinical Psych, BJ health Psych, I'm on the editorial board of
the BJ Math and Stat Psych, and the European J of Social Psych, and
I'm listed as a statistical editor for PLOS Clinical Trials, and the
whole BMC family of journals, I also do a lot of reviews (sometimes
one a week) for Quality and Safety in Health Care. I'm not saying
this to boast - no one ever asks me to review a paper for the
substance :( just to say that I read a lot of articles from all kinds
of areas (but I only have to concentrate on the results section) and I
can recall seeing only one one-tailed test in all of that lot. And it
was inappropriate.
Jeremy
--
Jeremy Miles
Learning statistics blog: www.jeremymiles.co.uk/learningstats
|