On 03-Aug-11 01:25:54, jo kirkpatrick wrote:
> Please forgive what might be a really dumb suggestion but
> could we magnify the significance of say a T-Test by feeding
> the same 12 results through 4 or 5 times? Please don't all
> scream at once, I am only an MSc student!
>
> Best wishes Jo
> [The rest of the inclusions snipped]
Jo,
If by this you mean stringing a set of 12 results together with
itself (say) 5 times, and then feeding the resulting 60 data
values into a t-test, then the answer is that you will indeed
magnify the significance!
The basic reason is that the sample mean of the 60 will be the
same as the sample mean of the 12, while the sample Standard
Error of the mean will be 1/sqrt(5) times that of the 12.
Hence the t-value for the 60 will be sqrt(5) = 2.236 times
the t-value for the 12. So if, say, your t-value for the 12
was 1.36343 (on 11 degrees of freedom) so that the 2-sided
P-value was then 0.20 (rather disappointing ... ), then if
you did the above you would get a t-value of 3.048722, and
the t-test procedure (being unaware of your deviousness)
would treat this as having 59 degrees of freedom, with the
resulting P-value then being 0.0034 which is much more
satisfying!
Your question is not as "dumb" as it might at first seem.
While it is clearly invalid to create a large dataset by
chaining together replicates of a small one, until you get
one large enough to give you an extreme P-value, this is
not grossly different from going back to the population
again and again, repeatedly sampling 12 each time until
you again get the desired result.
This is because, if the initial 12 were a fair sample,
future samples of 12 are unlikely to be grossly dissimilar
to the initial 12. So sooner or later (and with reference
to the above example probably with around 5 repetitions)
you could move from P=0.2 to P < 0.01 by repeated sampling.
The aggregate sample at any stage is then a valid sample
of that size from the population, as opposed to the invalid
"sample" generated by recycling the original small one.
What is invalid about the procedure is the intention to
keep going until you get a small enough P-value. This
will inevitably occur if you keep going long enough.
No Null Hypothesis is ever exactly true in real life.
If it is off by some small amount, then a large enough
sample (and you may need a very large one) will almost
surely result in a P-value smaller than your target.
The real question is: How far off is it? Is this difference
of any interest? This leads on to the question: If the
smallest difference which is of practical interest is,
say, D, then how large a sample would we need in order
to have a good chance of a significsant P-value if the
true difference were at least D?
Also, the "How far off is it?" question can be addressed
by looking at a confidence interval for the difference.
Such broader approaches should always be used, rather
than simplistic reliance on mere P-values.
Hoping this helps!
Ted.
--------------------------------------------------------------------
E-Mail: (Ted Harding) <[log in to unmask]>
Fax-to-email: +44 (0)870 094 0861
Date: 03-Aug-11 Time: 08:26:20
------------------------------ XFMail ------------------------------
|