This is basically the problem of NHTS (Null Hypothosis Testing).
For a large enough date-set, almost any hypothosis will be found to be
"significant".
Lets take an example of two diet plans, and say 1 million people participate
in each diet plan.
After 6 months group A lost an average of 2 kilos, group B lost an average
of 2.200 kilos.
The difference between the groups is statistically significant.
Does this tell us that diet B is superiour to diet A? Obviously not - since
a difference of 200 grams has no clinical meaning.
The problem is not with the statistical test, but with the hypothosis.
When we state that diet A is better than B, we don't mean that the
difference between them is larger than zero, rather it is larger than 10
kilos, any difference of less than 10 kilos has no clinical meaning.
The catch is that standard statistical packages are geared to testing the
null-zero hypothosis, and cause major confusion between "statistical
significance" and "clinical meaning".
From Tzippy
----- Original Message -----
From: "Mark Coleman" <[log in to unmask]>
To: <[log in to unmask]>
Sent: Tuesday, May 03, 2005 8:47 PM
Subject: Too much data?
> Greetings,
>
> I have recently been conducting some research that tests for the
> non-normality of asset price returns. My review of the literature in
> this field has found articles testing for normality of returns using
> both "small" (n < 30) and "large" data sets (n > 30,000). The null
> hypothesis of normality is routinely rejected, for any size n.
>
> I vaguely recall reading a comment in a recent journal (and I apologize
> for the vagueness of my inquiry) where the author notes that with a
> large enough data set, we can reject virtually any null hypothesis (or
> something to that effect). If so, should that leads me to question the
> usefulness of tests with very large data sets? Or am I simply
> mis-interpreting something about the power of such tests?
>
> Any comments or thoughts would be most appreciated.
>
> Best regards,
>
> Mark
>
|