 |
 |
you are right. the reviewer is wrong. As you say yourself, the chance that these six p-values are all drawn from a [0;1] uniform distribution is very small. Actually, the "correction" should go the other way: even if all of the were between (say) 0.05 and 0.15, the over-all conclusion might well be to reject the null hypothesis. What you might do is to add the squares of the six z-statistics. Assuming each z-statistic is N(0,1) and they are independent, the result would be chisquare (df=6) distributed. By the way, it doesn't matter if the degrees of freedom are the same for all the six tests. p-values are p-values. But I would prefer to combine the underlying test statistics rather than combining the p-values. If the p-values come from normal distributted test statistics (this is assymtotically true for many standard tests), then the chi-square test I suggested is valid, and it gives a p-value of 2.6E-11 in this case. Which should be signifcant enough even for a physics journal! On Wed, Aug 29, 2012 at 10:25 AM, Andy Cooper <[log in to unmask]>wrote: > Dear All, > > I have the following question which I am hoping native statisticians (I am > a > physicist by training) can help me address. To set the background as to > why I am asking this question: a manuscript recently submitted to a Physics > journal got rejected because 1 of the 3 reviewers claims that a result > presented in the given manuscript is wrong. I would therefore be very > grateful to hear the opinion of statisticians. > > The issue in question is as follows: Suppose we have 6 statistics (e.g > z-statistics), each derived from an independent data set (i.e 6 independent > data sets in total). We can assume that the number of degrees of freedom is > the same in each data set, so that the corresponding P-values are also > comparable. We can further assume that each independent data set is a > sample from an underlying population. Under the null hypothesis (z=0), the > P-values would be distributed uniformly between 0 and 1. Now, the observed > P-values are in fact (3e-9, 0.04, 0.05, 0.03, 0.02, 0.005), i.e they are > all less than 0.06. It is clear, at least to me, that the chance that these > P-values are drawn from a uniform distribution is pretty small (<1e-8). Yet > the reviewer in question claims that there is no overall significance. > His/her argument is based on the Bonferroni correction: using a threshold > of 0.05/6~0.008 only 2 P-values pass this threshold, which he/she then goes > on to claim is not meaningful enough. > > My response to the reviewer's comment is that the use of a Bonferroni > correction to establish the overall significance of the 6 P-values is > wrong. The Bonferroni correction is ill-suited for this particular > application since it is overly conservative, leading to a large fraction of > false negatives. Remarkably, the editor of the Physics journal in question > finds the reviewers arguments (i.e using the Bonferroni correction) as > "persuasive". > > I would be most grateful for your comments. > > A > You may leave the list at any time by sending the command > > SIGNOFF allstat > > to [log in to unmask], leaving the subject line blank. > You may leave the list at any time by sending the command SIGNOFF allstat to [log in to unmask], leaving the subject line blank.