Well, a Bayesian would say that they shouldnīt have computed all these p-values in the first place, rather they should have built a joint model (probably a multilevel one) for the whole battery of issues combined and then one can look at the posterior probabilties of whatever is of interest.
Coming up with a joint model isnīt easy though (some Bayesians may disagree) and itīd probably require a number of decisions that one wouldnīt feel to be in the position to make.
Still it is difficult for somebody like me who isnīt generally negative about p-values to avoid feeling that producing such a jungle of tests is not how one should use them.
Or rather, Iīd probably interpret them in some kind of exploratory manner, highlighting issues of potential interest for further study (and perhaps even then going for the 0.01 rather than the 0.05 level) without jumping to strong conclusions about anthing from just the published results (OK, if thereīs a p-value of 10^{-8} somewhere,chances are there really is something going on).
The way I think about Bonferroni, Benjamini-Hochberg and formal rules like this is that they all give you some information that could be of interest, but they donīt tell you what you "should" do. If you test things according to Bonferroni, the error probability of finding anything in case that nothing is really going on is <0.05 (or whatever level you chose), which is good to know, but you wonīt have a high power. Obviously, if you apply Bonferroni only over small sets of 2 or 3 tests, the guarantee only holds for those small sets. Benjamini-Hochberg controls the false dscovery rate - at the end if the day itīs your decision what you want to control, but even then this is a bit of an illusion because if you make the decision what you use dependent on the results that you have already seen in any way, all of these are invalidated. So my personal advice is "exploratory interpretation" rather than taking any "inference" too seriously.
Best regards,
Christian
*** --- ***
Christian Hennig
University College London, Department of Statistical Science
Gower St., London WC1E 6BT, phone +44 207 679 1698
[log in to unmask], www.homepages.ucl.ac.uk/~ucakche
________________________________________
From: A UK-based worldwide e-mail broadcast system mailing list <[log in to unmask]> on behalf of Kornbrot, Diana <[log in to unmask]>
Sent: 15 January 2017 15:51:40
To: [log in to unmask]
Subject: Correction for Multiple Testing
A published Ms. reports the results of 4 studies
Statcheck (http://statcheck.io/index.php) identifies 69 hypotheses tests, of which 44 are significant at the 0.05 level. there are 5 errors, but none would change decision at .05 confidence level.
The tests are generated from several data sets, many of which are factorial designs with factors having between 2 and 6 levels
This is not atypical of research in psychology.
MY question is how should one correct for multiple testing.
Is it sufficient to apply Bonferoni separately for each data set, e.g. testing 2 main effects and 1 interaction?
Or should one apply Benjamini-Hochberg to determine a Ms. level criterion for significance?
Believe me 69 hypotheses is NOT uncommon. Researchers are often expected to conduct several studies and may have good reason to have more than one response variable in each study. For example speed and accuracy, or change in several different health indices (depression, anxiety, self-efficacy etc)
Really appreciate views on this. Apologies for multiple testing
best
Diana
________________________________________
Professor Diana Kornbrot
Work
University of Hertfordshire
College Lane, Hatfield, Hertfordshire AL10 9AB, UK
+44 (0) 170 728 4626
[log in to unmask]<mailto:[log in to unmask]>
http://dianakornbrot.wordpress.com/
http://go.herts.ac.uk/Diana_Kornbrot
skype: kornbrotme
Home
19 Elmhurst Avenue
London N2 0LT, UK
+44 (0) 208 444 2081
You may leave the list at any time by sending the command
SIGNOFF allstat
to [log in to unmask]<mailto:[log in to unmask]>, leaving the subject line blank.
You may leave the list at any time by sending the command
SIGNOFF allstat
to [log in to unmask], leaving the subject line blank.
|
