 |
 |
Kim, in principle, in the hypothesis testing paradigm, the argument that p-values need adjustment for multiplicity of comparisons applies exactly the same for parametric and non-parametric analyses - though the user needs to be aware that, in contrast to the parametric case, for a given sample size the p-value can take only a finite set of possible values, which can be quite small for small sample sizes. BUT - we always need to consider very carefully whether the p-value is really the most helpfully informative measure of strength of effect or strength of evidence. Arguably, it is a flawed measure of strength of evidence (as it only relates to the weight of evidence against H0, not the weight of evidence favouring H1), and a quite useless measure of strength of effect - for instance, it cannot be interpreted meaningfully except in relation to the sample size. There is a growing emphasis on use of interpretable effect size measures, with CIs, rather than p-values as the main basis for inferences. Both absolute and relative effect size measures are available, for comparisons of means, comparisons of proportions, and relative measures such as U/mn = AUROC as a better expression of the outcome of a Mann-Whitney comparison. Effect size measures relate primarily to measurement, not to coincidence, so there is not such a compelling reason to adjust for multiple comparisons - only if you are really using the CI to see whether it includes the H0 value, i.e. significance testing by the back door. No, these effect size measures are interpretable in their own right. Some of them are of course already very well known and widely used - e.g. for two independent proportions, their difference, ratio and odds ratio. The ones relating to non-parametrics are less well known, but also potentially extremely useful. For a general introduction to effect sizes, see RJ Grissom & JJ Kim, Effect Sizes for Research, 2nd edition, Routledge, and my book - link below. At my website there is an Excel spreadsheet to calculate a CI for the U/mn measure, which also points to my 2006 articles that describe this measure and how to calculate a CI for it. Hope this helps. Robert G. Newcombe PhD CStat FFPH HonMRCR Professor of Biostatistics Cochrane Institute of Primary Care and Public Health School of Medicine Cardiff University 4th floor, Neuadd Meirionnydd Heath Park, Cardiff CF14 4YS Tel: (+44) 29 2068 7260 My book Confidence Intervals for Proportions and Related Measures of Effect Size is now published. Available at http://www.crcpress.com/product/isbn/9781439812785 See http://www.facebook.com/confidenceintervals Home page http://medicine.cf.ac.uk/person/prof-robert-gordon-newcombe/ From: Kim Pearce <[log in to unmask]> To: [log in to unmask], Date: 26/11/2013 12:20 Subject: Multiple comparisons correction - non parametric tests Sent by: A UK-based worldwide e-mail broadcast system mailing list <[log in to unmask]> Hello everyone, This is just a very quick question. When applying 'multiple comparisons' adjustments to p values from a table of statistical tests, all of the papers/texts I have ever encountered have dealt with p values derived from parametric statistics (Pearson's correlation coefficient, ANOVA etc). Am I correct in thinking that such multiple comparisons corrections e.g. sequential Bonferroni can also be applied to p values from non parametric tests? Many thanks for your advice on this matter, Kind Regards, Dr Kim Pearce PhD, CStat Senior Statistician Haematological Sciences Institute of Cellular Medicine William Leech Building Medical School Newcastle University Framlington Place Newcastle upon Tyne NE2 4HH Tel: (0044) (0)191 282 0451 You may leave the list at any time by sending the command SIGNOFF allstat to [log in to unmask], leaving the subject line blank. You may leave the list at any time by sending the command SIGNOFF allstat to [log in to unmask], leaving the subject line blank.