Kim, in principle, in the hypothesis testing
paradigm, the argument that p-values need adjustment for multiplicity of
comparisons applies exactly the same for parametric and non-parametric
analyses - though the user needs to be aware that, in contrast to the parametric
case, for a given sample size the p-value can take only a finite set of
possible values, which can be quite small for small sample sizes.
BUT - we always need to consider very carefully
whether the p-value is really the most helpfully informative measure of
strength of effect or strength of evidence. Arguably, it is a flawed measure
of strength of evidence (as it only relates to the weight of evidence against
H0, not the weight of evidence favouring H1), and a quite useless measure
of strength of effect - for instance, it cannot be interpreted meaningfully
except in relation to the sample size. There is a growing emphasis on use
of interpretable effect size measures, with CIs, rather than p-values as
the main basis for inferences. Both absolute and relative effect size measures
are available, for comparisons of means, comparisons of proportions, and
relative measures such as U/mn = AUROC as a better expression of the outcome
of a Mann-Whitney comparison. Effect size measures relate primarily to
measurement, not to coincidence, so there is not such a compelling reason
to adjust for multiple comparisons - only if you are really using the CI
to see whether it includes the H0 value, i.e. significance testing by the
back door. No, these effect size measures are interpretable in their own
right. Some of them are of course already very well known and widely used
- e.g. for two independent proportions, their difference, ratio and odds
ratio. The ones relating to non-parametrics are less well known, but also
potentially extremely useful.
For a general introduction to effect sizes,
see RJ Grissom & JJ Kim, Effect Sizes for Research, 2nd edition, Routledge,
and my book - link below. At my website there is an Excel spreadsheet to
calculate a CI for the U/mn measure, which also points to my 2006 articles
that describe this measure and how to calculate a CI for it.
Hope this helps.
Robert G. Newcombe PhD CStat FFPH HonMRCR
Professor of Biostatistics
Cochrane Institute of Primary Care and Public
Health
School of Medicine
Cardiff University
4th floor, Neuadd Meirionnydd
Heath Park, Cardiff CF14 4YS
Tel: (+44) 29 2068 7260
My book Confidence Intervals for Proportions
and Related Measures of Effect Size is now published.
Available at http://www.crcpress.com/product/isbn/9781439812785
See http://www.facebook.com/confidenceintervals
Home page http://medicine.cf.ac.uk/person/prof-robert-gordon-newcombe/
From:
Kim Pearce <[log in to unmask]>
To: [log in to unmask],
Date:
26/11/2013 12:20
Subject:
Multiple comparisons correction
- non parametric tests
Sent by:
A UK-based worldwide e-mail
broadcast system mailing list <[log in to unmask]>
Hello everyone,
This is just a very quick question. When applying 'multiple comparisons'
adjustments to p values from a table of statistical tests, all of
the papers/texts I have ever encountered have dealt with p values derived
from parametric statistics (Pearson's correlation coefficient, ANOVA etc).
Am I correct in thinking that such multiple comparisons corrections
e.g. sequential Bonferroni can also be applied to p values from non parametric
tests?
Many thanks for your advice on this matter,
Kind Regards,
Dr Kim Pearce PhD, CStat
Senior Statistician
Haematological Sciences
Institute of Cellular Medicine
William Leech Building
Medical School
Newcastle University
Framlington Place
Newcastle upon Tyne
NE2 4HH
Tel: (0044) (0)191 282 0451
You may leave the list at any time by sending the command
SIGNOFF allstat
to [log in to unmask], leaving the subject line blank.
You may leave the list at any time by sending the command
SIGNOFF allstat
to [log in to unmask], leaving the subject line blank.
