Dear Allstat members,
Here is a summary of the responses to my query. Many thanks to Martin
Bland, David Boniface, Philip McShane, and Jenny Freeman for their very
helpful advice, which is summarised below. I think I have got to the
bottom of the problem, which was due to some faulty assumptions on my
part. I was assuming that independent t-tests comparing factor level 1
to 2, 1 to 3, and 2 to 3, which were not corrected for multiple
comparisons, would provide the lowest significance values for these
comparisons, given that my reading thus far had suggested the post hoc
tests in SPSS ANOVA were more conservative (and therefore bound to
provide higher p-values). Discovering the p-values were extremely
similar confused me. However, if I now run my ANOVA and request LSD in
addition to Bonferroni and Tukey, then the p-values obtained for the
Bonferroni test are exactly 3 times larger than those obtained for the
LSD test (tukey results are roughly 3 times the LSD values also). So the
Bonferroni and Tukey tests are indeed correcting for multiple
comparisons, as I would have expected before I started double checking
my results against the t-tests, but the LSD test p-values happen to be
three times smaller than those for the t-tests, confusingly. I guess I
should have realised that the (uncorrected) post hoc p-values might
differ from the t-test results, because as David Boniface points out,
'when you run an independent 't' test the calculations only use
observations from the two groups involved. However, the full ANOVA uses
an estimate of the within groups Mean Square based on all observations
which is used in the post hoc tests. Hence you cannot directly compare
the two.'
I'm sorry my query turned out to be somewhat of a 'no-brainer': at least
this exchange turned up some useful links, kindly supplied by Jenny
Freeman, to sites which summarise post hoc tests in plain English [see
below]. I hope other Allstatters may find these helpful.
Thanks again,
Liz Hensor
................................................
Response from Martin Bland:
I think you are doing something wrong. You are right that the
Bonferroni P value should be about 3 times the independent t tests P
values, which should be similar to those you get using the LSD. Perhaps
your samples are very small and you have too few degrees of freedom,
e.g. only 3 observations per group would give you 4 df for each t test
and 7 for the Bonferroni.
................................................
Response from David Boniface:
When you run an independent 't' test the calculations only use
observations from the two groups involved. However, the full ANOVA uses
an estimate of the
within groups Mean Square based on all observations which is used in the
post hoc tests. Hence you cannot directly compare the two - they are
based on
different data. The SPSS post hoc tests are the more correct in this
situation.
...............................................
Response from Philip McShane:
The first point to note is that if you have 3 groups there are 2
independent comparisons, not 3 or 6. If the comparisons are not
independent then you cannot use Bonferroni. I don't find 'post- hoc'
tests very enlightening on the whole. If the ANOVA is significant they
are not all the same; if you want to look further you probably need to
think about what sort of differences it is reasonable to look for. Are
the groups ordered for instance, in which case regression might be
reasonable? If the overall test is not significant I would be wary of
basing conclusions on any further testing.
..............................................
Response from Jenny Freeman:
I recently decided to track down what all the post-hoc tests were in
SPSS and then collected the information in a single document. Most of
this info came from the web, so, as with any info on the web it is only
as good at the source, but they seem pretty reliable. Anyway, I have
attached the document to this mail, in case it is any help.
[Note from LH: this document was very helpful but is too large to
reproduce here. The information contained in it can be found at
http://www.uccs.edu/~lbecker/oneway.htm#5.%20Post%20Hoc%20Tests
<http://www.uccs.edu/~lbecker/oneway.htm#5. Post Hoc Tests>
http://www.id.unizh.ch/software/unix/statmath/sas/sasdoc/stat/chap30/sec
t18.htm#idxglm0337
http://www.sachina.edu.cn/ut/cgi-bin/topic_show.cgi?id=2018&h=1&bpg=1&ag
e=0
http://davidmlane.com/hyperstat/B98826.html
http://www.uvm.edu/~dhowell/StatPages/More_Stuff/MultComp/unequal_ns_and
_mult_comp.html
http://www.graphpad.com/instatman/Whichposttest_.htm
http://www.uky.edu/~ldesh2/mcpsummary.pdf
http://www2.chass.ncsu.edu/garson/pa765/anova.htm]
Dr Elizabeth M A Hensor PhD
Data Analyst
Academic Unit of Musculoskeletal and Rehabilitation Medicine
36 Clarendon Road
Leeds
West Yorkshire
LS2 9NZ
Tel: +44 (0) 113 3434944
Fax: +44 (0) 113 2430366
[log in to unmask]
|
