 |
 |
Dear Steve, >You wrote: >--------------------------------- >BUT THERE IS A MUCH MORE SERIOUS PROBLEM HERE... > >Sorry, you can't interpret the results at all if you are using a >fixed-effects analysis. You must do a random effects analysis. And >I am afraid that the results will be much less significant because of >the loss of all of those degrees of freedom. (So I hope that you >have a clear a priori hypothesis!) >---------------------------------- > >I assume you mean that one cannot use fixed-effects when comparing two >groups, e.g. normals and patients or males and females? > >Is there no sense in which you can compare two groups with a fixed-effects >model? I'm asking because it seems people often might want to compare men >and women, but their inclination would be to run a fixed-effects model. It would be absolutely wrong to compare two groups with a fixed-effects model. As I am no statistician, I will have to use a silly example to illustrate why. Let's imagine that you have done a study of the distance jumped by a certain 6 green-haired subjects on the long jump. Each subject performs 10 jumps on various Tuesdays scattered throughout the study period, and you have the mean and variance of these jumps (and they are roughly normally distributed). The means (in feet) are: 1.5, 2.1, 2.6, 3.7, 6.5, 10.9. The standard deviations are, in each case, less than 0.1 feet. So although some of your long-jumpers are not very talented, at least everyone in your sample is pretty consistent. All of these subjects have also done 10 jumps each on various Thursdays during the same study period. These means from those jumps were 2.0, 2.5, 2.6, 4.5, 7.9 and 12.0. Just eye-balling the results, it's pretty clear that there is something different about jumping on Tuesdays and jumping on Thursdays. Given how small the within-subjects variance is, we can be pretty sure that the change in the mean is significant. This is a bit like using a fixed-effects analysis, modelling Tuesday jumps and Thursday jumps separately for every subject, and drawing a confident inference because the residual variance is very small. Your study has proved so popular that the Wellcome Trust give you funding to extend your observations to people with blue hair. Unfortunately this bunch are only available on Tuesdays, but you want to find out if blue-haired individuals jump differently from green-haired individuals (at least, when the jumping is done on Tuesdays). By alarming coincidence, the figures you obtain for the blue-haired bunch are 2.0, 2.5, 2.6, 4.5, 7.9 and 12.0 (the same as the green-haired people gave on Thursdays). Does this constitute good evidence that they are better jumpers than the green-haired population? Absolutely not. There is a huge overlap between the distributions of jumps by green-haired and blue-haired jumpers. The fact that observations within individuals are so consistent doesn't help you very much here; it's the between-subjects variance which is most relevant. The appropriate test would be to take the mean from each blue-haired individual, and compare that group of 6 numbers with the 6 means from the green-haired subjects, without explicitly including the within-subjects standard deviations in the calculation. This is like a random-effects analysis. This example is intended to illustrate that the between-subjects variance becomes very relevant when you are comparing between groups. A fixed-effects analysis doesn't take appropriate account of the between-subjects variance, or the reliability with which you can predict the performance of a new individual from the test population. If you use a fixed-effects analysis to compare two groups and your within-subjects variance is very small but your between-subjects variance is very large, then you will erroneously interpret moderate differences between the two groups as significant, just because the residual variance (which doesn't contain any between-subjects variance) is small. This wouldn't just be 'not quite kosher'; you could be seriously misled. The statisticians tell us that the random-effects approach weights within-subjects and between-subjects variance appropriately for comparisons between groups. This might seem strange at first sight. After all, a random-effects analysis only takes one observation from each subject, so it doesn't seem to take account of within-subjects variance at all. However, a simple example illustrates why it isn't so strange. Imagine that, in fact, there is no between-subjects variance (the same subject kept sneaking back time after time, and her disguise completely fooled you). Would there be no variance at all at the second level? No, obviously there would still be some variance, and the reason why is because some of the within-subjects variance is carried through to the second level (by the effect of this variance on the single parameter estimate from each subject). I hope that this makes sense! Best of luck, Richard. -- from: Dr Richard Perry, Clinical Lecturer, Wellcome Department of Cognitive Neurology, Institute of Neurology, Darwin Building, University College London, Gower Street, London WC1E 6BT. Tel: 0207 679 2187; e mail: [log in to unmask]