Dear Richard,

I think this is an excellent heuristic example (at least from the
standpoint of another non-statistician.)  It should also be made clear,
however, that this really isn't about whether there are one or two groups
of jumpers, but is simply about what kind of question is really being
asked.  For example, even if we consider the data from only the
green-haired jumpers (incidentally, how many times have you watched Yellow
Submarine, Dr. Perry?), we could ask:

Does this group of individuals consistently jump significantly farther on
Thursdays than on Tuesdays?  (probably yes)

versus

Are Thursday's jumps by this group significantly longer than Tuesday's
jumps by this group? (probably no)


(very) analogous to the classic difference between doing a paired versus
unpaired t-test, which most people have a decent grasp of: I think what
often confuses people here is just the added terminology (fixed effects,
random effects, within group, between groups, etc.)


Dan Silverman, MD, PhD
Ahmanson Biological Imaging Center, Div. Nuclear Medicine
Department of Molecular and Medical Pharmacology
University of California, Los Angeles School of Medicine
[log in to unmask]


>Dear Steve,
>
>>You wrote:
>>---------------------------------
>>BUT THERE IS A MUCH MORE SERIOUS PROBLEM HERE...
>>
>>Sorry, you can't interpret the results at all if you are using a
>>fixed-effects analysis.  You must do a random effects analysis.  And
>>I am afraid that the results will be much less significant because of
>>the loss of all of those degrees of freedom.  (So I hope that you
>>have a clear a priori hypothesis!)
>>----------------------------------
>>
>>I assume you mean that one cannot use fixed-effects when comparing two
>>groups, e.g. normals and patients or males and females?
>>
>>Is there no sense in which you can compare two groups with a fixed-effects
>>model?  I'm asking because it seems people often might want to compare men
>>and women, but their inclination would be to run a fixed-effects model.
>
>It would be absolutely wrong to compare two groups with a
>fixed-effects model.  As I am no statistician, I will have to use a
>silly example to illustrate why.
>
>Let's imagine that you have done a study of the distance jumped by a
>certain 6 green-haired subjects on the long jump.  Each subject
>performs 10 jumps on various Tuesdays scattered throughout the study
>period, and you have the mean and variance of these jumps (and they
>are roughly normally distributed).  The means (in feet) are: 1.5,
>2.1, 2.6, 3.7, 6.5, 10.9.  The standard deviations are, in each case,
>less than 0.1 feet.  So although some of your long-jumpers are not
>very  talented, at least everyone in your sample is pretty consistent.
>
>All of these subjects have also done 10 jumps each on various
>Thursdays during the same study period.  These means from those jumps
>were 2.0, 2.5, 2.6, 4.5, 7.9 and 12.0.  Just eye-balling the results,
>it's pretty clear that there is something different about jumping on
>Tuesdays and jumping on Thursdays.  Given how small the
>within-subjects variance is, we can be pretty sure that the change in
>the mean is significant.  This is a bit like using a fixed-effects
>analysis, modelling Tuesday jumps and Thursday jumps separately for
>every subject, and drawing a confident inference because the residual
>variance is very small.
>
>Your study has proved so popular that the Wellcome Trust give you
>funding to extend your observations to people with blue hair.
>Unfortunately this bunch are only available on Tuesdays, but you want
>to find out if blue-haired individuals jump differently from
>green-haired individuals (at least, when the jumping is done on
>Tuesdays).  By alarming coincidence, the figures you obtain for the
>blue-haired bunch are 2.0, 2.5, 2.6, 4.5, 7.9 and 12.0 (the same as
>the green-haired people gave on Thursdays).  Does this constitute
>good evidence that they are better jumpers than the green-haired
>population?  Absolutely not.  There is a huge overlap between the
>distributions of jumps by green-haired and blue-haired jumpers.  The
>fact that observations within individuals are so consistent doesn't
>help you very much here; it's the between-subjects variance which is
>most relevant.  The appropriate test would be to take the mean from
>each blue-haired individual, and compare that group of 6 numbers with
>the 6 means from the green-haired subjects, without explicitly
>including the within-subjects standard deviations in the calculation.
>This is like a random-effects analysis.
>
>This example is intended to illustrate that the between-subjects
>variance becomes very relevant when you are comparing between groups.
>A fixed-effects analysis doesn't take appropriate account of the
>between-subjects variance, or the reliability with which you can
>predict the performance of a new individual from the test population.
>If you use a fixed-effects analysis to compare two groups and your
>within-subjects variance is very small but your between-subjects
>variance is very large, then you will erroneously interpret moderate
>differences between the two groups as significant, just because the
>residual variance (which doesn't contain any between-subjects
>variance) is small.  This wouldn't just be 'not quite kosher'; you
>could be seriously misled.
>
>The statisticians tell us that the random-effects approach weights
>within-subjects and between-subjects variance appropriately for
>comparisons between groups.  This might seem strange at first sight.
>After all, a random-effects analysis only takes one observation from
>each subject, so it doesn't seem to take account of within-subjects
>variance at all.  However, a simple example illustrates why it isn't
>so strange.  Imagine that, in fact, there is no between-subjects
>variance (the same subject kept sneaking back time after time, and
>her disguise completely fooled you).  Would there be no variance at
>all at the second level?  No, obviously there would still be some
>variance, and the reason why is because some of the within-subjects
>variance is carried through to the second level (by the effect of
>this variance on the single parameter estimate from each subject).
>
>I hope that this makes sense!
>
>Best of luck,
>
>Richard.
>--
>from: Dr Richard Perry,
>Clinical Lecturer, Wellcome Department of Cognitive Neurology,
>Institute of Neurology, Darwin Building, University College London,
>Gower Street, London WC1E 6BT.
>Tel: 0207 679 2187;  e mail: [log in to unmask]