Dear folks,

By now, I hope, most of you who are use inferential statistics are also
reporting the power of your analyses.....gulp, no?  Yeh, we've heard this
'power' stuff before but very few, including myself, have actually been
doing it.

I've just searched, downloaded and tested several free software packages
and add-ons for calculating and plotting power, alpha, effect size and
sample size.  The best listing of free power statistical software I found
was http://sustain.forestry.ubc.ca/cacb/power/#STAT-POWER.  Through this
page I found GPower an MSDOS program which is very simple, versatile and
powerful (and doesn't take a lifetime to download).  [I did find one MAJOR
bug, however...- if you use it let and I'll tell you!]

As a result of using GPower and based on previous experience, I'd suggest
the following guidelines for empirical outdoor education outcomes research:

POWER:  It is well accepted statistical advice in the social sciences that
we should plan and conduct studies with a power of at least .80 (i.e. so
that we have at least an 80% chance of correctly concluding a true
difference).  Despite this, the Hattie, et al. meta-analysis found an
average power of .65 (for 96 out.ed. outcome studies.  Future studies
should aim for .80.

EFFECT SIZE:  I recommend that, in general, we aim to detect a minimum
effect size of .20.  This is a rule of thumb based on:
a) Meta-analyses indicate an average outdoor education ES of just over .3. 
Hattie, et al. reported that approx. 68% of the reported outcomes were over
an ES .20;
b) The generally accepted Cohen benchmark for the social sciences is that
.2 is small, .5 moderate and .8 large.  Thus, it would seem to make sense
that our studies aim to detect 'at least a small effect'.
c) Attempting to detect smaller ESs necessarily involves larger sample
sizes; thus, it would not be practical, in general studies, to aim to
detect ESs much smaller than .20.

ALPHA: We should adopt the smallest alpha (to minimize Type II error) which
still allows the power to be above .80 (see Table).

Table *.  Two-tailed t test alpha levels for ES=.2 and Power = .80.

N	           Alpha	
>440	            .001	
340 to 439	.005	
300 to 339	.010	
200 to 299	.050	
160 to 200	.010	
115 to 159	.020	
Note. N's are rounded up to the nearest 10.

What does this mean.....?
- If you conduct a study with less than 115 people and want to try to
detect a small effect (i.e. .2), you will have less than an 80% chance of
doing so.  If you're study has more than 200 participants, then it is
recommended that you increase the normally used alpha of .05 to at least
.1.  Otherwise you'll still have a lower than acceptable power for the
study.
- If you conduct a study with between 200 and 299 participants, then use
alpha=.05.
- If your study has more than 300 participants, then it is suggested that
you minimize the likelihood of Type II errors by reducing the alpha.  This
is particular important for outdoor education studies where multiple
comparisons are performed on different outcomes.

I'm happy to elaborate on any of these points - and of course to hear
divergent views.

Cheers,
James


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%