James said:

On 12 February 2013 08:18, James Alvarez <[log in to unmask]> wrote:
>
> Speaking generally, there are often good practical and theoretical reasons to reduce the amount of information in data.  Too much information is often harder to understand and is liable to just be noise - AKA to not see the wood for the trees.  It's probably better to address this at the point of collection rather than later but with a good a priori rational then there is no reasonnot to.
>
> 'More is better' when applied to variables and levels within variables is not always the case and probably often wrong.  I would also say its misleading to compare it to having more participants as one is a question of power and the other of resolution.
>
And Rosemary said"

"In short, transformation and loss do not equate to the same thing - and, of course, there are all kinds of reasons why we might make reasoned decisions to discard data too."


There might be reasons to discard data for interpretational purposes, but it's rare to see a case for analytic purposes. (Although one of the references at the end addresses this).

It's pretty easy to demonstrate that transformation and data loss do equate to the same thing. Here's a baby simulation written in R.

I generate 1000 samples, of 100 people. The mean in the population  is 0.01. For each sample, I run a t-test and see if the mean is significantly different from zero.


x <- rep(c(1:1000))
x <- as.data.frame(cbind(x, 100))
tt <- function(n) t.test(rnorm(n)+0.03)$p.value

for(loop in c(1:1000)) {
    x$V2[loop] <- tt(100)
}

mean(x$V2 < 0.05)


Here's the result;

> mean(x$V2 < 0.05)
[1] 0.858


This means that I'm getting a statistically significant result 86% of the time.  


Because all assumptions were satisfied, I can also use a simple power analysis to do this:


power.t.test(n=100, delta=0.3, sd=1, type="one.sample")



     One-sample t test power calculation

              n = 100
          delta = 0.3
             sd = 1
      sig.level = 0.05
          power = 0.8439467
    alternative = two.sided


So the analytic approach says that I have 84% power.  Pretty close.


As soon as you start to violate assumptions, we can't use power analysis any more, and we're stuck with simulations. (It's not hard to run these in SPSS either, but I don't have it, so I can't. It's easier in R anyway).


So then I categorize my variable. I'll cut it at -1.5, -0.5, 0.5, 1.5 - I'll create a 5 point scale, which is still pretty close to normally distributed.


> mean(x$V2 < 0.05)
[1] 0.737



Now my power is 0.74 - I've lost some information when I categorized the scale, and hence I've lost some power.  The more I categorize, the more information I lose, and the more power I lose.  (It's also worth noting that I haven't lost a great deal of power - people sometimes argue that you should not use parametric methods on a 5 point scale, but it hasn't really hurt us a lot - but it did hurt us.)

How many people is that equivalent to losing? We can use the power analysis to find out.

> power.t.test(power=0.74, delta=0.3, sd=1, type="one.sample")

     One-sample t test power calculation

              n = 77.24355
          delta = 0.3
             sd = 1
      sig.level = 0.05
          power = 0.74
    alternative = two.sided


So by categorizing, we get the same power that we would have got if we hadn't categorized and had 77 people. By categorizing we have the equivalent of losing 23% of our sample.


It's useful to think in terms of sample size equivalence. For example, it's most powerful to assign people equally to groups in experiments, but if one condition is more expensive or difficult than the other, you can get more power for you money (as it were) by randomizng unequally.  You need to think about sample size equivalents when you do that.

Here's a paper on that, http://www.jeremymiles.co.uk/mestuff/publications/p39.pdf


If you do group randomized studies (for example, kids in schools) you also need to think about power in terms of sample size equivalence.  For example, if you have kids in schools and you don't randomize by groups, you'll lose some power because of contamination - some kids will get the wrong treatment. If you do, you lose power because of the design effect (and this email is already long and boring enough, so I won't go into it).  Here's an extremely long and tedious report on that: http://www.hta.ac.uk/execsumm/summ1143.htm


If you're interested in this sort of thing, here are some papers worth reading which are written by people much cleverer than me:

Cohen, J. (1983). The cost of dichotomization. Applied Psychological Measurement, 7, 249-253.