I agree with Josef. To borrow a tortuous analogy from Baayen, I'd rather ride in a car driven by a rube and built by an engineer than a car driven and built by a hobbyist.
The major problems with using simple tests like chi-square, t-test, etc. is that:
1. The assumptions inherent in the test are often violated, in which case the statistical inference does not follow. For instance, the chi-square test assumes that the data is infinite (which is obviously never true), and returns misleading results for small amounts of data; the way Fisher's exact test is computed may be complicated compared to chi-square test, but the result is guaranteed to provide a correct estimate of the p-value.
There are similar issues associated with the t-test and the coefficient of variance, to name two mentioned in this thread.
2. In sociolinguistics, there are frequently many, many relevant predictors, but only one can be considered by these simple tests, meaning that the statistical inference does not follow, since we violate the assumption of independence by not including these as predictors in the model. For instance, in a recent paper I discussed a classic case of what has been called Simpson's paradox that resulted from the use of a chi-square test on a contingency table. I'll self-quote (and therefore self-promote, sorry!), because I'm sure the edited version is more eloquent that what I'd come up with spontaneously:
"One of the best-known examples comes from Bickle et al. (1975), who were recruited to assess whether Berkeley graduate admissions showed a gender bias. Analyzing the data as a group, the authors found that a female applicant was significantly less likely to be admitted than a male applicant. However, analysis by individual department showed that in most departments which had a significant trend towards either women or men, it was that a female applicant was significantly \emph{more} likely to be accepted than a male one. How could this be? Bickle et al. conclude that women are simply more likely to apply to competitive departments (particularly in the humanities) than men. It is not that the group statistic was incorrect, but simply that the significance interaction of gender and admission was only an indirect association caused by a more-innocuous interaction between gender and department choice."
Omitting an predictive variable (in this case, department choice) gave the wrong result. This omission is mandatory, more or less, when you restrict yourself to 2 x 2 contingency tables. A complex world requires complex models if we are to make causal inferences.
Kyle Gorman
On Sep 30, 2010, at 10:29 AM, Josef Fruehwald wrote:
> Yuri, I have to strenuously disagree with you. What linguists need to do is learn about the formulas being used to analyze the data, and how to understand the output. Researchers in all other quantitative research fields do this. A psychology journal would never accept a paper with a "manual" analysis of the data. There is no reason why sociolinguists shouldn't learn about these statistical techniques, and every reason why we must.
>
> -Joe
>
> On Thu, Sep 30, 2010 at 3:10 AM, Yuri Tambovtsev <[log in to unmask]> wrote:
> Dear Anne Marie, you are quite correct. When you use the statpack you loose control over the data. Usually linguists do not know by what formula the data are analysed. They just rely on the statpack. Therefore, when your data is not more than 100, it is better to analyse it manually. Be well, Yuri Tambovtsev, Novosibirsk
>
> The Variationist List - discussion of everything related to variationist sociolinguistics.
>
> To send messages to the VAR-L list (subscribers only), write to:
> [log in to unmask]
>
> To unsubscribe from the VAR-L list, click the following link:
> http://jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=VAR-L&A=1
>
>
>
> The Variationist List - discussion of everything related to variationist sociolinguistics.
>
> To send messages to the VAR-L list (subscribers only), write to:
> [log in to unmask]
>
> To unsubscribe from the VAR-L list, click the following link:
> http://jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=VAR-L&A=1
>
########################################################################
The Variationist List - discussion of everything related to variationist sociolinguistics.
To send messages to the VAR-L list (subscribers only), write to:
[log in to unmask]
To unsubscribe from the VAR-L list, click the following link:
http://jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=VAR-L&A=1
|