Interesting... and after a bit of Googling it seems this rabbit hole gets a whole lot
deeper...! I've found this handy little online wizard walking you through a choice of
all different sorts of statistical tests, with tests attached to it at the end:
http://www.socscistatistics.com/tests/what_stats_test_wizard.aspx
Then there's this page offering a whole host of different effect size tests:
http://www.psychometrica.de/effect_size.html
Life was so much simpler before I knew about any of this! I think I need a cup of t.
Dave
--
Dr. Dave Sayers
Senior Lecturer, Dept Humanities, Sheffield Hallam University | www.shu.ac.uk
Honorary Research Fellow, Cardiff University & WISERD | www.wiserd.ac.uk
[log in to unmask] | http://shu.academia.edu/DaveSayers
On 28/01/2016 00:21, Angus B. Grieve-Smith wrote:
> It really sounds like you want effect size, then, and not significance, Dave.
>
> I find it usually helps me to understand the historical context of these tests.
> "Student" was actually a research scientist at the Guinness brewery in the early
> twentieth century, and developed his /t/-test to help the brewers ensure quality
> stout without testing every batch of ingredients. Here's a good article about him
> that was just published last month:
>
>> Gosset’s first problem involved figuring out exactly how many observations of malt
>> extract, a substance used in beer making, were necessary to be confident the
>> “degrees saccharine” of the extract was within 0.5 degrees of a targeted goal of
>> 133 degrees.
>
> http://priceonomics.com/the-guinness-brewer-who-revolutionized-statistics/
>
>
> On 1/27/2016 2:12 PM, Dave Sayers wrote:
>> To clarify a little detail here that I think I left a bit too ambiguous... the
>> sample in question was the one used during moderation (i.e. a colleague checking
>> the marks I've awarded to see if I've suddenly forgotten how to apply the mark
>> scheme - standard practice in the UK). I was checking my grades against the
>> moderated grades, to see how significant the difference was between the two, just
>> out of interest really (answer: not significant).
>>
>> I since applied the same thing to *all* grades from this year and last year on the
>> same module (also no significant difference).
>>
>> I'm probably making things worse by discussing significance in this way. Sorry
>> folks. I'm trying to keep up! Definitely going to add Peter's article suggestion to
>> my reading pile.
>>
>> Dave
>>
>> --
>> Dr. Dave Sayers
>> Senior Lecturer, Dept Humanities, Sheffield Hallam University | www.shu.ac.uk
>> Honorary Research Fellow, Cardiff University & WISERD | www.wiserd.ac.uk
>> [log in to unmask] | http://shu.academia.edu/DaveSayers
>>
>>
>>
>> On 27/01/2016 18:00, Angus Grieve-Smith wrote:
>>> I think David gets to a couple of good points here. Significance testing is all
>>> about answering the question, "Are the effect that I found, and the representative
>>> sample I used, large enough so that it's not likely I just grabbed a weird bunch of
>>> people/texts by accident?" If the answer is yes, then you are justified in
>>> generalizing the results to the population that you sampled.
>>>
>>> If you're saying you're only analyzing the grades for a random sample of your
>>> students, Dave, then yes, statistical significance means that you can generalize the
>>> results to the rest of your students, but that raises the question of why you didn't
>>> just analyze the grades for all your students.
>>>
>>> Presumably you're looking to make generalizations not just about your students, but
>>> about all the students at Sheffield Hallam, or all sociolinguistics students in the
>>> UK, or some other larger population. The problem with that is that your sample is
>>> biased, so you really can't, and there is no such thing as statistical significance
>>> for a biased sample.
>>>
>>> This is not at all the same as effect size testing, which answers the question, "Is
>>> the effect that I found big enough that it's worth paying attention to?"
>>>
>>> I hope this clears things up. Please let me know if it doesn't!
>>>
>>> On 1/27/2016 12:08 PM, David Bowie wrote:
>>>> Well, if you’re testing the entire population of students who have taken your
>>>> classes, then you have a (trivially) representative sample of students who have
>>>> taken your classes. But yes, it's not something that can be simply extended to the
>>>> population at large.
>>>>
>>>> This is a known issue in the scholarship of teaching and learning, though, since
>>>> you don’t generally get to choose your students to provide a random sample. Despite
>>>> that issue, however, it doesn’t keep SoTL research from providing useful results.
>>>>
>>>> Anyway, my main reason for replying: I’d suggest that effect size testing might be
>>>> as useful as, if not more useful than significance testing for this sort of thing.
>>>> The problem is that significance testing is tied so tightly to sample size, and
>>>> once you get up to a suitably large number of tokens, nearly any difference at all
>>>> will be statistically significant. Effect size testing gets around this issue, and
>>>> lets you see if a significant result actually reflects anything that’s large enough
>>>> to be meaningful.
>>>>
>>>> David
>>>>
>>>> On 26/Jan/16, 10:34 AM, "Teaching Linguistics on behalf of Angus B. Grieve-Smith"
>>>> <[log in to unmask] <mailto:[log in to unmask]> on behalf of
>>>> [log in to unmask] <mailto:[log in to unmask]>> wrote:
>>>>
>>>> Statistical significance is a good idea, Dave, but before you can
>>>> push your students (and yourself) to test for it you need to get
>>>> representative samples. If the students in your classes are not
>>>> assigned at random, they are not a representative sample of anything.
>>>>
>>>> http://grieve-smith.com/blog/2014/01/you-cant-get-significance-without-a-representative-sample/
>>>>
>>>>
>>>> On 1/26/2016 6:11 AM, Dave Sayers wrote:
>>>>
>>>> Hi TeachLingers,
>>>>
>>>> Having pestered my students for years to test their research data for
>>>> statistical significance, I suddenly wondered why I'd never
>>>> interrogated their grades in the same detail! I started today by
>>>> comparing my grades with the grades of the internal second marker. I
>>>> just suddenly realised that our decisions over whether a difference in
>>>> our grades warranted adjustment had so far been a bit unscientific.
>>>>
>>>> Running a statistical significance test used to require a fair amount
>>>> of training (that separated the wheat from the chaff in my A-level
>>>> biology class!) but nowadays it's really trivially easy with one of a
>>>> number of free online utilities, e.g.
>>>> http://graphpad.com/quickcalcs/ttest1.cfm.
>>>>
>>>> I'm also thinking that this could be of use for other purposes, e.g.
>>>> to compare different yearly cohorts to track improvement of a module,
>>>> or to compare a single student's grades across the years of their
>>>> degree to track their progress. There are lots of other possibilities
>>>> I'm sure.
>>>>
>>>> Has anyone used statistical analysis on grades before? What were the
>>>> effects? Were those effects... significant?
>>>>
>>>> Dave
>>>>
>>>> --
>>>> Dr. Dave Sayers
>>>> Senior Lecturer, Dept Humanities, Sheffield Hallam University |
>>>> www.shu.ac.uk
>>>> Honorary Research Fellow, Cardiff University & WISERD | www.wiserd.ac.uk
>>>> [log in to unmask] <mailto:[log in to unmask]> |
>>>> <http://shu.academia.edu/DaveSayers>http://shu.academia.edu/DaveSayers
>>>>
>>>>
>>>> ---
>>>> This email has been checked for viruses by Avast antivirus software.
>>>> https://www.avast.com/antivirus
>>>>
>>>>
>>>> --
>>>> -Angus B. Grieve-Smith
>>>> <mailto:[log in to unmask]>[log in to unmask]
>>>>
>>>
>>>
>>> --
>>> -Angus B. Grieve-Smith
>>> [log in to unmask]
>>>
>>
>> ---
>> This email has been checked for viruses by Avast antivirus software.
>> https://www.avast.com/antivirus
>
> --
> -Angus B. Grieve-Smith
> [log in to unmask]
>
---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus
|