To clarify a little detail here that I think I left a bit too ambiguous... the sample
in question was the one used during moderation (i.e. a colleague checking the marks
I've awarded to see if I've suddenly forgotten how to apply the mark scheme -
standard practice in the UK). I was checking my grades against the moderated grades,
to see how significant the difference was between the two, just out of interest
really (answer: not significant).
I since applied the same thing to *all* grades from this year and last year on the
same module (also no significant difference).
I'm probably making things worse by discussing significance in this way. Sorry folks.
I'm trying to keep up! Definitely going to add Peter's article suggestion to my
reading pile.
Dave
--
Dr. Dave Sayers
Senior Lecturer, Dept Humanities, Sheffield Hallam University | www.shu.ac.uk
Honorary Research Fellow, Cardiff University & WISERD | www.wiserd.ac.uk
[log in to unmask] | http://shu.academia.edu/DaveSayers
On 27/01/2016 18:00, Angus Grieve-Smith wrote:
> I think David gets to a couple of good points here. Significance testing is all
> about answering the question, "Are the effect that I found, and the representative
> sample I used, large enough so that it's not likely I just grabbed a weird bunch of
> people/texts by accident?" If the answer is yes, then you are justified in
> generalizing the results to the population that you sampled.
>
> If you're saying you're only analyzing the grades for a random sample of your
> students, Dave, then yes, statistical significance means that you can generalize the
> results to the rest of your students, but that raises the question of why you didn't
> just analyze the grades for all your students.
>
> Presumably you're looking to make generalizations not just about your students, but
> about all the students at Sheffield Hallam, or all sociolinguistics students in the
> UK, or some other larger population. The problem with that is that your sample is
> biased, so you really can't, and there is no such thing as statistical significance
> for a biased sample.
>
> This is not at all the same as effect size testing, which answers the question, "Is
> the effect that I found big enough that it's worth paying attention to?"
>
> I hope this clears things up. Please let me know if it doesn't!
>
> On 1/27/2016 12:08 PM, David Bowie wrote:
>> Well, if you’re testing the entire population of students who have taken your
>> classes, then you have a (trivially) representative sample of students who have
>> taken your classes. But yes, it's not something that can be simply extended to the
>> population at large.
>>
>> This is a known issue in the scholarship of teaching and learning, though, since
>> you don’t generally get to choose your students to provide a random sample. Despite
>> that issue, however, it doesn’t keep SoTL research from providing useful results.
>>
>> Anyway, my main reason for replying: I’d suggest that effect size testing might be
>> as useful as, if not more useful than significance testing for this sort of thing.
>> The problem is that significance testing is tied so tightly to sample size, and
>> once you get up to a suitably large number of tokens, nearly any difference at all
>> will be statistically significant. Effect size testing gets around this issue, and
>> lets you see if a significant result actually reflects anything that’s large enough
>> to be meaningful.
>>
>> David
>>
>> On 26/Jan/16, 10:34 AM, "Teaching Linguistics on behalf of Angus B. Grieve-Smith"
>> <[log in to unmask] <mailto:[log in to unmask]> on behalf of
>> [log in to unmask] <mailto:[log in to unmask]>> wrote:
>>
>> Statistical significance is a good idea, Dave, but before you can
>> push your students (and yourself) to test for it you need to get
>> representative samples. If the students in your classes are not
>> assigned at random, they are not a representative sample of anything.
>>
>> http://grieve-smith.com/blog/2014/01/you-cant-get-significance-without-a-representative-sample/
>>
>>
>> On 1/26/2016 6:11 AM, Dave Sayers wrote:
>>
>> Hi TeachLingers,
>>
>> Having pestered my students for years to test their research data for
>> statistical significance, I suddenly wondered why I'd never
>> interrogated their grades in the same detail! I started today by
>> comparing my grades with the grades of the internal second marker. I
>> just suddenly realised that our decisions over whether a difference in
>> our grades warranted adjustment had so far been a bit unscientific.
>>
>> Running a statistical significance test used to require a fair amount
>> of training (that separated the wheat from the chaff in my A-level
>> biology class!) but nowadays it's really trivially easy with one of a
>> number of free online utilities, e.g.
>> http://graphpad.com/quickcalcs/ttest1.cfm.
>>
>> I'm also thinking that this could be of use for other purposes, e.g.
>> to compare different yearly cohorts to track improvement of a module,
>> or to compare a single student's grades across the years of their
>> degree to track their progress. There are lots of other possibilities
>> I'm sure.
>>
>> Has anyone used statistical analysis on grades before? What were the
>> effects? Were those effects... significant?
>>
>> Dave
>>
>> --
>> Dr. Dave Sayers
>> Senior Lecturer, Dept Humanities, Sheffield Hallam University |
>> www.shu.ac.uk
>> Honorary Research Fellow, Cardiff University & WISERD | www.wiserd.ac.uk
>> [log in to unmask] <mailto:[log in to unmask]> |
>> <http://shu.academia.edu/DaveSayers>http://shu.academia.edu/DaveSayers
>>
>>
>> ---
>> This email has been checked for viruses by Avast antivirus software.
>> https://www.avast.com/antivirus
>>
>>
>> --
>> -Angus B. Grieve-Smith
>> <mailto:[log in to unmask]>[log in to unmask]
>>
>
>
> --
> -Angus B. Grieve-Smith
> [log in to unmask]
>
---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus
|
