At least with the way my colleague has been pulling it, the DiD output doesn't include degrees of freedom. In another section, where she pulled some crosstab frequencies of covariates after matching, there is a 'design df' that equals the total number of observations minus 1, but I wasn't sure that applied to the DiD findings.
Margaret Schultz Patel | Research Associate, Workforce Specialist
Social IMPACT Research Center at Heartland Alliance
www.socialimpactresearchcenter.org
-----Original Message-----
From: A UK-based worldwide e-mail broadcast system mailing list [mailto:[log in to unmask]] On Behalf Of Stas Kolenikov
Sent: Thursday, April 02, 2015 10:10 PM
To: [log in to unmask]
Subject: Re: Calculating degrees of freedom with a weighted sample
This is only... sort of... correct for probability weights with unequal probability sampling, and no other features of the study design. For matching, this is most likely incorrect. I'd say that the degrees of freedom is at least the lowest sample size, or, if sampling weights are involved, the effective sample size of that sample. What does Stata give you for the degrees of freedom?
-- Stas Kolenikov, PhD, PStat (ASA, SSC)
-- Principal Survey Scientist, Abt SRBI
-- Education Officer, Survey Research Methods Section of the American Statistical Association
-- Opinions stated in this email are mine only, and do not reflect the position of my employer
-- http://stas.kolenikov.name
On Thu, Apr 2, 2015 at 5:40 PM, Helene Hoegsbro Thygesen <[log in to unmask]> wrote:
> The effective sample size is the squared sum of the weights, divided
> by the sum of the squared weights.
>
> On Thu, Apr 2, 2015 at 9:52 PM, Maggie Schultz Patel
> <[log in to unmask]> wrote:
>>
>> For a quasi-experimental impact study of a job training program, we
>> are using propensity score matching and a difference-in-difference
>> analysis. We have elected to use kernel matching because it gives us the closest match.
>> Since kernel matching compares the treated with a weighted average of
>> all untreated persons (i.e. with the highest weight given to those
>> with propensity scores closest to the treated), we’re a little stuck
>> in terms of interpreting statistical significance. For the
>> difference-in-difference, Stata only includes a t-stat. Since t-stats
>> require one to know the degrees of freedom to determine significance,
>> the question is how to determine degrees of freedom in this instance.
>>
>> Should we be calculating the degrees of freedom based on the number
>> of control group members with a weight greater than zero? Or is it
>> technically correct to use the entire (trimmed) control group
>> regardless of whether or not weight is being applied? Our sample size
>> is fairly large (over 500 for the treatment group and about 5000 for the total trimmed control group).
>>
>>
>> Any insights would be much appreciated.
>>
>> You may leave the list at any time by sending the command
>>
>> SIGNOFF allstat
>>
>> to [log in to unmask], leaving the subject line blank.
>
>
> You may leave the list at any time by sending the command
>
> SIGNOFF allstat
>
> to [log in to unmask], leaving the subject line blank.
You may leave the list at any time by sending the command
SIGNOFF allstat
to [log in to unmask], leaving the subject line blank.
Unless specifically indicated, nothing in this email should be interpreted as a digital or electronic signature that can be used to form, execute, document, agree to, enter into, accept or authenticate a contract or other legal document. This electronic mail transmission and any attached documents may contain confidential or privileged information for the sole use of the intended recipient(s). Any review, use, distribution or disclosure by anyone other than the intended recipient(s) is strictly prohibited. If you believe that you have received this message in error, please notify the sender by reply transmission and delete or destroy the message without copying or disclosing it.
You may leave the list at any time by sending the command
SIGNOFF allstat
to [log in to unmask], leaving the subject line blank.
|