Hello List Members,
Thanks to all that replied to my query. There is not full agreement
about the "answer" to my question, but the reasoning is helpful.
Following are the answers...

Hello list members,

I have a question for you: Would you advocate "case 1" or "case 2"
below (or do you have a preferred  "case 3")?

Case 1.
If p is less than or equal to alpha, then reject null.
If p is greater than alpha, then fail to reject null.

Case 2. 
If p is less than alpha, then reject null.
If p is greater than or equal to alpha, then fail to reject null.

As you can see, for completeness I'm asking for your thoughts about the
highly unlikely (but possible) situation where p=alpha. 

For example, when using an alpha level of .05, what would you do in the
unlikely situation where the observed p-value is equal to .05 (i.e.,
alpha is set at .05 and the observed p=.05 to as many places as the
computer prints out). 

If you recommended case 1, I have a follow-up question about rounding:
What observed p-value would you consider close enough to be considered
"equal to .05" in the procedure? (The late Jacob Cohen offered a
convention that a p-value of .00 to .05 was sufficiently small, but
.051-1.00 was not sufficiently small to reject the null). 

Thanks in advance for your thoughts!

Burke Johnson

RESPONSES:

Burke, 
From a technical standpoint, with continuous variables Pr(P=(0.05))=0,
so it doesn't matter. In practice, if H0 is true, Pr(P=0.05) is a 
function of how many digits are displayed.  Most computers round rather
than truncate, so 0.0500, for example, could be anything between
and 0.04995 and 0.05004999999999999 barring a typo on my part.  It also
means that 0.0499 is less than 0.04995. I would follow whatever
prescription was stated in the protocol, which is typically that
P<0.05.  If someone wanted to call P=0.0500 statistically significant, I
would be
inclined to let him/her have his/her say depending on the consequences.
 What makes me angry is when I see someone round 0.05499 to 0.05
and try to claim P<0.05.
--Jerry.

Burke, 
Case 1 is your answer. The rounding off scheme is arbitrary. How about
0.0500000001?  Where do you round off. A similar scenario is if you
construct a 95% CI versus 94% CI and the decisions are different, what
we do then? Once you decide or obtain, you have to stay with it...
Regards,
--Satya Mishra


Burke,
In some sense you question is of little import. I don't mean to
denigrate you, or your thought process by saying this, but we must
remember that there is nothing "magical" about a p<0.05. There is no
science behind the choice of 0.05 as indicating significance as compared
to any other value. What is the difference between a p<0.04 which we say
is significant and one that is <0.06 which we say is not significant?
They both differ form the magical 0.05 by 0.01! The choice of 0.05 comes
from R.A. Fisher who pulled the value out of the air. It is, I believe
far better to give the effect size along with a measure its precision
(i.e. the SE) and a p value, or perhaps better the effect and a 95%
confidence interval around the effect without getting tied into knots
determining what is statistically significant and what is not. It is all
to easy to fall into the trap of saying that one will pay attention to a
test associated with a p<0.05 (or <=0.05) and ignore results with any
larger p value. 
We must also remember statistical significance does not mean a result
is important, and conversely.
--John David Sorkin M.D., Ph.D.


John (and others?), 
Please do not assume too much from my post. I am aware of and have
positions about everything you mentioned in your post (and, BTW, I agree
with what you said). I recently got into a debate about the specific
issue in my post, with a relatively well known quantitative methods
professor in psychology/education, and I wanted to see if any of you
agreed with his position (and, if so, to hear the reasons). For many
years I have been teaching Case 1, in conjunction with the
qualifications you mentioned, and many more. I also thought it might be
interesting to discuss the issues a little, again. For example, I would
argue that it is important to have a starting point case (case 1, case
2, or a case 3), when teaching NHST in introductory statistics classes.
Then, by adding and discussing the other related issues, we try to help
them learn to conduct thoughtful/reasonable statistical practice. I
believe that the social/behavioral/health sciences would have developed
more quickly if effect sizes and effect confidence intervals had been
used for the past 75 years. We still will get to where we want, but it
is going to take longer. For example, the APA Publication Manual, which
is used by multiple disciplines, has made some (perhaps not enough)
improvements on statistical practice and reporting, but many journals
are lagging behind.  
Cheers,
--Burke


Burke,
The technicality is that if p=alpha you fail to reject the null, but as
John Sorkin pointed out this has elements of farce about it.  It is like
saying that your cholesterol should be below 5.0, so if it is 4.999 you
are in perfect health, but if it is 5.000 you should start making
funeral arrangements.
Hope this helps
--Paul Wilson

Dear Burke
I would not consider p=0.05. I would calculate the exact p and them
think 
about my problem. 
Basilio de Bragança Pereira

Burke, 
Might be worth looking at: Gigerenzer, G. (2004) Mindless Statistics.
The Journal of
Socio-Economics, 33, , 587-606.
http://library.mpib-berlin.mpg.de/ft/gg/GG_Mindless_2004.pdf 
Regards .. Paul Barrett

Hello Burke .. thanks .. there was a chapter by Gerd and colleagues on
the same theme in: Gigerenzer, G., Krauss, S., and Vitouch, O. (2004)
The Null Ritual: What
you always wanted to know about significance tests but were afraid to
ask. In David Kaplan (ed.) The Sage Handbook of Quantitative
Methodology
for the Social Sciences, Chapter 21, pp. 391-408. Thousand Oaks: Sage
Publications. ISBN:0-7619-2359-4.
Some additional arguments are in there.
--Regards .. Paul

Burke,
Definition of p-value:  The smallest level of significance at which the
null
hypothesis can be rejected (below this we cannot).

So, if p-value= 0.032,

* Reject Ho at alpha.>= 0.032 (because you can still reject at 0.032),
and
** Do not reject Ho at .03199....

I say and preach this in my classes, however, I just saw the following
in
the textbook that we use for our ST 210,

"Reject Ho if alpha >p-value, and do not reject otherwise."

[Introductory Statistics by Prem Mann, John Wiley and Sons 2007, sixth
edition, page 388]

As alpha=p-value is a low probability item,  generally, a lot of people
play around with including or excluding alpha=p-value in Case 1.
However, the definition clearly states that as long as alpha does not
fall below the computed p-value, we would be safe in deciding  "reject
of Ho."

Actually, including alpha=p-value amounts to smaller beta value and
hence larger power.

Now, there is a whole lot of discussion on "whether to compute  p-value
first, then compare it with the given alpha or jut stay with p-value
alone."
It is a philosophical discussion and generally both sides claim
victory. Father of modern statistics, RA Fisher (of Cambridge, UK)
preached for p-value approach and another top-ranked statistician, Jerzy
Neyman of UC Berkeley [of Neyman-Pearson Lemma (1929) fame] preached
testing using the fixed alpha approach. While it is easier to explain to
students "testing with fixed alpha approach (due to cumbersomeness  of
manual computations of the p-value) using standard recipes;" the easy
access to p-value computation using various software has reduced the
problem to interpretation of the results.

Satya Mishra

Burke,
Though I agree with almost everything in the responses you got (of
course only those that copied the list), I had the feeling that no one
gave a direct answer to your question. In short prose, theory tells me
that the significance level is the probability to reject H0 if it is
true. This is warranted in case 1: if you reject H0 whenever your p
value is truly equal to 5%, the significance level is 5% and you are
fine. Now, if you use case 2, your true significance level MAY turn
out
to be smaller than 5%. Assuming a test statistic which has a
continuous
distribution, it is not TRULY smaller than 5% -- whatever smaller
significance level delta you would try, you could always find that it
cannot be the true significance level, as there is a p value > delta
but
smaller than 5%. So the true significance level is 5%, but you require
some kind of asymptotic, which you do not for case 1. So for me, it is
clearly case 1 already for ease.

Now, you can imagine examples where a p value of 5% is not only of
theoretical interest. This can occur if you have a discrete
distribution, and/or if you use a test statistic that follows a
discrete
distribution, or if you determine the null distribution by means of
re-randomizations (i.e. permutation tests). In the latter case,
imagine
a setting where you have 20 possible randomizations. Then, the one
yielding the largest test statistic would go with a p value of exactly
5% (in a one sided case). If you apply case 1, you'd have a chance to
reject H0, and you would do so whenever the test statistic is the
largest possible one. If you use case 2, you'd have no chance to
reject
H0, and then the power of the test would be 0 (!!!) no matter how big
the true effect. So it CAN indeed make a huge difference. 

Regarding your question on the required precision: I guess this is a
matter of taste. In theory, yes, you should determine the exact p
value,
but that's not always possible. Even less as your measurements are not
precise: If you assume a normal distribution of the residuals and
measure only 4 digits, then you loose some information. Formally, you
could try to determine the interval in which the p value will lie,
given
the precision of your measurement, and if the upper bound is below or
equal to 5%, you are fine, otherwise you cannot reject H0. For me it
sounds like a bad idea to rely on the 17th digit of the p value when
your measurements have only few digits. 

Back to practice: I agree with some posts that a p value of 5.1% tells
the same story as one of 4.9%, and that the levels are arbitrary.
Maybe
you want to look at the paper by Gelman and Stern in The American
Statistician 60 (2006) 328-331. But I am also aware that for some
people, this will make the difference between disappointment and
enthusiasm. And for those people, I prefer to use case 1, which has
slightly higher power in exceptional cases and will nevertheless allow
the same interpretation of the significance level. 
--HTH, Michael

You may leave the list at any time by sending the command

SIGNOFF allstat

to [log in to unmask], leaving the subject line blank.