** APOLOGIES FOR CROSS-POSTING **
What follows below is -- I think readers will agree -- a conscientious
and attentive series of responses to questions raised by Phil Davis
about our paper testing whether the OA citation Advantage is just a
side-effect of author self-selection (Gargouri et al, currently under
refereeing) -- responses for which we did further analyses of our data
(not included in the draft under refereeing).
Gargouri, Y., Hajjem, C., Lariviere, V., Gingras, Y., Brody, T., Carr,
L. and Harnad, S. (2010) Self-Selected or Mandated, Open Access
Increases Citation Impact for Higher Quality Research.(Submitted) http://eprints.ecs.soton.ac.uk/18346/
We are happy to have performed these further analyses, and we are very
much in favor of this sort of open discussion and feedback on pre-
refereeing preprints of papers that have been submitted and are
undergoing peer review. They can only improve the quality of the
eventual published version of articles.
However, having carefully responded to Phil's welcome questions,
below, we will, at the end of this posting, ask Phil to respond in
kind to a question that we raised about his own paper (Davis et al
2008) a year and a half ago...
RESPONSES TO DAVIS'S QUESTIONS ABOUT OUR PAPER:
On 8-Jan-10, at 10:06 AM, Philip Davis wrote:
> PD:
> Stevan,
> Granted, you may be more interested in what the referees of the paper
> have to say than my comments; I'm interested in whether this paper is
> good science, whether the methodology is sound and whether you
interpret
> your results properly.
We are very appreciative of your concern and hope you will agree that
we have not been interested only in what the referees might have to
say. (We also hope you will now in turn be equally responsive to a
longstanding question about your own paper on this same topic.)
> PD:
> For instance, it is not clear whether your Odds Ratios are
interpreted
> correctly. Based on Figure 4, OA article are MORE LIKELY to receive
> zero citations than 1-5 citations (or conversely, LESS LIKELY to
receive
> 1-5 citations than zero citations).
> You write: "For example, we can say for the first model that for a
one
> unit increase in OA, the odds of receiving 1-5 citations (versus zero
> citations) increased by a factor of 0.957. Figure 4.. (p.9)
You are interpreting the figure incorrectly. It is the higher citation
count that is in each case more likely, as co-author Yassine Gargouri
pointed out to you in a subsequent response, to which you replied:
> PD:
> Yassine, Thank you for your response. I find your odds ratio
> methodology unnecessarily complex and unintuitive but now
> understand your explanation, thank you.
Our article supports its conclusions with several different,
convergent analyses. The logistical analysis with the odds ratio is
one of them, and its results are fully corroborated by the other,
simpler analyses we also reported, as well as the supplementary
analyses we append here now.
> PD:
> Similarly in Figure 4 (if I understand the axes correctly), CERN
articles
> are more than twice as likely to be in the 20+ citation category
> than in the 1-5 citation category, a fact that may distort further
> interpretation of your data as it may be that institutional effects
may
> explain your Mandated OA effect. See comments by Patrick Gaule and
Ludo
> Waltman on the review http://j.mp/8LK57u
Here is the analysis underlying Figure 4, re-done without CERN, and
then again re-done without either CERN or Southampton. As will be
seen, the outcome pattern, as well as its statistical significance,
are the same whether or not we exclude these institutions.
SUPPLEMENTARY FIGURE S1:
http://eprints.ecs.soton.ac.uk/18346/7/Supp1_CERN%2DSOTON.pdf
On 11-Jan-10, at 12:37 PM, Philip Davis wrote:
> PD:
> Changing how you report your citation ratios, from the ratio of log
> citations to the log of citation ratios is a very substantial
change to
> your paper and I am surprised that you point out this reporting
error at
> this point.
As noted in Yassine's reply to Phil, that formula was incorrectly
stated in our text, once; in all the actual computations, results,
figures and tables, however, the correct formula was used.
> PD:
> While it normalizes the distribution of the ratios, it is not without
> problems, such as:
>
> 1. Small citation differences have very large leverage in your
> calculations. Example, A=2 and B=1, log (A/B)=0.3
The log of the citation ratio was used only in displaying the means
(Figure 2), presented for visual inspection. The paired-sample t-tests
of significance (Table 2) were based on the raw citation counts, not
on log ratios, hence had no leverage in our calculations or their
interpretations. (The paired-sample t-tests were also based only on
2004-2006, because for 2002-2003 not all the institutional mandates
were yet in effect.)
Moreover, both the paired-sample t-test results (2004-2006) and the
pattern of means (2002-2006) converged with the results of the (more
complicated) logistical regression analyses and subdivisions into
citation ranges.
> PD:
> 2. Similarly, any ratio with zero in the denominator must be thrown
out
> of your dataset. The paper does not inform the reader on how much
data
> was ignored in your ratio analysis and we have no information on the
> potential bias this may have on your results.
As noted, the log ratios were only used in presenting the means, not
in the significance testing, nor in the logistic regressions.
However, we are happy to provide the additional information Phil
requests, in order to help readers eyeball the means. Here are the
means from Figure 2, recalculated by adding 1 to all citation counts.
This restores all log ratios with zeroes in the numerator (sic); the
probability of a zero in the denominator is vanishingly small, as it
would require that all 10 same-issue control articles have no citations!
The pattern is again much the same. (And, as noted, the significance
tests are based on the raw citation counts, which were not affected by
the log transformations that exclude numerator citation counts of zero.)
SUPPLEMENTARY FIGURE S2:
http://eprints.ecs.soton.ac.uk/18346/12/Supp2_Cites%2B1.pdf
This exercise suggested a further heuristic analysis that we had not
thought of doing in the paper, even though the results had clearly
suggested that the OA advantage is not evenly distributed across the
full range of article quality and citeability: The higher quality,
more citeable articles gain more of the citation advantage from OA.
In the following supplementary figure (S3), for exploratory and
illustrative purposes only, we re-calculate the means in the paper's
Figure 2 separately for OA articles in the citation range 0-4 and for
OA articles in the citation range 5+.
SUPPLEMENTARY FIGURE S3:
http://eprints.ecs.soton.ac.uk/18346/17/Supp3_CiteRanges.pdf
The overall OA advantage is clearly concentrated on articles in the
higher citation range. There is even what looks like an OA
DISadvantage for articles in the lower citation range. This may be
mostly an artifact (from restricting the OA articles to 0-4 citations
and not restricting the non-OA articles), although it may also be
partly due to the fact that when unciteable articles are made OA, only
one direction of outcome is possible, in the comparison with citation
means for non-OA articles in the same journal and year: OA/non-OA
citation ratios will always be unflattering for zero-citation OA
articles. (This can be statistically controlled for, if we go on to
investigate the distribution of the OA effect across citation brackets
directly.)
> PD:
> Have you attempted to analyze your citation data as continuous
variables
> rather than ratios or categories?
We will be doing this in our next study, which extends the time base
to 2002-2008. Meanwhile, a preview is possible from plotting the mean
number of OA and non-OA articles for each citation count. Note that
zero citations is the biggest category for both OA and non-OA
articles, and that the proportion of articles at each citation level
decreases faster for non-OA articles than for OA articles; this is
another way of visualizing the OA advantage. At citation counts of 30
or more, the difference is quite striking, although of course there
are few articles with so many citations:
SUPPLEMENTARY FIGURE 4:
http://eprints.ecs.soton.ac.uk/18346/22/Supp4_IndivCites.pdf
--------
REQUEST FOR RESPONSE TO QUESTION ABOUT DAVIS ET AL'S (2008) Paper:
Davis, PN, Lewenstein, BV, Simon, DH, Booth, JG, & Connolly, MJL (2008)
Open access publishing, article downloads, and citations: randomised
controlled trial British Medical Journal 337: a568
http://www.bmj.com/cgi/content/full/337/jul31_1/a568
Critique of Davis et al's paper:
"Davis et al's 1-year Study of Self-Selection Bias: No Self-Archiving
Control, No OA Effect, No Conclusion"
http://www.bmj.com/cgi/eletters/337/jul31_1/a568#199775
Davis et al had taken a 1-year sample of biological journal articles
and randomly made a subset of them OA, to control for author self-
selection. (This is comparable to our mandated control for author self-
selection.) They reported that after a year, they found no significant
OA Advantage for the randomized OA for citations (although they did
find an OA Advantage for downloads) and concluded that this showed
that the OA citation Advantage is just an artifact of author self-
selection, now eliminated by the randomization.
What Davis et al failed to do, however, was to demonstrate, in the
same sample and time-span, that author self-selection generates the OA
citation Advantage. Without doing that, all they have shown is that in
their sample and time-span, they found no significant OA citation
Advantage. This is no great surprise, because their sample was small
and their time-span was short, whereas the many of the other studies
that have reported finding an OA Advantage were based on much larger
samples and much longer time spans.
The question raised was about controlling for self-selected OA. If one
tests for the OA Advantage, whether self-selected or randomized, there
is a great deal of variability, across articles and disciplines,
especially for the first year or so after publication. In order to
have a statistically reliable measure of OA effects, the sample has to
be big enough, both in number of articles and in the time allowed for
any citation advantage to build up to become detectable and
statistically reliable.
Davis et al need to do with their randomization methodology what we
have done with our mandating methodology, namely, to demonstrate the
presence of a self-selected OA Advantage in the same journals and
years. Then they can compare that with randomized OA in those same
journals and years, and if there is a significant OA Advantage for
self-selected OA and no OA Advantage for randomized OA then they will
have evidence that some or all of the OA Advantage is just a side-
effect of self-selection. Otherwise, all they have shown is that with
their journals, sample size and time-span, there is no detectable OA
Advantage at all.
What Davis et al replied in their Authors' Response was instead this:
http://www.bmj.com/cgi/eletters/337/jul31_1/a568#200109
> PD:
> "Professor Harnad comments that we should have implemented
> a self-selection control in our study. Although this is an
> excellent idea, it was not possible for us to do so because,
> at the time of our randomization, the publisher did not permit
> author-sponsored open access publishing in our experimental
> journals. Nonetheless, self-archiving, the type of open access
> Prof. Harnad often refers to, is accounted for in our regression
> model (see Tables 2 and 3)... Table 2 Linear regression output
> reporting independent variable effects on PDF downloads for six
> months after publication Self-archived: 6% of variance p = .361
> (i.e., not statistically significant)... Table 3 Negative
> binomial regression output reporting independent variable effects
> on citations to articles aged 9 to 12 months Self-archived:
> Incidence Rate 0.9 p = .716 (i.e., not statistically significant)...
This is not an adequate response. If a control condition was needed in
order to make am outcome meaningful, it is not sufficient to reply
that "the publisher and sample allowed us to do the experimental
condition but not the control condition."
Nor is it an adequate response to reiterate that there was no
significant self-selected self-archiving effect in the sample (as the
regression analysis showed). That is in fact bad news for the
hypothesis being tested.
Nor is it an adequate response to say, as Phil did in a later posting,
that even after another half year or more had gone by, there there was
still no significant OA Advantage. (That is just the sound of one hand
clapping again, this time louder.)
The only way to draw meaningful conclusions from Davis et al's
methodology is to demonstrate the self-selected self-archiving
citation advantage, for the same journals and time-span, and then to
show that randomization wipes it out.
Until then, our own results, which do demonstrate the self-selected
self-archiving citation advantage for the same journals and time-span,
show that mandating the self-archiving does not wipe it out.
Meanwhile, Davis et al's finding that although their randomized OA did
not generate a citation increase, it did generate a download increase,
suggests that with a larger sample and time-span there may well be
scope for a citation advantage as well: Our own prior work and that of
others has shown that higher earlier download counts tend to lead to
higher later citation counts.
Bollen, J., Van de Sompel, H., Hagberg, A. and Chute, R. (2009)
A principal component analysis of 39 scientific impact measures
arXiv.org, arXiv:0902.2183v1 [cs.CY], 12 Feb. 2009, in PLoS ONE 4(6):
e6022, http://dx.doi.org/10.1371/journal.pone.0006022
Brody, T., Harnad, S. and Carr, L. (2006) Earlier Web Usage Statistics
as Predictors of Later Citation Impact. Journal of the American
Association for Information Science and Technology (JASIST) 57(8)
1060-1072.
http://eprints.ecs.soton.ac.uk/10713/
Lokker, C., McKibbon, K. A., McKinlay, R.J., Wilczynski, N. L. and
Haynes, R. B. (2008) Prediction of citation counts for clinical
articles at two years using data available within three weeks of
publication: retrospective cohort study
BMJ, 2008;336:655-657 http://www.bmj.com/cgi/content/abstract/336/7645/655
Moed, H. F. (2005) Statistical Relationships Between Downloads and
Citations at the Level of Individual Documents Within a Single Journal
(abstract only)
Journal of the American Society for Information Science and
Technology, 56(10): 1088- 1097
O'Leary, D. E. (2008) The relationship between citations and number
of downloads
Decision Support Systems. 45(4): 972-980 http://dx.doi.org/10.1016/j.dss.2008.03.008
Watson, A. B. (2009) Comparing citations and downloads for individual
articles
Journal of Vision, 9(4): 1-4
http://journalofvision.org/9/4/i/
|
