We continue on the interesting (but alas evidence-poor) question of
discipline-differences in the preprint/postprint difference ("DIFF"),
and, in particular, the question of what percentage of submitted papers
never gets published by any journal in any form.

On Wed, 6 Dec 2000, George Lundberg wrote:

> large numbers of papers submitted to biomedical journals are of
> insufficient quality to appear (either at all or in the form in which
> they were originally submitted/rejected) in any "good" journal

Helene was asking about the percentage in the "not at all" category,
rather than the revise-and-resubmit category, although both would be
of interest (if only anyone had actual data!).

> i am not at all sure that Stephen Lock's frequently quoted 1984 number
> bears any relation to current experiences

Lock reported that in biomedical research just about everything
eventually appears somewhere, in some form. So in the end the function
of peer review is to determine where (and, equally important, in what
form, with what content) a paper should appear: Peer review is not a
passive red-light/green-light filter, it is a dynamic, interactive,
iterative, corrective filter that actively changes the contents and
form of preprints.

    Lock, Stephen. A difficult balance : editorial peer review in
    medicine / Stephen Lock.  Philadelphia : ISI Press, 1986.

So, as a dynamic quality-shaper and certifier, peer review sign-posts
the level of quality of a paper at the locus where it eventually
appears -- a hierarchy of journals, from those with the highest
quality, rigour of refereeing, rejection rate, and impact factor at
the top, grading all the way down to journals so unrigorously reviewed
as to be little more than a vanity press.

(I am describing the standard lore here: I do not have data either.)

The function of this sign-posted hierarchy is to guide the reader and
the user, who have finite reading time and research resources, and need
to make sure they are reading reliable work, worth taking the risk of
building upon and worth citing. Researchers can pick their own level,
depending on their time, resources, and the aspired quality level of
their own work. They can decide for themselves how low in the
hierarchy they wish to go.

> At JAMA for my 17 years we rejected roughly 85% of all articles
> received. Many did appear in other journals, but a huge number seemed to
> simply disappear. We believed that was a good thing. i do not know of
> any recent study that hangs credible numbers on those observations.

Nor do I know of recent studies on this. (Does anyone?) But note that
apart from JAMA's 85% rejection rate (which attests to its being one of
the journals at the top of the clinical-medical hierarchy, along with
NEJM, Lancet and BMJ), George is not in a position to provide objective
data on what proportion of JAMA's rejected papers never went on to
appear anywhere, in any form. That would require a systematic follow-up
study (taking into account, among other things, title changes, and
possibly stretching across several years after the original rejection).

It would be splendid if someone gathered (or already had) such data.

I think we can all agree that in clinical medicine, where erroneous
reports can be hazardous to human health, it would be a good thing if
they never appeared anywhere, in any form. But in the online age
especially (what with child porn and hate literature proving so
difficult to suppress), this problem is well beyond the powers of
journals and journal editors.

    Harnad, S. (2000) Ingelfinger Over-Ruled: The Role of the Web in
    the Future of Refereed Medical Journal Publishing. Lancet (in
    press)
http://www.cogsci.soton.ac.uk/~harnad/Papers/Harnad/harnad00.lancet.htm

In the vast majority of research that has no bearing on human health
and welfare, however, it is not clear how strongly we should be
believing that it would be "a good thing" if a a "huge number" of
preprints rejected at one level of the hierarchy "simply disappeared"
rather than moved downward till they found their own level (including,
at the very bottom, permanent unrefereed status in the preprint sector
of the eprint corpus -- the eprint archives' "vanity press").

Who is to say what would be "a good thing" here for research, across
disciplines, a priori? This is the problem of the "wheat/chaff" ratio
that inevitably dogs every area of human endeavour: We would like to
have only the cream, and not the milk, but alas not only does human
performance invariably take the shape of a bell curve, but there is no
known way of ensuring that one can filter out the top 15% of that curve
without letting it all flow. (Not to mention that, peer review, being
human too, often misfilters, mistaking [to mix metaphors] wheat
for chaff and vice versa. The only protection against this is time,
and a retrospective record, for possible second thoughts about a piece
of work.)

    Harnad, S. (1986) Policing the Paper Chase. (Review of S. Lock, A
    difficult balance: Peer review in biomedical publication.) Nature
    322: 24 - 5.

> But i believe that the notion of posting "all" those that disappeared in
> some kind of unfiltered pre-print archive for the world to see would be a
> ridiculous waste of time and other resources and could seriously mislead

Are you sure? And have you been reading the diametrically opposite
views being expressed on this subject from other discipline (maths,
physics), where it is the unfiltered preprint whose praises some
people are singing, and the filtered postprint that they find a waste
to time?

But let me caution readers not to write all of this off as mere
discipline-differences: Neither the math/phys contingent nor the
bio/med contingent is providing data: it is just provide subjective
hunches. There is also a Simon-Says/Simon-Does ambiguity here, in
which practitioners may be (honestly) reporting that what they think
they are doing is X, whereas what they are actually doing is Y; the
same applies to their reasons for doing X/Y and their hunches about
the true functional role of X/Y. (Mathematicians' amnesia for the fact
that they are still submitting their papers to refereed journals, just
as they always did, even though they say all they care about is
unrefereed preprints, is an instance of this; the rationalization that
they submit to refereed journals "only for tenure" probably also ought
to be taken with a grain of salt.)

So caveat emptor, if one is trying to draw conclusions here, whether
they are on the basis of anecdotal evidence from authors, readers,
editors, or archivists. The objective data are not known.

And, I must add again, these data are not NEEDED in order to validate a
sure thing: that it would be "a good thing" to free the refereed
literature (postprints) online through eprint self-archiving right now:
The immediate desirability and optimality of that is not contingent in
any way on these ruminations about peer review, publication, rejection
rates, and the true +/- value of preprints.

Let us hope it is not held back by them either.

--------------------------------------------------------------------
Stevan Harnad                     [log in to unmask]
Professor of Cognitive Science    [log in to unmask]
Department of Electronics and     phone: +44 23-80 592-582
             Computer Science     fax:   +44 23-80 592-865
University of Southampton         http://www.cogsci.soton.ac.uk/~harnad/
Highfield, Southampton            http://www.princeton.edu/~harnad/
SO17 1BJ UNITED KINGDOM