Hi Phillippe,

If looking only at the figs. 3A,B,C in the PNAS paper alone, yes, I would 
agree with you that the proposed correlation is quite weak. Without the help 
of the trend lines, I would probably conclude that there is no correlation 
between the IF and number of retractions by a simple look. Of course the R 
squares are quite telling on the quality of the fitting already. On the 
other hand, the same authors had done similar analysis on a smaller pool of 
samples (17 journals) in an earlier study: 
http://iai.asm.org/content/79/10/3855.full, figure 1. It seems that when 
including way less journals, the trend stood out quite nicely - leading the 
authors to say "a strong correlation" in the earlier publication. I am not 
sure if the earlier clearer trend was a result of cherry picking, as the 
choice of journals looks quite normal – like a standard pool of journals 
one particular lab would consider to publish papers on.

It would also be interesting for us on the CCP4BB to try picking only the 
journals that we would consider to publish structures on, and plot the RI:IF 
graph to see what would happen. Compared to other fields of biology, frauds 
in crystallography is probably easier to detect, thus we need worry less 
about the false negatives: the low impact papers that were fraudulent or 
erroneous, but nobody cared to spend their effort battling.( I think when 
taking consideration of this, drawing conclusions from figs. 3A,B,C  would 
be even more dangerous.)

I would also like to bring two more issues for discussion:

One, in the 2011 IAI paper's fig. 1, the authors plotted Retraction Index, 
which is the total # of Retractions multiplied by 1000 then divided by total 
number of publication, whereas in the 2012 PNAS paper figures 3A,B,C, the 
plots simply used number of retractions to plot against the IFs. I wonder 
what they will look like if the figures 3A,B,C were plotted as RI vs IF – 
considering that many low or moderately-low IF journals publish huge numbers 
of papers.

The second issue is, in the PNAS figures 3A,B,C, at the lower left corner, 
although the dots have a dense looking, the viewers have to realize that 
most of them only represent 1 to 3 retractions. Ten of such points contain 
the same number of retractions that one point at the upper halves of the 
panels A and B contains. Maybe simple bar graphs for numbers of retractions 
in each IF bin would provide more help. Also, the fact that the averaged IFs 
landed at ~8 and ~12 for the fraud and errors cases (fig 3D) suggests that 
the absolute number of retractions occurred in high IF journals is quite 
significant, especially considering that there are way fewer journals with 
IF>10 than the ones with IF<10 in the 200-300 journals. So in my view, maybe 
trying to fit a straight line to the distribution is overly idealistic, some 
sort of partition does exist.

Zhijie


--------------------------------------------------
From: "DUMAS Philippe (UDS)" <[log in to unmask]>
Sent: Friday, October 19, 2012 6:15 AM
To: "Zhijie Li" <[log in to unmask]>
Subject: Re: [ccp4bb] PNAS on fraud

>
> Le Jeudi 18 Octobre 2012 22:52 CEST, Zhijie Li <[log in to unmask]> a 
> écrit:
>
> Thank you for this funny (and yet significant) comment.
> But I do not see clearly whether you agree with me or with the PNAS 
> paper....
> For me, this conclusion in the PNAS paper is just ridiculous.
> Philippe D
>
>> On curve fitting:
>>
>> http://twitpic.com/8jd081
>>
>>
>> --------------------------------------------------
>> From: "DUMAS Philippe (UDS)" <[log in to unmask]>
>> Sent: Thursday, October 18, 2012 1:52 PM
>> To: <[log in to unmask]>
>> Subject: Re: [ccp4bb] PNAS on fraud
>>
>> >
>> > Le Jeudi 18 Octobre 2012 19:16 CEST, "Bernhard Rupp (Hofkristallrat 
>> > a.D.)"
>> > <[log in to unmask]> a écrit:
>> >
>> > I had a look to this PNAS paper by Fang et al.
>> > I am a bit surprised by their interpretation of their Fig. 3: they 
>> > claim
>> > that here exists a highly signficant correlation between Impact factor 
>> > and
>> > number of retractations. Personnaly,  I would have concluded to a 
>> > complete
>> > lack of correlation...
>> > Should I retract this judgment?
>> > Philippe Dumas
>>
>
>
>
>
>