In reading this paper there are certain obvious flaws (in my opinion) looking at it in terms of methodology and statistics.
1. There is no summary table of results of their twenty papers with additional pertinent factors.
In doing a study of any type the question comes up if the study's results are not due to other factors. The persons doing this study may have done nothing more than identify a subset of journals which have poor standards or there might be some systematic weakness in academic publishing by a sub-class of publications.
I would think that in this study the following factors in a table: How long has the journal been published?; How many subscribers does the journal have?; How long have the editors been editors?; How many reviewers did each paper get?; How many journals did they submit the paper to?; Did the journal ask for a fee?; Who were the publishers?; What journals turned down the paper?; What is the geographic location of publishers?.
Also, factors of methods of review. What was the turn around time for the journal in response to submission, turn around time for a response to reviewers, time to review final edited paper? There might be some factor in the process of accepting papers for publication by a journal, not an inherent flaw in these fields of critical studies.
2. How many journals did each paper get submitted to?
For example they states, "As for our performance, 80% of our paper overall went to full peer review at major journals across the field."
It would be useful to see a table of the journals each paper was submitted to. It might be that they are just identified some bad journals out of a larger set of journals.
Also, if you keep resubmitting papers over and over it might be that eventually you will find a weak point.
Also it is not clear if the papers were revised between submissions. There in a listing in the paper an item, "6 retired as fatally flawed or beyond repair." We "repair" being done with each rejection? Is there one paper being resubmitted or a paper that is evolving to get past editors.
Sooner or later the participant in this study found an overworked staff and reviewers and got past them.
The methodology of the study in my opinion is missing critical information.
3. Sample size
How many journals did these papers get submitted to? If the twenty papers that got accepted were submitted to a total of 200 journals it merely means that out of 200 journals 7 of them accepted the paper giving a rate of 3.5% for editorial failure.
Also, seven is a small number. What would the error bars be for this? There is no replication in this study. If run a few times, would most of the results be one through three? We don't know. We don't know because we are missing data on the submission rate.
We could estimate based on some table of papers, submissions, and timelines and get some rough, very rough idea.
One paper getting special recognition is possibly, what is called in statistics, a fluke, outlier, etc.
4. The authors are claiming that certain fields of study are flawed.
They don't have any comparison data. Did they pick other fields of study to do a similar publishing effort?
It might be that they have discovered that academic publishing is very dependent on assuming that the paper is submitted in good faith.
Also, how would they do such a comparison study being that they obviously have an agenda?
5. There is an estimate that they think they could have published 10 to 12 papers out of the 20. However, I don't see how they come with this estimate. Given that this is a key element in their assertion I would think that a calculation would be supplied.
6. Closure of the study and bias of result.
As the paper states, "... thus stop the study, before it could be properly concluded." I agree that it wasn't properly concluded.
The authors of this study however rushed to publish so they could discredit fields of study. There is the claim that seven papers are still in play. Were they really in play or were they really papers that were "fatally flawed?"
This paper has in my opinion fatal flaws.
To unsubscribe from the CRIT-GEOG-FORUM list, click the following link: