Influences on Peer Review: Authors as Reviewers
July 27th, 2009
I usually stick to education research topics, but part of what influences what I see there is what actually gets published. And that is influenced by peer review. Peer review is a hotly debated topic in academia. This is probably not surprising since so much of professors’ evaluation, both formal (within the university) and informal (prestige in the field), depends on the outcomes. So, we fairly often see analyses of the types of things that may affect acceptance or rejection during peer review.
A group of researchers led by Lonnie Aarssen at Queen’s University looked at whether having published in top tier journals affected rejection recommendations when the authors served as reviewers (results published at PLoS ONE). They surveyed researchers (in ecology) and asked them to estimate their rejection rate in their reviews and whether they had published in any of a variety of journals the researchers identified as “top tier”. I have redone the crosstab table the authors present to include the percentages in each cell (this makes it easier for me to see the patterns).
| Rejection Recommendation | WITHOUT publication (percentage of column) | WITH publication (percentage of column) |
| 0-25% | 139 (31.4%) | 121 (18.4%) |
| 26-50% | 185 (41.9%) | 265 (40.2%) |
| 51-75% | 88 (19.9%) | 201 (30.5%) |
| 76-100% | 30 (6.8%) | 72 (10.9%) |
| Total | 442 (100%) | 659 (100%) |
From this, it seems clear that those with publication in the prestigious journals are much more likely to reject 51-75% of papers they review and much less likely to reject 0-25% of the manuscripts they review.
You might think that age is playing a factor here, with more senior scientists more likely to be published in the higher tier journals and more likely to reject. However, the researchers found no relationship between age and rejection rates.
I’m also interested in the top panel of Figure 1, showing the relationship between the number of high tier journals published in and rejection rates. It would be nice to have a scatterplot of all of the results, but we’ll go with what we have. To me, the growth in rejection rate from being published in 0 to 6 of the high tier journals is relatively modest. Then, there is a large jump in rejection rates for those who have published in 8 or 9 of the high tier journals. I wonder if the results would still be significant if those who had those very large numbers of journal publications were removed from the data.
I wonder if the journals that reviewers review for might interact with these findings. Is it more likely that people who publish in high tier journals will review for those journals? Do people adjust their standards based on which journal they are reviewing for?
Finally, I agree with the statement the authors make in the discussion, “Our results show that econlogists do not become more critical reviewers over time per se, but that they do likely become more critical as they publish more in high-IF journals.” I’m not sure about that cause-and-effect statement that getting published makes you more critical. However, I think this statement is more accurate than the statement in the abstract that says, “These results indicate that the likelihood of getting a paper accepted for publication may depend upon factors in addition to scientific merit.” I would argue that just because they are more critical does not mean that the reviewers are not judging on scientific merit. Although I suppose the probability of acceptance may depend on whether you get reviewers with a lot of high tier publications, to me those reviewers are still judging on scientific merit. These results do indicate that reviewers may have different standards, but I don’t think it is as damning as if there were evidence that the reviewers were basing their judgements on non-scientific factors. There does not seem to be evidence of that here.
Aarssen, L., Lortie, C., Budden, A., Koricheva, J., Leimu, R., & Tregenza, T. (2009). Does Publication in Top-Tier Journals Affect Reviewer Behavior? PLoS ONE, 4 (7) DOI: 10.1371/journal.pone.0006283
Tags: peer review
Posted in Publication | Comments (0)
Leave a Reply
Recent Posts
- Games and Simulation in Science
- Simulations and Labs: Either/or?
- Open Source, E-Textbooks, Traditional Texts… oh my!
- Predicting eLearning Dropout
- Development of Help-Seeking
Categories
- Assessment (3)
- Curriculum and Materials (5)
- Policy/ Government (2)
- Publication (1)
- Research Methods (4)
- Schools (1)
- Students (13)
- Teachers (8)
- Technology (8)
- Uncategorized (3)
