Refereeing Journals and Rants

Over at Brian Leiter’s blog there was a long thread recently about journal refereeing and reviewing practices. I thought I’d make a few points here that are getting lost in the crush.

1) In my experience, most absolute disasters with delays about refereeing concern (a) potential referees who simply don’t answer requests to referee, and (b) cases where the editors run out of people they know/trust on the relevant topic. If everyone who received a request to referee a paper could answer it, even in the negative, that day, and if answering negatively suggest 1-3 names of people with some expertise in the field, that would make things flow much more smoothly.

2) Relatedly, I think a lot of people, when refereeing, don’t take into account how time sensitive it is. Imagine you’ve got a paper that you’ve promised to referee within the month. And you’ve got a project of your own that is due at the end of that month. And you’ve got enough time in the month to do both. What should you do? I think the answer is that you should referee the paper straight away. Usually getting your paper done earlier won’t make a difference to anyone. Getting the report done earlier will make a difference. I think the system would work a lot more smoothly if every referee, upon getting a paper, seriously considered the question “Can I do this today?” Obviously if you have to present a lecture that day, or the next day, and it isn’t done, then the answer is no. But often times the answer is yes. It’s not like you’ll often spend more than a few hours on the paper, or that doing the paper that day will take more time, but it will make a difference to editors and writers.

3) If we want to keep the model of some journals being run through departments, rather than through publishers, then some amount of delay is going to be inevitable. If nothing else, most journals run by departments have a support staff of 1. If that one person is sick, or on annual leave for a time, the whole system basically creaks to a halt. If that person is spending literally all their time for a two or three week period getting an issue readyopt print, nothing happens with submissions. I’ve never had to deal with this, but I imagine if you don’t have good staff (or, more likely, don’t have good staff management) things are worse.

Probably the single biggest thing that could be done to improve journal response times would be to find a way to keep the system running when less than fully staffed. But it’s hard to do that in a small operation, when you can’t simply move staff from elsewhere onto the project.

4) The journal management software systems that are currently being rolled out make a huge difference. There’s nothing as good as keeping a paper from dropping off the face of the earth as reminders every few days that your report on it is overdue. (Since I sign off on every paper on Compass, I get a lot of these, but I’m not that late on too many.) Potentially these systems can, by automating processes now done by staff, help a lot with point (1). And that’s important, because otherwise point (1) seems to me to be intractable short of handing over all the journals to commercial presses.

Having said that, everyone hates the software when it is being rolled out. But it really makes all the difference in the world.

5) There’s been some discussion of cutting back on referee reports. I think this is basically a good idea. It’s true that referees need to say something to editors about what’s good or bad about a paper. But from experience I’ve learned that it’s much easier to find something informative to say about a paper to an editor than it is to say something informative and polite to an author. And anything that speeds up the process is probably good.

6) But I really don’t think the comments thread at Leiter is taking seriously how much of the problem is caused by there being too many papers being submitted. If every paper being submitted was a real philosophical advance, that wouldn’t be a problem – it would be paradise. But I don’t really think this is so.

Lots of papers I see to referee are basically glorified blog points that don’t attempt to make more than a very small point. Some of them would be quite good blog posts. But most journals aim a little higher than that. (Note this is different to the length point. Lots of good papers, even papers in top anthologies, are short. But they are all ambitious.)

Disturbingly, many papers seem to be largely unaware of the relevant literature, especially with the most recent developments. I see too many papers that simply don’t pay attention to relevant work from the last 10 years.

Now I don’t want to pretend that I’ve never written (or published) papers that fall in one or other of these categories. But I do think that many papers get sent out when the author could profitably have either rolled the paper into a larger paper, or spent time talking to colleagues/friends/blog readers about relevant literature that should be consulted.

I used to think this was a tragedy of the commons problem. (Mark van Roojen makes this suggestion in the Leiter thread.) The pressures to publish meant not quite cooked papers were being frequently released. And that’s too bad, but an inevitable consequence of everyone acting in enlightened self-interest. But really I don’t think that’s true.

That’s because I don’t think most people appreciate how important very very good papers are to one’s philosophical career. If you’re Tim Williamson or David Lewis you can write several papers a year that are important and groundbreaking. But most of us aren’t like that. Most of us will be such that most papers we write will sink without much trace. The vast bulk of attention will be paid to just a few papers. This can be seen in public through looking at citation rates. (Here are mine on Google Scholar for example.) The most cited papers have an order of magnitude more citations than the bulk of papers, especially when self-citations are removed.

And if we care about professional advancement as much as contribution to philosophical thought, the same story really holds. People tend to get hired based on their best papers. (And they tend to get passed over based on their worst papers.) This shouldn’t be too surprising. People are busy. They don’t have time to read a job candidates full dissertation, let alone their full output if they’re more senior. They read what is (reputed to be) the best work. And that’s what goes into hiring decisions. As we see every year when looking at junior hires, it doesn’t really matter if that best paper was published in Philosophical Review, the Proceedings of the Philosistan grad conference, or (more likely) the candidate’s own website. What matters is how good it is, or appears. As a rule, spending more time improving your best paper will do more for your professional prospects than sending it off and moving on to another paper.

Indeed, even if one just cares about publication, I imagine a lot of people (probably me included) could do with being slower on the “submit” button. Most, though not all, bad papers get rejected. And that takes time. Spending time making a good paper very good, rather than submitting the (seemingly) good paper may well mean one fewer rejection, and hence quicker publication.

So, simple solution to the problem of journals being so slow – don’t submit so much!

5 Replies to “Refereeing Journals and Rants”

  1. Problem is, like road traffic, everyone will think the solution is for everyone else to stop submitting so much, freeing up referees’ time for their stuff.

    Also, I have some doubts about the reasons you mention why one should work really hard on making one or two papers really good. Mainly because there’s a very good chance that these will end up amoung the uncited ones. Which papers get widely cited is not that well correlated with quality in my experience. Things just snowball (which explains the order of magnitude gap between the most cited papers and the rest) – once a few people get talking about a paper, more people notice it, and so on. And sometimes the reason for a high citation count is simply that there are lots of easily correctable errors. (One reason, incidentally, why I think citation counts are such a dubious marker of research quality.) One reason people like Lewis and Williamson have lots of multiply-cited papers is that once you get a reputation people start reading everything you publish, and will also go back and read stuff that probably – like most work by most people – sank without trace at the time of its original publication.

    That’s not to say one shouldn’t work really hard on one or two papers, if one prefers to work that way. But if one’s motivated by career concerns and professional respect it might well be a bad way to go. It might be better to get as many papers out there as possible, in the hope that some of them will catch the eye of the philosophical world. Then, if you end up with a Lewis/Williamson reputation, hopefully people will go back and read and cite your other papers too.

  2. I don’t think that theory of reputation is right. For some evidence, note that Identity and Discrimination has only 66 citations. That’s not a lot for a book by someone at a leading university. For a superstar, it’s practically nothing.

    Having said that, I don’t think that “Epistemic Modals in Context” is my best work, so I agree that citations aren’t the same as quality. But arguably it is the largest contribution to the literature that I’ve made. Writing a great paper that’s never read isn’t making the world a better place after all.

  3. I have some sympathy with this; I submit a lot less than I used to. I note, though, some tension between what you say in this post and the previous one. If a paper is unpublished, or published in the J Phlogiston St., I suspect that we will think less of it, even when we are well placed to judge it, because peer assessment matters to us. We can show that prior judgment acts as an anchor on expert judgment, even when the prior judgment is known to be inexpert and we are. When we are not experts on the topic, it should matter even more. If a paper has got past the reviewers at Phil Review, we should think more of it, and if we see a confusion in it, think twice.

    I completely disagree that the reason for exposing oneself to our peers is merely prudential, as you said in the last post. Knowledge is a social product. And submitting to journals is part of the process of hypothesis testing and correction. Getting papers to referees, and then to the community, is indispensable for refining ideas. So here’s a caveat on the idea that you should wait until your papers are really good before submitting. Say (to take a completely hypothetical example) you work on free will in Australia, where few people work on the problem. Suppose when you give papers on free will, the only response is “but we sorted all this out in the 60s”. Then your relevant peers are in the US, and you may find that you can present to them, at most, every 18 months or so. Then the peer review system becomes more important to you. People only have so much tolerance for blog posts, in terms of length (of course this doesn’t excuse trigger happy submitters who don’t wrestle with the tyranny of distance).

  4. I have doubts about whether your story about professional advancement holds outside toppish departments. In junior hiring, certainly no one is going to read more than one or two of your papers; but unless you’re fresh out of grad school, you may need publications to get to the point where anyone reads your paper at all, or reads it closely. If you’re not coming out of a top department, you may need publications when you’re fresh out of grad school. And I haven’t been involved in any tenure decisions, I know of at least one person who took the polish-your-best-work approach and was denied tenure for not having enough publications. (I think the department recommended tenure and the administration denied it.) So in both cases I think it’s probably safer to have several publications than to hope that your one publication will be cited lots.

    And if you need n publications by a certain time, then you need to send out some multiple of n submissions; and it’s not clear how much you can lower the multiple by improving the paper as much as possible. (Very rarely can you lower it to 1.) So I think it really is a tragedy of the commons.

  5. “Identity and Discrimination has only 66 citations. That’s not a lot for a book by someone at a leading university. For a superstar, it’s practically nothing.”

    But I bet it’s more than I&D would have if it weren’t for Vagueness, Knowledge and Its Limits &c. It’s noticeable how recent many of the citations of I&D are.

Leave a Reply