In the previous post, I mentioned some problems with Web of Knowledge. It might be best to say what these are more explicitly. (I’m grateful here to work Neil Levy did.)

• Web of Knowledge doesn’t include ‘early view’ articles on a journal website, and is a little slow to update on recent issues.
• Web of Knowledge doesn’t survey at all some journals, especially newer journals. (The most notable absence I’ve found so far is Philosophers’ Imprint, but obviously there are many others.)
• Web of Knowledge, by design, doesn’t cover citations in edited volumes.
• Web of Knowledge isn’t great for citations of things it doesn’t track cites in. So it isn’t great for citations of things in either newer journals, or edited volumes. Note that there aren’t many such articles in the Healy list; it would be interesting to see whether that’s a consequence of gaps in Web of Knowledge. I haven’t found an article that’s missing but should be there, but I haven’t looked hard. (I did check one article. But for all its citations, Epistemic Modals in Context hasn’t turned up much in the big 4 journals.)

That said, Web of Knowledge is not bad for citations in top journals of articles in top journals, provided you allow for some lag time. And that’s why I’ve focussed on journal-to-journal citation.

I’ve had some requests for looking at other relatively general journals. I actually wasn’t originally that interested in them; I was interested in the specialist journals, and wanted the more general journals as baseline. But for interest, here are the numbers for PPR.

Journal of Philosophy: 31
Mind: 41
Philosophical Review: 34

Brian Leiter summarised the data as saying these three journals cite other journals “not a lot”:http://leiterreports.typepad.com/blog/2013/07/how-often-do-articles-in-jphil-mind-and-phil-review-cite-articles-in-other-journals.html. I think that’s half-true; these three journals cite other journals with similar missions (e.g., Philosophical Studies, or Philosophy and Phenomenological Research) plenty. What they don’t do is cite the specialist journals, even ones on issues that I would have thought were pretty important parts of philosophy, such as feminism and legal theory.

One last data point about self-citation, because I really want to drive home the point that this isn’t something distinctive to the top journals. The number of articles in Philosophical Quarterly that cite other articles from 2000-2013 in Philosophical Quarterly is 106. Over the same period, it cites the Journal of Philosophy 14 times, and Philosophical Review 11 times. This is in part because, like Mind, Philosophical Quarterly has an active, and very good, discussion section. (One that I’ve been happy to publish in.) It is also in part because it has established itself as a home for certain discussions (especially about anthropic principle type reasoning), and these articles cite the leading articles in this discussion, which of course are in the Quarterly. There’s nothing particularly nefarious going on here, it’s part of running a good journal.

I’ve been playing around with the data about citation patterns in top journals of recent articles, and I started to think that a lot of it was very noisy. So I was looking for ways to aggregate the data that would be revealing. I decided to look at how many times recent articles, that is articles from 2000 onwards, from different journals were cited in each of the ‘top 3’ philosophy journals – _Journal of Philosophy_, _Mind_ and _Philosophical Review_. A few disclaimers before I start.

• This data is all from Web of Knowledge, with all the problems that entails.
• The number of articles in each of the top 3 journals in the time period in question is 489 in Journal of Philosophy, 1589 in Mind, and 672 in Philosophical Review, though most of those in Mind and the Review are book reviews. (I thought previously that book reviews in Mind and the Review were treated differently by Web of Knowledge. That was a mistake.)
• I’m not including Nous in this because Web of Knowledge treats Philosophical Issues and some years of Philosophical Perspectives as special issues of Nous, and separating out those from genuine issues of Nous is too hard for me to do with any accuracy.
• The counts below are of how many articles in each of the top 3 journals cite any article from the ‘source’ journal published in 2000-2013. In practice, this undercounts how many times the top 3 journal cites the source journal, since of course an article might cite multiple articles from the 1 source.

That said, here are the data. I’ll leave commentary on the data for another time, because they are interesting enough on their own. I would be very interested in hearing in the comments for suggestions for other journals that might suitably be analysed this way.

UPDATE 7/12: I’ve added some more thoughts in a “new post”:http://tar.weatherson.org/2013/07/12/update-on-journal-to-journal-citation/.

• Ergo, a new open-access philosophy journal, is now accepting submissions. I’m really happy that so many people have put so much work into getting this started, and that University of Michigan is hosting it. (Go Blue!) I’ve got at least 3 papers almost ready to send to them, and I hope this will make it much easier for people to send more of their papers to open-access venues.
• Christy Mag Uidhir (one of the many people behind Ergo) has some data about active faculty in aesthetics at top philosophy departments. I do worry that it is hard to measure from the outside how active aesthetics is at various departments. Aesthetics was more central to the research program at Rutgers than I had expected – as can now be seen in the work Gabe Greenberg is doing. And it is very active here at Michigan, although we have no active faculty who have published particularly recently in aesthetics. (Ken Walton is emiritus, but still around a lot, Dan Jacobson hasn’t published an aesthetics paper for a while, but is active on student committees and will probably publish in aesthetics again, etc.) But the data is still useful to have.
• And Steve Elliot (Arizona State) made this nice spreadsheet of highly cited journal articles via Google Scholar, which is a useful complement to the Healy data that has been much discussed in recent weeks.

One of the things to look for when doing citation searches is self-citations. If Joe Bloggs’s 1999 masterpiece has lots of citations, but it turns out that’s only because Bloggs has been citing Bloggs 1999 in everything he writes, then it isn’t clear that citation counts are measuring impact correctly. For that reason, many measures of citation exclude self-citation by authors. There are complications with this, especially with co-authored pieces. (If a paper by Stanley and Williamson cites Stanley and Szabo 2000, is that a self-cite? Should it be, given the rationale for excluding self-cites?) But overall I think it gives a better measure of citation impact.

We can ask the same question about journals. How much is the work in journal X being cited overall is a different question from how much it is being cited elsewhere. To get a sense of how big an issue this is, we can look at how frequently the journals publish articles citing other articles in that very journal. And that’s what I did. The table below counts how many articles in each of the big 3 journals cite articles published in the big 3 journals from 2000 to the present.

A few caveats before the table. I’ve taken this from Web of Knowledge, because I couldn’t figure out how to do the same thing with Google Scholar. I haven’t included Nous, partially because I’m lazy, and partially because of technical challenges. Perhaps most importantly, Web of Knowledge includes book reviews in Mind as articles, but not book reviews in Philosophical Review. That means the raw number of articles in Mind is much greater than in the other 2. (Roughly 600 articles in Mind, versus under 200 in Philosophical Review, and 350 in Journal of Philosophy.) That said, here’s the table.

As you can see, journals cite themselves a lot. The huge number in the middle in part reflects Mind’s love of discussion notes, replies to discussion notes, and so on. (And in part reflects the odd fact that book reviews in Mind are counted in Web of Knowledge.) But don’t let that distract you too much; every journal is like this to some degree.

Of course, this is what you’d expect if the journals specialised. If article authors think that discussions about conceivability and possibility go to Philosophical Review, about conditionals go to Mind, and about the nature of function go to Journal of Philosophy, and they think that on the excellent ground that that’s where such articles in the past have gone, then of course those articles will end up citing largely other articles from the same journal. (This fact, that people send articles on X to places that have recently published on X, I think explains the bulk of the data.)

It’s also what you’d expect if either (a) editors demanded that authors recognise their perceptiveness in what they had previously chosen to publish by citing it well, or (b) authors believed that editors demanded this, whether or not they do.

And it’s also what you’d expect if editors of journal X were aware that a paper previously published in X was relevant to some new article, and properly insisted on it being cited, while they were not as aware of (or not as insistent about) relevant literature in journals X and Y.

Figuring out which of these three explanations – or others – is correct is for another time. I just wanted to note the data.

One last point. I started to do this because I’d been asked whether the odd data in the previous post (about the _Journal_ not citing some classic papers in recent philosophical semantics) was part of a broader trend of the _Journal_ not citing recent literature as much as the other top journals do. I don’t think this data alone either confirms or disconfirms that hypothesis. It’s true that the _Journal_ cites the other top 2 journals at a lower rate than either Mind or Phil Review cite the other top 2 journals. But it’s not by much, and this could be a consequence of the _Journal_ having (either because of editorial policy or submission patterns by authors) a different range of topics than the other top 2.

Three of the most important papers in the philosophy of language/semantics overlap in recent years are:

• Dorothy Edgington, _On Conditionals_, Mind 1995
• Jason Stanley and Zoltan Szabo, _On Quantifier Domain Restriction_, Mind & Language 2000
• Jason Stanley, _Context and Logical Form_, Linguistics & Philosophy 2000

Between them they have nearly 1500 Google Scholar cites, and nearly 500 Web of Knowledge cites. They appear in “Kieran Healy’s dataset”:http://kieranhealy.org/blog/archives/2013/06/19/lewis-and-the-women/ 57 times between them. You might suspect that would mean that they would have to appear prominently in all four of the journals Kieran looks at. You’d be wrong.

According to Web of Knowledge, those papers have been mentioned, between them, exactly once in the _Journal of Philosophy_. In Jason Turner’s “Ontological Pluralism”:http://www.personal.leeds.ac.uk/~phljtt/papers/OP.pdf there is a mention, in passing, of the Stanley and Szabo paper. The other 56 citations in the Healy dataset come from the other 3 journals he looks at.

According to Google Scholar, things have recently gotten better. There has been one more citation of one of these papers. In Andreas Stokke’s “Lying and Asserting”:http://andreasstokke.net/docs/LyingAndAssertingPenultimate.pdf, published in the next-to-most-recent issue of the _Journal_, there is a mention, among a list of dissenters from radical contextualism, of Stanley’s _Context and Logical Form_. I assume that’s too recent to make the latest update to Web of Knowledge. (Neil Levy pointed out in comments to the previous post that Google Scholar often gives more accurate citation counts than Web of Knowledge. He’s right, especially about recent publications, and that prompted the search which led to this paragraph. Thanks Neil!)

But that’s it. There are no papers, at all, from the tradition of formal philosophy of language that Stanley and Szabo are writing in who cite their work in the _Journal_. There is no one at all who cites Edgington. By contrast, her paper has been cited in _Mind_ at least 15 times that I found, and I possibly didn’t find all the cases.

Obviously any journal will have areas it thinks are less interesting, or less important, than other journals do. That’s a good thing. But I think this particular gap in the _Journal_’s coverage of philosophy is unfortunate.

If you look at which philosophers have made the most impact by publishing in refereed philosophy journals over the last 10 years, it is easy to conclude that the one with the most impact, across academia, is Joshua Knobe. He not only has some very highly cited articles, he has a lot of articles that are cited many times. Indeed, his Web of Knowledge h-score for the last decade is, I believe, 11, which is nearly as good as some journals.

The only place Knobe’s impact doesn’t show up is in the Top 4 philosophy journals. He’s been cited 5 times in Journal of Philosophy, twice in Philosophical Review (both in the last year), 4 times in Nous (twice in his own articles), and _never_ in Mind.

That’s not because he is only getting cited outside philosophy. He has 20 citations in Philosophical Studies, 19 in Mind & Language, 9 in Analysis, 8 in PPR, and so on. It is something very distinctive about those four journals that means work like Knobe’s isn’t getting cited there.