Thoughts on Citation Data

As you probably have seen, Kieran Healy has done some amazing work extracting and presenting data about the citation patterns in Philosophical Review, Mind, Nous and Journal of Philosophy over the past 20 years. The point of this post is to pull out the data about work published in the last 11 years, and make a few brief comments on that work.

Here are the journal articles since 2002 that have 10 or more citations in some combination of Philosophical Review, Mind, Nous and Journal of Philosophy. They are presented in reverse chronological order:

“The Limits of Self-Awareness”, M. G. F. Martin, Philosophical Studies, 2004 (14 cites)
“The Silence of the Senses”, Charles Travis, Mind, 2004 (12 cites)
“What Conditional Probabilities Could Not Be”, Alan Hajek, Synthese, 2003 (12 cites)
“Some problems for conditionalization and reflection”, Frank Arntzenius, Journal of Philosophy, 2003 (10 cites)
“Color Realism and Color Science”, Alex Byrne and David Hilbert, Behavior and Brain Sciences, 2003 (13 cites)
“Assertion, Knowledge and Context”, Keith DeRose, Philosophical Review, 2002 (15 cites)
“Evidence, Pragmatics and Justification”, Jeremy Fantl and Matthew McGrath, Philosophical Review, 2002 (15 cites)
“The Transparency of Experience”, M. G. F. Martin, Mind, 2002 (14 cites)
“Basic Knowledge and the Problem of Easy Knowledge”, Stewart Cohen, PPR, 2002 (10 cites)
“(Anti-)Sceptics Simple and Subtle: G.E. Moore and John McDowell”, Crispin Wright, PPR, 2002 (10 cites)

And then there’s several more articles once we start going back to 12 or more years ago.

Here are the recent-ish books that have appeared more than 10 times in these four journals.

A Virtue Epistemology: Apt Belief and Reflective Knowledge, by Ernest Sosa, 2007 (12 cites).
The Philosophy of Philosophy, by Timothy Williamson, 2007 (11 cites)
Knowledge and Practical Interests, by Jason Stanley, 2005 (26 cites)
Insensitive Semantics, by Herman Cappelen and Ernest LePore, 2005 (11 cites)
Knowledge and Lotteries, by John Hawthorne, 2004 (36 cites)
Action and Perception, by Alva Noe, 2004 (15 cites)
The Realm of Reason, by Christopher Peacocke, 2004 (13 cites)
Evidentialism: Essays in Epistemology, by Earl Conee and Richard Feldman, 2004 (10 cites)
The Things We Mean, by Stephen Schiffer, 2003 (17 cites)
Making Things Happen, by James Woodward, 2003 (16 cites)
A Philosophical Guide to Conditionals, by Jonathan Bennett, 2003 (15 cites)
Thinking How to Live, by Allan Gibbard, 2003 (14 cites)
Reference and Consciousness, by John Campbell, 2002 (18 cites)
Beyond Rigidity: The Unfinished Semantic Agenda of Naming and Necessity, by Scott Soames (10 cites)

A few quick reflections on these lists.

All the entries are by men. I don’t think the best, or even the most influential, books and articles over this time period have all been by men. So this is more evidence in favour of the thesis that there is something wrong with our citation practices as a profession.

I believe that at least three of the papers (Martin 2004, Cohen and Wright) are from conference proceedings, not the usual blind submission to a journal. That’s a little bit fluky – none of the 11 papers on the list from 2000-2001 are conference proceedings. (Though 2 of them were from the special J Phil issue on causation.) Still, it’s striking that in the last 11 years, only 7 papers that were sent to blind review have made it onto this list of most cited entries. It takes a long time to get this kind of recognition.

There are no metaphysics papers. The only metaphysics book is James Woodward’s 2003 Making Things Happen, and you could just as easily classify that as philosophy of science as metaphysics. There is nothing from the material that you see most discussed at recent metaphysics conferences: existence, grounding, meta-ontology, indeterminacy, etc. From 2001 and further back, there are plenty of metaphysics books and papers on the list. The Healy data makes it look like metaphysics is much less central to philosophy than it was in, say, 2000. (This is a reason why I don’t like talking about ‘core’ parts of philosophy; the core changes rapidly.)

There are also no papers in ethics, political philosophy or history of philosopy. If you look at books, you get just one, Gibbard’s Thinking How to Live. If you go further back than 2002, you do start getting more ethics books – including books in normative ethics, not meta-ethics. But the most recent ethics paper I see on the list is David Widerker’s 1995 piece on Frankfurt cases (assuming free will is in ethics, broadly construed, not metaphysics). And I’m not sure I see any others after 1990. In part I think this reflects different subdisciplinary norms; value theorists write books more than, say, epistemologists do. And in part it reflects that the four journals we’re looking at don’t really represent philosophy as a whole. There is so much more ethics and history in the Philosophical Review than in the others, for instance, and even it has less ethics and history than many might like.

The 10 papers also don’t really match up with my internal feel for what’s being central to philosophy over that time, but that’s probably because my ‘feel’ isn’t perfectly reliable. (Although I have seen a few open searches recently, so I have some feel for what grad students are working on.)

The four epistemology papers towards the end of the list have been important, and widely discussed. The Arntzenius paper I believe gets there because one of its key examples, the Shangri La example, is so nice, and so widely discussed.

But the Hajek paper is I think a good example of why citation counts don’t necessarily correlate with what’s being talked about. Al’s paper is great. It is completely convincing, and I often cite it, both in papers and in conversation. But it’s not because I’m talking about that subject. It’s because it points out a common mistake, and it’s a mistake people keep making, and rather than explain why it’s a mistake I simply say “Don’t make that mistake, and read Hajek (2003) on why it’s a mistake.” But it’s not like there’s a rolling conversation about the nature of conditional probability that this represents.

The four philosophy of mind papers seem even less connected to what the bulk of people I see are talking about – though that’s perhaps less true of the Byrne and Hilbert paper, which is obviously central to active ongoing debates about color.

I saw several hundred job applications last year. Not that many of them were in philosophy of mind, despite UM specifically encouraging people with cog sci interests to apply. Those that were in mind typically were doing very different, much more empirical, work than Martin or Travis. That’s in part a selection effect due to our ad. But I didn’t see many of the people in the Leiter hiring thread who were doing this kind of philosophy of mind either. So I don’t really think the fact that the Martin and Travis papers are that often cited is that good a guide to what people are talking about; it might simply reflect how good those papers are, or something about the journals that cite them.

As well as the omissions mentioned above (women, metaphysics, ethics), there are two other surprising omissions from this list.

One is that there aren’t any book chapters represented. I don’t know whether that’s in part because the citations of those are a mess, and they don’t always turn up cleanly in databased, or simply that people aren’t citing them as much as I would have guessed. But given the explosion in edited volumes there seems to have been in recent years, I’m a little surprised that journal articles and books exhaust the list.

The other surprise is that there’s nothing about relativism, or at least nothing pro-relativism. (Relativism is a target, though not the primary target in two of the books shown.) John MacFarlane’s 1580 Google cites aren’t enough to get him into this company. That will change soon enough, especially when he publishes a book, but it’s one striking omission. And a quick search by hand through the Google Scholar citations of the widely cited relativism papers suggests that none of them are near 10 references in these four journals. Relatedly, there are no philosophy of language papers on the list, though there are four books.

The point is that it is a little simple to say that these four journals over emphasize language and epistemology relative to the rest of the profession. They do, but not all language and epistemology is created equal. To a first approximation, what’s widely cited in these four journals is something of a lagging indicator of what people are talking about. And that’s true even if we look at recent work that’s widely cited.

One final omission that perhaps isn’t that surprising: young people. None of these philosophers got their undergraduate degrees in the last 20 years. Only a handful (Cappelen, Fantl, McGrath and Stanley if I’m counting correctly) got theirs in the last 25. (Though there are several others who got degrees between 1988 and 1992 on the list if we include papers published before 2002: Pryor, Paul, Rysiew, Huemer and perhaps some others.) It takes a long time to get this kind of recognition.


Updates: June 25, 1:07 PM

  • Be sure to read Jennifer Nagel’s excellent comment below, using Google Scholar to track down some interesting citation patterns.
  • Via Twitter, Mark Eli Kalderon argued that my points about the Martin/Travis papers weren’t particularly strong, and I think he’s right. I started with the thought “This data doesn’t match my priors.” When that happens, the two things to do should be to either investigate further, or adjust credences. Really, what I should have stressed was just this mismatch. And the simplest explanation of that is that I have a partial view of the field (as everyone does.
  • In fact by a number of measures, the Martin and Travis papers have been talked about a lot. Jennifer Nagel’s comment discusses their Google Scholar counts. By web of knowledge they also have a lot of citations for articles in the last 10 years. I’m looking more into this, and hopefully have more to report soon, but for now I wanted to walk back a little what I said about Martin and Travis.
  • For more data relating to gender and citations, see the Gendered Citation Campaign database here.

5 Replies to “Thoughts on Citation Data”

  1. Hi Brian.

    I share your sense that there is some gap between the recent articles cited most heavily in the “Top 4” journals and the recent articles that have mattered the most. One way to look at the extent to which the top journals are idiosyncratic would be to look at the overall citation rates for recent articles heavily cited by those journals. When an article is fussed over at JPhil, is it comparably heavyweight in the world beyond?

    It’s a bit tricky to say, but one way we could do this would be to check, for each article, whether there is something in its citation trail (=some more recent piece citing it) which now overshadows it (=has more citations in Google Scholar). Interestingly, for nine out of the ten articles the answer is “yes”, and perhaps even more interestingly, for six out of those nine the overshadowing (and so perhaps now broadly more important) article or book is something not appearing on the top journal top 10 lists.

    Here are the overshadowed articles, and the pieces that cited them and are themselves now more cited:
    1. “The Limits of Self-Awareness”, M. G. F. Martin, Philosophical Studies, 2004 (14 cites); 195 cites in Scholar, topped by Clark, Supersizing the Mind, cited by 916

    2. “The Silence of the Senses”, Charles Travis, Mind, 2004 (12 cites); 181 cites in Scholar, topped by Chalmers, “The Representational Character of Experience”, cited by 226

    3. “What Conditional Probabilities Could Not Be”, Alan Hajek, Synthese, 2003 (12 cites)—173 cites in Scholar, topped by Sober, Evidence and Evolution, cited by 200

    4. “Some problems for conditionalization and reflection”, Frank Arntzenius, Journal of Philosophy, 2003 (10 cites), 65 cites in Scholar, topped by Elga, “Reflection and Disagreement”, cited by 206

    5.“Color Realism and Color Science”, Alex Byrne and David Hilbert, Behavior and Brain Sciences, 2003 (13 cites), 226 cites in Scholar, topped by Barrett, “Solving the Emotion Paradox”, cited by 373

    6.”(Anti-)Sceptics Simple and Subtle: G.E. Moore and John McDowell”, Crispin Wright, PPR, 2002 (10 cites) 79 cites in Scholar, topped by Pryor, “What’s Wrong with Moore’s Argument?”, cited by 185

    Three of the top ten articles got overshadowed by something that was itself on the top ten lists: the articles by DeRose, Fantl & McGrath and Cohen all got overshadowed by Hawthorne’s Knowledge and Lotteries.

    The one heavily-cited-in-the-top-journals article that isn’t overshadowed by anything in its citation trail is Martin’s “Transparency of Experience” (256 citations in Google Scholar), whose top-citing downstream article is Martin’s own “Limits of Self-Awareness”, which has 195 citations. That’s a perfectly healthy citation rate, but it’s below some papers which were more heavily cited elsewhere (for example, John MacFarlane’s “Future Contingents and Relative Truth”, cited by 241, which didn’t make the list of the ten most-cited articles).

    The general overshadowing phenomenon applies to articles rather than books: only one out of the top ten books got overshadowed by something not on the top ten list (Campbell’s Reference and Consciousness, overshadowed by Clark’s Supersizing the Mind).

    There’s one possible ground for optimism in the overshadowing phenomenon: the fact that an article gets cited a lot in JPhil doesn’t mean that it is destined to be the most talked-about thing on the topic across the profession. Our citation practices are dynamic enough (and diverse enough across the discipline) that new work comes up and gets recognized, at the expense of work that has been cited most heavily in the highest-status journals.

    There’s even an overshadowing work not by a male author, Barrett’s “Solving the Emotion Paradox”, which has 373 citations, overshadowing Byrne & Hilbert’s colour paper (226 citations). But Barrett is in psychology, which has quite different citation practices, and (perhaps this is not a coincidence) a different gender landscape.

    Because citation practices are different in different disciplines, one would of course have to interpret total citation rates with extreme caution. Work with interdisciplinary appeal does not necessarily have a value to philosophy that is proportional to its total citation rate.

    About your suggestion that what is talked about in those journals is a lagging indicator of what people are talking about: maybe one reason is the notoriously long refereeing and publication times at those journals. Perhaps these journals have not published articles on the articles that are now hottest elsewhere in the profession in part because their pipelines are clogged with things written quite a while ago. All the more reason to keep up with the fresher open-access venues, I say.

  2. One problem with the overshadowing method suggested by Jennifer Nagel is that philosophy is idiosyncratic in various ways, so you get different results if you measure citations within philosophy, narrowly construed, and citations across disciplines. Some papers and books attract citations from outside philosophy, and there are many more people outside philosophy than within, and speed of publication is much greater outside philosophy than within (for a number of reasons, some that reflect badly on philosophy – inordinately long review times, for instance – and some that don’t). The highest cited philosophers are those who attract citations outside philosophy as well as in, like Dennett and Chalmers.

    Obviously it is really hard to draw the inside/outside distinction. But using the H4 is not a bad heuristic for it. Some of Jennifer’s examples seem to be cases in which citations are explained by attracting interdisciplinary interest. The Clark book is a clear case; I think the Sober probably is too.

  3. In the case of the Travis and Martin papers (without questioning their intrinsic interest) maybe what’s going on is something you point out in the case of the Hajek article. I myself quote them in every paper where I talk about perceptual representations (to acknowledge that not everyone thinks that there are such things). So they get quoted in papers that are really not about the arguments put forward by Travis and Martin. Does this diminish their value? Certainly not. (see your point about Hajek…)

  4. That could certainly be true.

    But I think I expressed myself very badly in that paragraph. I wasn’t meaning to ‘diminish the value’ of those papers. I was expressing surprise that there were so many papers on perceptual representation in the top 4 journals, compared to (say) how many papers there were on truth relativism. I thought it had become more of a niche area – much like debates about grounding and fundamentality in metaphysics, or about vagueness in philosophy of language, etc. (I’m sad that vagueness went that way; it felt like a more mainstream topic for a few years in the late 90s early 200s.)

    Of course if there is a focus on perceptual representation (or its non-existence!) in the top journals it’s not surprising that there should be discussion of Travis and Martin; they’re great philosophers! But I was a little surprised that the antecedent was truer than I realised.

  5. And I might turn out to be wrong about the fundamentality/grounding stuff. It certainly feels to me like very few non-metaphysicians are talking about that. But maybe, especially thanks to Sider and Chalmers’s books, it will start to make an impact in debates in, for example, meta-ethics, and that will give it a broader appeal.

    But it certainly feels for now that it is primarily of interest to metaphysicians. Contrast the stuff that metaphysicians worked on more in the 80s and 90s: causation, laws, dispositions, intrinsicness, color, etc. Those debates were widely cited by non-metaphysicians, especially philosophers of science and philosophers of mind. I don’t think that will be true of the grounding stuff – though my track record at predicting this stuff isn’t great.

Leave a Reply