Philosophy in the New York Times

There is a small symposium in the New York Times today about the recent trend in analytic philosophy towards experimental philosophy.

As some of the contributors note, it’s easy to overstate the trend that’s going on here. It’s not that for the 20th Century, philosophers used only armchair methods, and with the dawning of the 21st century they are going back to engaging with the sciences. When I was in grad school in the 90s, it was completely common to rely on psychological studies of all of uses, especially studies on dissociability, on developmental patterns, and on what was distinctive about people with autism or with Capgras Syndrome. And the influence of Peter Singer on work in ethics meant that purely armchair work in ethics was out of the question, whatever one thought of Singer’s conclusions.

This was hardly a distinctive feature of philosophy in south-eastern Australia. Indeed, we were probably more armchair-focussed than contemporary American philosophers. As Ernie Sosa notes in the entry linked above, 20th century metaphysics is shot through with arguments from results in 20th century physics. The importance of objective chance to contemporary nomological theories is obviously related to the role of chance in different branches of physics and biology, and modern theories of it involve a lot of attention to various sciences. And I’ve lost count of the number of debates I’ve been in in philosophy of language where appeal has been made at one stage or other to cross-linguistic data, which is presumably not armchair evidence unless we assume that the person in the armchair knows every human language. So it’s a bit of a stretch to say, as Joshua Knobe does, that in that time “people began to feel that philosophy should be understood as a highly specialized technical field that could be separated off from the rest of the intellectual world.” I’m really not sure which of the great philosophers of the 20th century could be characterised this way. (Perhaps if you included mathematics in philosophy and not the “rest of the intellectual world” you can get a couple of great 20th century philosophers in. But I doubt it would get much beyond that.)

That’s not to say there’s nothing new that’s been happening in the last fifteen years or so. In fact I think there are three trends here that are worth noting.

One purely stylistic, and actually rather trivial, trend is that philosophers are now a bit more inclined to ‘show their workings’. So if I want to rely on Daniel Gilbert’s work on comprehension and belief, I’ll throw in a bunch of citations to his work, and to the secondary literature on it, in part to give people the impression that I know what’s going on here. You won’t see those kind of notes in, say, J. L. Austin’s work. But that’s not because Austin didn’t know much psychology. I suspect he knew much much more than me. But because of very different traditions about citation, and because of differences in self-confidence between Austin and me, his philosophy might look a bit further removed from empirical work.

A more interesting trend is picked up by Ernie Sosa – philosophers are doing a lot more experiments themselves than they were a generation ago. This is presumably a good thing, at least as long as they are good experiments!

The university that Ernie and I work at, Rutgers, has a significant causal role in this. We encourage PhD students to study in the cognitive science department while they are at Rutgers, and many of them end up working in or around experimental work. That’s not to say I’m at all responsible for this – I’m much more sedentary than my median colleague. But many of my colleagues have done a lot to encourage students interested in experimental work.

The third trend, and this one I’m less excited about, is the reliance on survey work in empirical work designed to have philosophical consequences. It seems to me that surveying people about what they think about hard philosophical questions is not a great guide to what is true, and isn’t even necessarily a good guide to what they think. We certainly wouldn’t take surveys about whether people think it should be legal for an Islamic community center to be built around the corner from here to be significant to political theory debates about freedom of religion.

A slightly more interesting result comes from a survey that Matthew Yglesias posted this morning. If you trust Gallup, only 26% of Americans believe in “the power of the mind to know the past and predict the future”. This is a more than a little nuts, at least as interpreted literally. I know that I had blueberries with breakfast, and I can confidently and reliably predict that the Greens will not win the Australian election currently underway. And I know these things in virtue of having a mind, and in virtue of how my mind works. There’s the power of the mind to know the past and predict the future in action!

Of course, the 74% of people who apparently denied that the mind has the power to know the past and predict the future probably don’t really deny that I have these powers. The survey they were asking was about paranormal phenomena generally. And I left off part of the question they were asked. It asked whether they believed in clairvoyance, which they ‘clarified’ as the power of the mind to know the past and predict the future. Presumably at least some of the people who answered ‘no’ (or ‘don’t know’) interpreted the question as not being about the power of the mind to know stuff through perception, memory and inference, but through some more extraordinary method.

It’s in general extremely hard to understand just what qustion people are answering in surveys. And this makes it hard to know how much significance we should place on different surveys. This matters to some live puzzles. For instance, as Jonathan Schaffer recently wrote, there is an “emerging consensus in experimental philosophy, according to which … the magnitude of the stakes does not affect intuitions about knowledge.” (By ‘the stakes’ he means the stakes faced by a person about who we’re asking whether they know that p, when the person has to make a decision to which p is relevant.) This consensus is largely because the experimenters asked subjects whether certain fictional characters, some facing trivial decisions and some facing quite momentous decisions, knew that p, where p is something that would be important in their deliberations. Generally, they didn’t find a difference in the responses.

But there is quite a bit of evidence, including a lot of experimental evidence (PDF), that differences in stakes in this sense really do matter to cognitive states. In particular, what it takes to have settled the question to one’s own satisfaction of whether p is true, depends on what is at stake, and if you ask them the right way, survey respondents agree that it depends on what is at stake. Assuming, as everyone in this debate does, that knowledge requires settling questions to one’s own satisfaction, this means we have empirical evidence that stakes matter to knowledge. What does this mean for the consensus that Schaffer reports? I suspect it means, like in the Gallup survey, that different people are interpreting the survey questions differently, but there are lots of alternative explanations. In any case, I’d want a lot more evidence than surveys though before I overturned a well established result in experimental psychology.

41 Replies to “Philosophy in the New York Times”

  1. It seems that “I like the idea of experimental philosophy, but it just relies too much on the survey method” has become a common refrain in criticisms of experimental philosophy. (I’ve heard it from a few of my peers too.) I’ve always found this line of attack a bit puzzling, or at the very least, imprecise.

    As far as I understand, the survey method — in the salient sense — simply involves collecting data by asking for people’s self-reports, rather than, say, by observing and coding their behavior or scanning their brains. (Another sense, in which surveying involves no experimental manipulation, clearly does not apply to many experimental philosophy studies.) This method is commonly employed in social and (some) cognitive psychology. For example, studies in the confabulation literature use the survey method. They ask people to report on their judgments and how they come to those judgments. Of course, the experimental manipulations of those studies are what allows the interpretations of the data that led to the conclusion that people are unreliable in their self-reports about some mental processes.

    I take it the thrust of the anti-survey refrain is in fact not about the survey method at all, but about the manipulations and interpretations of some experimental philosophy studies. Fair enough. But not all experimental philosophy studies use the same experimental manipulations and the same approach to interpreting the data. So, it seems to me, specific criticisms of manipulations or approaches to interpretation would be much more helpful than broad ones. At the very least, broad criticisms should identify what the problematic manipulations or approaches to interpretation have in common, and which studies fall prey to these problems. It looks like the experimental philosophy community has done a decent job keeping itself in check on those counts (e.g., this and that) even though progress certainly comes incrementally and slowly.

    Moreover, once clarified, the thrust of the anti-survey refrain impacts philosophy’s interactions with empirical disciplines generally, and not just experimental philosophy. Perhaps the stake-manipulation study-designs do not adequately address epistemologists’ concerns and the straightforward inference from folk responses to philosophical conclusion is overly hasty. Even so, these are simply the kind of problems that frequently arise with interactions between philosophy and psychology (and likely other empirical disciplines too). Psychologists designing the experiments could easily, and perhaps even more likely given their lack of conceptual familiarity, miss philosophers’ concerns too. They might not know what the appropriate questions to ask either. Similarly, philosophers could misinterpret psychological findings. Additionally, in my experience, psychologists not infrequently misinterpret their own findings by making stronger conclusions than the data warrants; it would be bad, too, if philosophers were to draw philosophical implications by simply taking psychologists’ words at face value. This is not to say that philosophy ought not interact with empirical disciplines, but just that it’s hard generally. Consequently, I find the singling-out of experimental philosophy in this line of attack puzzling.

    There are a lot of good methodological criticisms of specific experimental philosophy studies or sets of studies, including criticisms from experimental philosophers themselves, as noted earlier. As such, let’s lay to rest the anti-survey criticism of experimental philosophy in general. Minimally, it’s bad sloganeering.

  2. There are at least two reasons one might not like the reliance on surveys in experimental philosophy.

    First, one might worry that administering surveys isn’t a very reliable way of learning about the considered judgments of the folk concerning various hypothetical examples, perhaps because the surveys are often poorly designed, or because of general problems with surveys (people don’t pay attention, etc.) It sounds like this is the sort of criticism you (samliao) have in mind.

    But one might also be skeptical about the use of surveys as the primary tool in experimental philosophy not because they’re not good at revealing the considered judgments of the folk, but because one might think that not much of philosophical interest follows from learning what those judgments are.

    This position is consistent with thinking that there’s quite a lot to be learned from empirical work in psychology, cognitive science, and neuroscience that’s relevant to philosophy. One might just think that the philosophically significant empirical findings generally don’t concern what the folk think about various examples. For example, when someone like Harman appeals to results from situationist psychology to criticize virtue ethics, the claims he’s appealing to don’t primarily concern what the folk think. Rather, they concern whether people actually have stable character traits of the sort that play a central role in virtue ethics; what the folk think about this question is a different matter.

    I suspect much criticism of the reliance on surveys implicitly takes the second form—often, it’s not that critics think the surveys fail to get at facts about folk judgments, but that they think that facts about folk judgments don’t have philosophically important upshots. While I’m certain that this position isn’t always the appropriate one, there are at least some cases in which I’m sympathetic to it.

  3. I actually do mean to be making the first criticism of the two Dan mentions. (Though I also think the second one is important.)

    I think it’s just very hard to go from how people answer questions to what they think. Answering any question other than the very simplest ones usually involves bringing a lot of background assumptions about just what is being asked into play. And I don’t think, as a rule, that this complication is properly accounted for in experimental work.

    As came up in the comments thread over at Crooked Timber, this NYT symposium is an example of this in action. It’s hard to find a clear question to which the different contributions are all answers. I think that’s because the different philosophers read very different things into the question. Some of the responses took the question to very much be about the kind of thing that’s usually discussed on the Experimental Philosophy blog. Others (esp. Tim Maudlin and Brian Leiter) did not. My guess, and I think this is just echoing a point that Simon Cullen has made, is that this kind of thing happens a lot in contemporary survey-based experimental work.

  4. Dan and Brian, what do you have in mind when you say that experimental philosophy uses surveys or the survey method? Are you thinking just pencil and paper, verbal probes? Or do you have in mind actual sample survey work (i.e., polling)? The difference is important, since the former often includes manipulations (as samliao points out). The latter seems open to the criticisms that you level—that getting at what people really think is hard to do and even if we knew, philosophy isn’t a popularity contest. But the former doesn’t seem so vulnerable to me. Also, while there is definitely some survey sampling in experimental philosophy, most work in experimental philosophy is not simple polling but involves manipulations.

  5. Ditto what Sam said. Well put.

    Related to his comment, I’m not sure we should jump to conclusions about knowledge attributions given the evidence from psychology about the importance of stakes to other cognitive states. I haven’t had a chance yet to look in detail at the studies, but I wouldn’t be surprised if the psychologists weren’t at all looking (and so finding anything out about) folk judgments. Schaffer’s claim was that there seems to be a consensus from x-phi that stakes don’t matter to folk intuitions. The name of the game in the philosophical debates here is whether people’s judgments change. Yet psychologists aren’t usually interested in this. They just look for effects. But the fact that factor F affects judgment J that P doesn’t entail that J went from J(P) to J(~P). (P could e.g. be: Hannah knows that the bank will be tomorrow.) In fact, it often seems to be overlooked that my co-authors and I found an effect of stakes. We just contended that this is no victory for Stanley because he should predict that the judgments would change, not just be affected to some extent or other.

    On Dan’s second criticism: It’s true one might go that route. But I can think of at least two problems.

    First, it gets most of its plausibility from focusing on rejecting what the folk think about general principles and philosophical theories. But most experimental philosophy is done by asking people what their intuitive reaction to a specific case is. And this is suppose to uncover something like the application of a concept with which they are meant be competent. (Thus, it might sound crazy for a linguist to be “concerned with what the folk think” if this is meant to be a concern with what they think about the principles of grammar. But it’s not so crazy if one is just interested in whether ordinary people think this or that sentence is grammatical.)

    Second, the assumption that it’s somewhat relevant what the folk think about certain cases isn’t specific to experimental philosophy. Many x-phi has been done testing what seem clearly the empirical claims that philosophers have made about specific scenarios (e.g. DeRose, Stanley, Schaffer, Wolf). I’m often puzzled by this sort of worry as well. Do you not think these philosophers are engaging in worthwhile endeavors? One might reply that these philosophers aren’t interested in folk attributions. But most of them explicitly are. They’re interesting in ordinary linguistic practices, intuitions, etc. (Wolf is much less explicit, and her topic might be amenable to the response that it is something of a technical term in the field: moral responsibility.)

  6. Jlive,

    I mean the kind of studies conducted in papers like this one (PDF). The short version is that the experimenters presented the subject a short story, then asked the subjects who in the story a particular character was talking about. This was then interpreted as revealing something about the subject’s views on reference.

    Now this is a particularly striking case of what I’m worried about, because the connection between what someone says a character is “talking about” and what the subject’s tacit theory of reference is looks rather indirect. It’s just very hard to know exactly what question someone is answering when you ask them who a character is talking about.

    To be sure, since people have made this criticism, there has been subsequent work trying to clarify just what the question is. But I think this kind of problem is endemic to the work I’ve seen.

    As far as I can see, these experiments are the first kind of survey work you mention. But I don’t see why that makes the experiments invulnerable to worries about manipulation.

  7. Brian, it’s surely difficult to interpret data. But you seem to be suggesting that there are some special problems with survey-data. I don’t see why. The issues you mention, for example with the Gallop poll, seem to just be problems arising with having humans as subjects. Just as it is difficult to learn anything about the human mind from surveys, it’s equally difficult to do so with fMRI scans or whatever the given tool is.

    In fact, I’d think that the surveys can sometimes be more straightforward. A potential case in point: Stanley says we get reaction X & Y about the two main bank cases. Then we go out and present people with the cases to see whether they get reactions X & Y. Of course there are potential problems here, the gap doesn’t at all seem especially wide compared to other empirical methodologies. In fact, one might say: “At least we’re not relying on some elaborate evolutionary explanation or brain scans of rough regions that are only associated with certain general kinds of mental processing!”

    There might be special limitations to the survey method if we’re trying to learn grand things about the mind. But I don’t see why they should be considered utterly worthless for testing whether people tend to have certain reactions to simple cases that employ concepts with which they’re supposed to be competent.

  8. Josh,

    It’s true that subjects are, more or less, competent with the concepts involved in these stories. But there are lots of salient concepts around here, and they can easily run them together.

    And, of course, there are performance errors. (Of the “The mouse the cat the dog chased ate was brown” variety.)

    And there are theoretical errors, where people misapply a concept, such as grammaticality, because they have a false theory of it. (E.g., “Every doctor checked their pager”.)

    Actually though, I doubt we need to appeal to any of those phenomena to cover your cases. On my version of interest-relativity, what matters is whether the bet on p the agent faces is at favourable odds. So let’s put your examples through that screen.

    If Hannah is wrong about the bank being open on Saturday, in the ‘high stakes’ case, she risks a $25 overdraft fee. (At least that’s how I’d naturally interpret the story, given how contemporary banks work.)

    If Hannah had checked the bank hours, she’d possibly have to pay a dollar or two for parking (that’s the price of short-term parking in Santa Barbara, right?) so assuming the bank is open potentially saves her that cost, as well as the hassle of finding a parking spot, walking to the bank etc.

    If the only reason she has for thinking the bank isn’t open is that banks sometimes change their hours, the good side of that bet looks like assuming it will be open, and living with the overdraft fee if she’s unlucky.

    It might be a little different if you ran the experiment in a more commercial environment (like Lower Manhattan) where many of the banks are Monday-Friday. (A quick check of banking hours around Santa Barbara suggests it’s kind of the reverse of New York – more central banks tend to be open on Saturday, and less so for the suburban ones. But I’m no expert on this.)

    Given all that, I’m a little surprised that you got as much of an effect as you did. Maybe adding in “very important” made some people think there was more on the line than the $25 overdraft fee, though it wouldn’t have had that effect on me.

    Anyway, here’s what I’d like to know about this experiment.

    (a) What did people think was going to happen to Hannah if she was wrong about the bank hours?
    (b) How difficult did people think it would be for Hannah to check the bank hours? Would she be able to just look them up on her iPhone (or on Sarah’s iPhone – Hannah is driving) or would she have to get out and read the bank window?
    (c) What was people’s views about how often banks in the story changed their hours?

    If the answer to (a) is an overdraft fee, the answer to (b) is that it would be a bit of a pain, and the answer to© is not very often, this isn’t even really a test of interest-relativity.

    I also think it’s wrong to say that Jason relies on intuitions about these cases, but that’s another matter altogether. Certainly the best case for Jason’s view (and my view) involves extreme odds – eternity in hell versus a dollar type extremes. If the intuitions about those were different to what we think, we might have to reconsider. But I don’t think we need to, or should, make any predictions about complicated realistic cases.

  9. Josh,

    I agree that philosophers engaged in debates about the correct semantic theory of “knows” make empirical claims about what subjects would be likely to say about various cases, and that these claims can be tested experimentally. If you’re interested in coming up with the correct semantic theory for “knows,” then you need to look at what English speakers say.

    That said, there are often cases where philosophers aren’t trying to to come up with the correct semantic theory for some bit of natural language. For instance, somebody who finds that the folk don’t use “free will” in line with her preferred theory of free will might be perfectly reasonable in holding that English usage doesn’t capture the most philosophically fruitful concept in the neighborhood; you might think that if we want a concept that will play certain theoretical roles (e.g., interact with moral theories about punishment, responsibility, etc.), we need a slightly technical one that isn’t obviously expressed by any piece of natural language.

    This may even be true in epistemology—while many philosophers are interested in the semantics of “knows”, a lot of epistemologists tend to prefer to focus on slightly technical terms (e.g., “justification,” as used by epistemologists) that are supposed to play certain theoretical roles, (e.g., it’s the sort of thing that increases in paradigm cases of confirmation) but which may not be neatly captured by any piece of natural language.

    The more you tend to think that the interesting philosophical work involves something closer to Carnapian explication than standard conceptual analysis—the more you think we should be figuring out which concepts can play which theoretical roles, as opposed to which concepts are expressed by which parts of natural language—the less straightforward it will be to get important philosophical upshots from facts about ordinary speakers’ judgments about cases.

  10. Brian,

    I think I agree with your worries about the problems here with surveys and the bank cases. I’m not sure they’re devastating for our project, though, given that we were just trying to see what people would say. And our contention was only that this would be interesting for philosophical theorizing.

    I’m not sure what to think about how Jason is really making his case for his view. He certainly seems in the book to be appealing to what we’d ordinarily say about the cases as strong evidence (of course defeasible, though, since he thinks even IRI can’t capture all the cases). That of course doesn’t mean he can’t change his view and rest his case more on the principles connecting knowledge and action. And that doesn’t mean he can’t dispute some of the details of our experiment or contend that his scenario isn’t really a good candidate for being presented to the folk (which he’s said on Certain Doubts). Our project, again, was more modest. As we make clear, we weren’t trying to send IRI to the grave. We were just trying to undermine an argument that Jason seems to give in the book, or at the very least an argument one might give that looks a lot like the one in Jason’s book.

    So I still see no reason to think there is anything especially (and systematically) problematic about using survey methodology in some areas of philosophy. All it seem we’re doing in the comment thread here is engaging in philosophical debate about the details. If avoiding that is the criterion for an accepatable methodology, then it seems we’d have to throw out the more standard philosophical methods as well!


    I think I agree with everything you said, and without qualification! That’s why I said Wolf is likely not in the same boat as recent epistemologists. And,frankly, I think the focus on language in epistemology is not the best focus. However, I do think, as you seem to as well, that the ordinary notion of knowledge (though probably not justification) is more important than the ordinary notions of responsibility, free will, and the like to the debates in those fields.

    I hope this all just goes to show that an x-phier can be an x-phier, and x-phi can be x-phi, while holding the clearly needed middle ground that surveys are limited. The worry from the x-phi end, as I see it, is that sometimes people come off as if they hold the opposite extreme: that surveying ordinary folks in philosophy is practically worthless.

  11. This thoughtful post and replies have certainly given me a lot to think about, but as an x-phi non-expert I’d like to make a case for one use of x-phi that hasn’t really been directly addressed above—the place for x-phi in pedagogy. If we wish students to be fully informed about how our interests matter to people, then x-phi results have a rather natural place in just about any curriculum I can think of, even if only to show extents of agreement, dissent, or confusion about key terms and concepts. I’ve done anonymized surveys in classes for over two decades and find the results to be eminently useful in many ways both to individual classes and as collective results over time. The pedagogical justification for x-phi, in my opinion, is sufficient for its pursuit and refinement.

  12. I think we can all agree that it is often very difficult to conduct studies in just the right way, so as to tap into exactly the intuitions that are most relevant. Clearly, the best approach here involves a continual discussion in which researchers note potential problems with existing studies, and then other researchers conduct new studies to resolve those problems. (For a good example, consider the way DeRose’s recent paper notes three specific problems with existing research on contextualism, and then Buckwalter responds by conducting a new study that resolves those three problems.)

    But this does not seem to be a difficulty involving experimental studies in particular. Rather, it seems to be a perfectly general difficulty that arises for any approach, whether experimental or armchair, that involves intuitions about hypothetical cases.

    For example, studies consistently show that people’s intuitions about cases can be impacted by the order in which the cases are presented, and experimental philosophers are therefore careful to either give each participant only one case or present the cases in a randomized order. But studies also show that philosophy professors are vulnerable to precisely this same effect, and yet papers that use more armchair appeals to intuition are forced to present cases in a single fixed order.

    In other words, I think the critics are exactly right to emphasize the subtlety of the issues surrounding experimental design, but these issues arise with equal, if not greater, force for approaches that do not use actual experimental studies.

  13. Brian,

    That paper by MMNS is a difficult blend. The criticism they launch depends on an estimate of the difference between the semantic intuitions about reference in two different populations. If the probes fail to get at the relevant intuitions (and I think they do), then the comparison is worthless. However, the approach is definitely not an attempt to survey people on the assumption that they will tell us what is true.

    What I’m worried about is that when critics of x-phi talk about “survey methods” they seem to think that x-phiers are simply polling ordinary people under the assumption that whatever the majority of ordinary people say is likely to be true. I think you suggest such a view when you write, “It seems to me that surveying people about what they think about hard philosophical questions is not a great guide to what is true.” Indeed! Simple polling about hard questions is an awful guide to the truth, pretty much regardless of discipline. (I’m assuming the hard questions aren’t of the form, “What do people think about p?”) But I’m not convinced that most x-phi actually looks like that. MMNS’s work certainly doesn’t.

    Now, whether or not you think MMNS’s work or any of a number of other x-phi projects was worth doing might plausibly depend on what you think about the role of intuitions in philosophy. I am sympathetic to your (and Cian Dorr’s) line that ordinary use of “intuition” is just a colorful way of flagging undefended premisses. As a normative claim about how “intuition” ought to be used in philosophy, I completely agree. I’m skeptical, however, that it is descriptive of what most philosophers think they are doing. If your view of intuitions were widespread among philosophers, a good chunk of x-phi would be really pointless, since it aims to convince philosophers that they have no special intuitive faculty that reliably tracks intellective truth.

    Anyway, I didn’t say that studies that manipulate some feature of a verbal probe are invulnerable to criticism. And I’m not saying that they are less vulnerable to Dan’s first criticism. I am optimistic that that criticism can be met, but I don’t think it is met automatically or that it is not a valid concern.

    What I meant to be saying is that studies that manipulate some feature of a verbal probe in order to measure differences are less vulnerable to Dan’s second criticism because they are less likely to involve asking people what they think in order to slavishly accept the majority opinion. And the results of such studies are more likely to have some philosophical interest. (Maybe that last claim is wrong, but I think it carries with it a bunch of issues about the relationship between philosophy and empirical research generally.)

    So, am I missing something? Am I being unfair?

  14. Hey all,

    I agree with Dan and Brian (and Simon Cullen) that it would be bad to straightforwardly draw philosophical conclusions from people’s responses. But this is an issue with interpretation, not surveying as such. Moreover, I am not convinced all x-phi, or the majority, especially the recent works, do something like this. Hence, it is better to have specific criticisms of interpretation issues.

    I agree with Dan and Brian (and Simon Cullen) that it is often difficult to make sure the manipulations a study employs really do vary the state the people are in, and the controls do minimize other unrelated variations. But this is an issue with design, not surveying as such. Moreover, I believe that the majority, if not all, x-phi try their best to use appropriate manipulations and controls. Of course, these can always be improved upon, and the people doing the studies seem receptive to suggestions. I am sure specific suggestions would help projects move forward.

    Am I just making too much of a merely terminological fuss? I don’t think so. Terminology brings associations. X-phi has plenty of kinks to work out, but it seems surveying in the relevant sense here is the least of their problems. However, if we continue the anti-survey refrain instead of pointing out specific problems with interpretations and designs, then we are prone to the analogy that Brian seems to make with the Gallup poll.

    The anti-survey refrain associates x-phi with doing something like asking people “do you think IRI is true?” and then say IRI is false when only 24% of the participants say yes. That, obviously, is nothing like the studies that have been run. But it’s easy to think something like that, and make analogies to Gallup polls, because there is another sense of surveying that involves no experimental manipulation. I am all for specific criticisms — in fact, I think the responses to the cross-cultural name stuff tend to be quite persuasive — but the anti-survey refrain points us away from, rather than toward, the kind of interesting and helpful criticisms about interpretation and design.

  15. Brian,

    In your last paragraph you suggest that the emerging x-phi consensus conflicts with well established psychological results, and conclude: “I’d want a lot more evidence than surveys though before I overturned a well established result in experimental psychology”. I agree with you that the x-phi stuff would be very questionable if it did conflict with well established psychological results. But I don’t think there is any real conflict:

    1. The psychological results seem to show that subjects who perceive themselves to be in high stakes situations are less likely to leap to belief. (They will tend to search for more information, think more systematically, be less susceptible to certain cognitive biases, and manifest a lower need for closure, inter alia. As you nicely put it, they will need more to settle the question to their satisfaction.)

    2. The x-phi survey data gathered to date suggest that subjects evaluating the knowledge of third parties tend not to treat stakes as making a difference. Crucially these studies all attempt to hold fixed the belief states and the evidence of the third parties being evaluated, in order to test whether stakes make any /additional/ difference.

    So there is no incompatibility. Someone who perceives herself to be in a high stakes situation may be less likely to leap to belief, but given that she does (and holding fixed her evidence etc.) the stakes do not seem to matter as far as our willingness to say she knows.

    (Of course this is not to defend the methodology of x-phi, nor to defend the extant studies about knowledge ascriptions against possible confounds, nor to rule out the prospect of new x-phi results possibly reversing the emerging consensus against stakes sensitivity. This is only to make the very limited point that the emerging consensus against stakes sensitivity in knowledge ascriptions is not in any conflict with the well established psychological results about high stakes subjects. Whatever else you might want to hold against x-phi, don’t hold that against x-phi!)

  16. Just to follow up on Jonathan Schaffer’s point, it is true that most of the x-phi surveys on this topic do control for belief, so there might not be a straight forward conflict with the psychology literature on this point. But there is a related worry. Those surveys might not have properly controlled for the perceived evidence across the vignettes (they are supposed to be the same). As Jennifer Nagel points out (comment 6 on the July 14 post in Certain Doubts Blog—sorry don’t know how to do a link), In the high stakes case, subjects might expect the protagonist in the vignette to have collected higher quality evidence (because the stakes are high). After all, in those experiments the protagonist in the high stakes vignettes do report, in effect, that they are satisfied with their evidence since they say: “I know that the bank will open tomorrow”. For those interested, I discuss this possible confound in a paper “Some recent work in Experimental Epistemology” available on my website.

  17. I’d like to second Jonathan’s reply to the claim that the x-phi work on stakes is in conflict with psychological findings. I made the same point two years ago in a discussion of a paper by Ram Neta and myself at Certain Doubts (discussion from August 1, 2008, comment 67). It is also made in the current descendent of that paper (which can be found here: And it is continually made in discussions of the X-phi work on stakes, where this sort of objection continues to be a common one.

    Also, in the paper I just referred to, we tried to control for the kind of objection Angel raises, about expectations for varying quality of evidence across vignettes. We did this by including the following sentence in all our vignettes: “Kate could gather further evidence that she is on Main Street (she could, for instance, find a map), but she doesn‘t do so, since, on the basis of what the passerby tells her, she already thinks that she is on Main Street.” (Of course, as Angel points out in his work, this paper specifically addresses stakes sensitivity of evidence, not knowledge.)

    Finally, it’s sometimes claimed that the X-phi work on stakes suggests that those epistemologists who argue from intuitions about cases to stakes sensitivity of epistemic concepts or features are making up their intuitions and somehow fooling others into believing them, without offering any explanation of how these epistemologists could do this so successfully. But the main point of my paper above is that it’s natural for people to have this intuition when high and low stakes are juxtaposed in philosophically common contrast cases. But, like demand characteristics in psychology, it’s an artifact of the thought-experimental design (of juxtaposing cases), and not clearly something indicative of how we think of evidence. In fact, the tendency to judge cases in the relevant way disappears when we consider the cases independently, whereas other clearly epistemically relevant features continue to affect our assessments when we consider cases independently. (This explanation is, obviously, applicable to the relevant work on knowledge.)

  18. Jonathan – this is Jason Stanley. Actually, the Kruglanski study I discussed on Certain Doubts is about confidence levels. It seems that people adjust their confidence levels according to stakes. I’ll have to check again, but it looks awfully like people are adjusting their subjective probabilities in response to stakes (“fear of invalidity” was the particular factor used). Assuming people are adjusting their subjective probabilities in accord with what they think is the evidence, this suggests that people take evidence to be stakes sensitive, which would conflict with the X-phi stuff. The psychology suggests a complexity that isn’t really being registered in the X-phi discussions. I am planning to set aside a few months (years?) to make my way through it (being only up to 1987!). But I really wish somebody else would do it (and preferably not someone with an established commitment to a certain outcome, and someone better versed in psychology than I am).

  19. Jonathan,

    Also, Angel Pinillos has used X-phi techniques to get the result that people do have stakes-sensitive intuitions about knowledge. So there is no uniformity in the data (also, May and Zimmerman found at least effects of stakes of third-person knowledge ascriptions). I realize that you guys did Pinillos’s experiment, with “believe” instead of “know”, and also found effects of stakes. That is to be predicted on any view (like mine) that says that people adhere to the norm, “Believe that p only if you know that p”, and assume that others adhere to that norm as well (I think Angel made this point too on the X-phi blog). So even using the distinctive methods of X-phi, there is no uniformity in the results.

  20. Jason,

    1. On your first post: I agree that “people are adjusting their subjective probabilities in response to stakes”. More precisely, I’d say that all else being equal, people who perceive themselves to be in high stakes situations are less likely to leap to belief, and less likely to have high credence (perhaps due to a lowered need-for-closure, ala Kruglanski).

    But the point I was making is that this is completely consistent with the emerging consensus in x-phi that stakes play no further role /once belief and evidence are fixed/.

    Anyway I certainly agree that these are complex and subtle matters, and that the x-phi material ought ideally to be done in full appreciations of all the subtleties. In particular (as Angel alluded to in comment 16), there is the possible confound that the participants in the x-phi studies did not really treat belief and evidence as fixed across the low and high stakes scenarios. But this is a point where social psychology can usefully be brought to bear in looking for likely confounds (which can then be tested), not a point where x-phi is in any conflict with psychology!

    2. On your second post: This isn’t the place to argue over the Pinillos experiments, or how to interpret the results of May et al (which found that people tended to attribute knowledge in both high and low stakes cases, but with a slightly greater tendency in the low stake cases). I am actually officially neutral on stakes sensitivity. I’ll eat whatever the data serves.

    All I meant to be defending here was the legitimacy of using x-phi methodology, against Brian’s claim that the methodology was called into doubt due to its allegedly generating results in conflict with established results in experimental psychology.

  21. Jonathan,

    You write:

    “On your first post: I agree that “people are adjusting their subjective probabilities in response to stakes”. More precisely, I’d say that all else being equal, people who perceive themselves to be in high stakes situations are less likely to leap to belief, and less likely to have high credence.”

    I wasn’t thinking of that as more precisely. I was thinking of that as different. I don’t have Kruglanski 1987 in front of me now, but my recollection is that people in higher stakes situations thought that something was less likely than they thought it would be in lower stakes situations. It’s a consequence of this that would be less likely to form a full belief in high stakes situation. But I’m not seeing how “reluctant to form a full belief” would less one to lessen one’s subjective credence.

    One thought is that stakes mediates between partial belief and full belief. In a high stakes situation, you might need .97 credence for full belief, and in a low stakes situation, .93 would do. That’s what you’re suggesting. But that wouldn’t explain why people shift their credences in accord with stakes.

  22. Also, several small points. First, I don’t think that there is an “emerging consensus”, unless we rule Angel out as not legitimate because his results differ. Secondly, I think we all should not say things like “I am actually officially neutral on stakes sensitivity. I’ll eat whatever the data serves.” It so happens that the X-phi work you have done lines up perfectly with the results of your theoretical position in epistemology (and your previous armchair claims). In that sense, you are clearly not “officially neutral”. Neither am I. I am pretty convinced that there is a real phenomenon that Cohen, DeRose, Fantl, McGrath, and others have been described – I use it with my students in intro epistemology classes (“look at the guy outside – does he know he has hands? Class: yes! Me: What if he was offered a bet for a dollar if is right, and his whole family gets killed if he is wrong – does he still know he has hands? Class: no!”). So I am antecedently dubious of experiments that don’t reveal these effects. And I have no problem being upfront about that. On the other hand, if we’re all supposed to pretend that we don’t have theoretical stakes in the matter, then I can pretend that too.

  23. Jonathan,

    I’m in full agreement on this:

    “There is the possible confound that the participants in the x-phi studies did not really treat belief and evidence as fixed across the low and high stakes scenarios. But this is a point where social psychology can usefully be brought to bear in looking for likely confounds (which can then be tested), not a point where x-phi is in any conflict with psychology!”

    Yes, this is a possible confound, a very serious one. It is I think hard to correct for. Another possible confound with surveys is the effects of focal stress, especially if the surveys are written. Whether focal stress affects the truth-conditions of knowledge-ascriptions (as you and Zoltan maintain) or just pragmatic implications, I wonder how people are thinking of the intended contrast classes, etc. That isn’t just a worry with X-phi for knowledge-ascriptions, but all sorts of written survey work.

  24. Jonathan,

    Briefly – I worry that if we think that whether one ought to form a full belief is a stakes-dependent matter, then that will make it even trickier to correct for the confound that people are not keeping evidence fixed across high and low stakes cases. If I am told that someone in a high-stakes situation believes that p, and I assume that warrentedly forming the full belief that p requires you to have more evidence in a high-stakes situation, then I’m going to assume that that person has more evidence.

  25. Jason,

    We seem now to be talking past each other. (This is not helped by your attributing to me views I don’t hold concerning credences and belief!) I am granting that stakes can impact credences, which is what Kruglanski shows. What I am trying to explain is that we still need to distinguish at least two different models about how stakes might impact knowledge ascriptions:

    1. Traditional: The stakes of the subject have no direct impact on our willingness to attribute knowledge to her. At most they may have an indirect impact on our willingness to attribute knowledge to her, by impacting her credences, her evidence, or one of the traditional factors that most everyone accepts as relevant.

    2. Stakes Sensitivity: The stakes of the subject have a direct impact on our willingness to attribute knowledge to her, which is not mediated by any of the traditional factors (credences, evidence, etc.)

    The Kruglanski stuff only shows that stakes impact credences, and is thus completely neutral between models 1 and 2. Yes? The x-phi stuff attempts to hold fixed the traditional factors, and look at minimal pairs that differ only in the stakes. It thus attempts to test 1-vs.-2 by looking at whether any impact of stakes will be screened off by fixing the traditional factors.

    All I have been trying to point out here is that there is no conflict between the claims that (i) stakes impact credences, and (ii) stakes do not impact knowledge ascriptions once credences are held fixed. Are you claiming otherwise??

  26. Jason,

    My #25 was only a response to your #21. When I posted it I then saw that you’d made three more posts in the interim (!!).

    On #22: I certainly don’t mean to marginalize Angel. He might well prove to be vindicated. But surely one may speak of an emerging consensus in a field even if there are one or two holdouts (and even if the holdouts are very smart and have very interesting data that needs further discussion).

    Also I really am officially neutral on stakes sensitivity! For the record this is what Josh Knobe and I say in our forthcoming: “[T]he contrastive model does not directly predict any stakes effect on its own. The contrastive model could be used to predict a stakes effect, if coupled with the further speculative claim that high stakes tends to trigger consideration of a wider range of scenarios (Schaffer 2006). But given the empirical evidence to date we see
    no need to pursue this speculation further.” I am committed to contrast sensitivity and have tried to show empirical support for contrast sensitivity. I make no claims to neutrality on the matter! But stakes sensitivity I could take or leave. Were it to be found, I would then (non-neutrally!) be committed to fitting it within a contrastive model.

    On #23: I think we are in perfect agreement as to the possibility of these confounds, and how difficult it can be to correct for them. I would only add that this is not a reason to give up on x-phi, but only to pursue it in as careful and informed a way as possible.

    On #24: This is indeed one possible confound, that participants in the studies will take the subject’s evidence to differ. Again I think we agree on the matter, and I would only add that this is further reason to pursue x-phi in a careful and informed way.

  27. Jonathan,

    Hmm. Your post makes me worry that there has been a lot of talking past each other. If the situation is what you describe, even taken at face-value, the X-phi work doesn’t impact any stand anyone has taken in the literature, right?

    My view is that all epistemic notions, including evidence, are stakes-sensitive. Or at least I favor the view according to which evidence and knowledge are stakes-sensitive, and dislike the view that only knowledge, and none of the other attendant notions, are stakes-sensitive – as I say in the conclusion of my book (and make more clear in the PPR symposium). I guess I’ve always suspected that stakes-sensitivity comes primarily from the notion of evidence, but also worried about knowledge-first views.

    That’s why Neta and Phelan’s paper was targeting my work – they were trying to argue that warranted credences weren’t stakes-sensitive.

    Secondly, if evidence is stakes-sensitive, one can give a nice explanation of the social psychology (I make this point at length in some blog post somewhere arguing with the X-phi folk). We lower our credences in high-stakes situations, because we try to align our credences with our evidence. On my view, some bit of information might be evidence in a low stakes situation, but not count as evidence in a high stakes situation. That’s why I take the social psychology literature to provide evidence for the view I have defended.

    I reject the view that it’s knowledge alone that’s stakes-sensitive. I say I don’t like it my book, and later in this PPR discussion I openly reject it.

    In fact, come to think of it, nobody who advocates IRI has argued that it’s only knowledge that is stakes-sensitive. In Chapter 4 of Fantl and McGrath, they argue that justification is stakes-sensitive. I am attracted to the view that all epistemic notions are stakes-sensitive, and one dispute between Fantl and McGrath and me is whether this is true – they think that epistemic probability isn’t stakes-sensitive. I’m not sure exactly what Hawthorne thinks on the matter, but he is at least occasionally sympathetic to E=K.

    So it should at least be made clear that there is no proponent of the view that only knowledge is stakes-sensitive.

  28. For what it’s worth, I’m arguing in a paper I’m writing now that the view that Weatherson takes in his “Can We Do Without Pragmatic Encroachment?” is in the end a version of pragmatic encroachment. It’s pragmatic encroachment on ought to believe. I’m totally sympathetic with the view that what is really pragmatically encroached upon are the norms for belief, and knowledge only derivatively. Similarly, Fantl and McGrath are sympathetic to the view that what is really pragmatically encroached upon is justification, and knowledge only derivatively.

  29. What a terrific discussion! I feel late to the party.

    1. On the issue of whether survey methods have any special infirmities, Sam, Josh, and Jonathan L are right on the money. With any task in a psychology experiment, one can raise concerns about how the subjects interpreted the instructions, whether there are performance errors, and so on. And there are plenty of ways of accommodating such concerns, as an ordinary part of experimental design. Whether or not x-phi studies have always and uniformly done so is beyond question: clearly a number of them have not. (The Weinberg, Nichols, and Stich SES study is a noteworthy example of design FAIL in this regard.) But there are plenty of x-phi studies that have done pretty well in this regard, too. There’s just no special difficulties here, just a good reason to, as Jonathan S put it in the last comment (at the time of my writing this one), “pursue x-phi in a careful and informed way”.

    2. Concerning the extant x-phi studies that have not found stakes effects (and putting aside the question of whatever degree of consensus there is or isn’t — it’s really very early days on all that! and Angel’s work is totally awesome, no one is disputing that!), there’s not even a tension with, let alone a putative “overturning” of, the body of social psychology results that Jennifer has made such brilliant use of. In addition to the issues discussed so far about controlling for beliefs and credences, there’s another, clearer source of a gap between the psych work and the recent x-phi work: namely, that the Kruglanski et al work is all first-personal, and the x-phi work is third-personal. It sure seems to be a fact of psychology that raising stakes decreases agent’s degree of belief; but unless the x-phi subjects know that fact and put it to work, there’s simply no reason to expect them to adjust their attributions to other agents accordingly. So the two bodies of results are fully consistent with each other. (I would note that Jennifer is very clear, in her paper, that the psychological work in question does not actually on its own make any predictions about these sorts of attributions; see p. 284.) Now, if it turned out on further investigation that subjects are aware of this fact, and do put it to work in their attributions, then there would be a tension between the existing psych work and the x-phi work. That’s a strong psychological claim, but clearly one that evidence could possibly be found in favor of. But at this time, there just isn’t.

    3. “Assuming people are adjusting their subjective probabilities in accord with what they think is the evidence, this suggests that people take evidence to be stakes sensitive, which would conflict with the X-phi stuff.” This is a great (if likely unintended) illustration of the importance of x-phi. Because why should we go around assuming strong psychological claims like that, when we can figure out how to test them? (Note that Jason’s assumption here, to do the work he wants, can’t just be the innocuous one that changes in evidence yield changes in subjective probabilities; he needs the other direction, so that it’s not some other sort of non-epistemic factor that explains the shifts in reported confidence judgments.) There is a little bit of relevant psychology work already out there (especially the work on miscalibration in prediction vs. miscalibration in perception, which I think tells slightly against Jason’s assumption), but I don’t think that what there is is sufficient to settle the questions that philosophers want to ask. Basically, we care about some very fine distinctions here that just haven’t been relevant to scientific psychological inquiry, so the psychologists (very reasonably) haven’t been asking the questions that we philosophers might want asked. So, if we wanted to take Jason’s hypothesis seriously, we will likely need to be involved with their asking. Now, it doesn’t matter to me whether such future work evaluating such an assumption be done by philosophers themselves, or psychologists working closely with philosophers; it’s an x-phi investigation either way, by my lights.

    4. Some raw speculation here: one thing that I think creates some friction in philosophers’ attempts to evaluate x-phi, is that there are very different norms in psychology and philosophy as to just what one should claim to have accomplished in a paper, in that philosophy papers are often intended to be more self-standing than psychology papers are. Not that philosophy papers aren’t often (usually, even) configured to operate within an existing dialectic, and certainly very few papers are capable of being understood without seeing them in the context of a broader set of texts. But still, I think we have a norm that the argumentative work of a given paper should be fairly complete. One manifestation of this is that any really big premises we may want to use, we frequently just conditionalize our arguments on. The paper is saying, “If you give me A, B, and C, then I’m going to show you how to get to D”, and a good paper should make a very strong case, on its own, for something like such a conditional.

    But no one study in psychology can do that. You just can’t control for all the things that you would want to control for, and there’s I think ab it more of an expectation as well that some rival researchers are going to come up with some good ideas for confounds, and so you’ll always have to go back to the lab at least a few times. Scientific results are established over a series of studies, any one of which might try to control for some subset of the plausible worries.

    So that the MMNS study is consistent with other interpretations is, in an important way, just unproblematic. What would be problematic is if they claims to have shown anything decisively in that paper (but they don’t; see p. B8), or if they refused to go back and try to do more studies in the face of proposed confounds (but they have exactly done such follow-ups, e.g., their exchange with Marti concerning linguistic vs. metalinguistic intuitions).

    So any one study is going to look incomplete. And that’s how it should look — any one study is going to be incomplete! What I think pretty much all x-phi practitioners are clear on is that results have to emerge out of a large, ongoing body of work, that is only nascent for everything in x-phi (except maybe the Knobe effect literature). And x-phi papers are pretty uniformly written with caveats in various places, expressing an appropraite humility along such lines. But it may be that, while this is fairly well-understood within the x-phi community, philosophers who are not so used to actually operating under these norms may be expecting more from any individual study than it is appropriate to expect. So the work may sometimes seem to be weaker than it actually is.

    (Which is not to say, again, that there aren’t some studies out there which fail to rise to the level of even being an appropriate initiator of a chain of further studies. There are. But the badness of some x-phi papers does not tell against the whole movement, any more than the badness of some armchair papers (surely we agree there are an unfortunately large number of those, too), do not tell against the armchair.)

  30. Jason,

    Just a small point of clarification as to what is held fixed in the x-phi studies on stakes sensitivity. In #25 I described it as “the traditional factors”, including “credences” and “evidence”. In #27 you worried that the x-phi studies could not possibly challenge your view if they attempt to hold fixed the evidence while varying the stakes, since by your lights the evidence itself is stakes sensitive. This is a confusion (no doubt my fault for using the contested term “evidence”). What the studies held fixed is evidence as traditionally understood. Given the likelihood that we will not find any neutral vocabulary, what the studies held fixed is perhaps best clarified by example:

    -In bank cases, what is held fixed is that the subject recalls that the bank was open on a Saturday two weeks ago
    -In airport cases, what is held fixed is that the subject has a three-week old printout of the flight intinerary, which says that there is a layover in Chicago
    -In detective cases (the cases Josh K and I use to test for contrast sensitivity), what is held fixed is that Mary the detective finds Peter’s fingerprints on the safe

    The conclusion that several x-phi studies have drawn (incl. Feltz and Zarpentine, who tested the cases from your book verbatim) is that when we hold fixed the subject’s credences and the sort of factors just illustrated, stakes have no remaining impact on knowledge ascriptions.

    To put the point somewhat crudely, I don’t think that the x-phi-ers are talking past you when they test your cases verbatim.

  31. Jonathan (Schaffer),

    First, on the bank cases – they aren’t my cases, they are due to Cohen and DeRose. I don’t care about the particular cases. I care about whether knowledge is sensitive to stakes. It seems bizarre to me to test something like this using just predicates involving banks. Secondly, the argument in my book is not meant to be based mainly on intuition, but is instead meant to rely quite crucially on a bi-conditional linking knowledge and action (I’ve been neglecting to emphasize this in these exchanges). I didn’t do an investigation of intuitions because I thought the most controversial aspect of my project was defending the knowledge-action links (which I did go ahead to defend, in a paper with Hawthorne). After one defends the knowledge-action links, the only intuitions one need to rely on are ones linking action and stakes.

    I do agree that I gave a bit of a misimpression by claiming that it relied on intuitions about specific cases, and indeed specific cases involving banks. The purpose was not to emphasize points about banks, but rather to describe the logical structure (high-stakes vs. low stakes, ignorant high stakes, low-attributor, high-stakes etc.). The occurrence of “bank” in the examples was supposed to be something of a schematic letter. I’m totally fine with the view that predicates besides those involving banks must be used to elicit the intuitions. And I suspect Brian is right that more extreme cases are needed to draw out the intuitions.

    More importantly – you have given an illustration of possible confounds. Suppose that the subject is told that the high-stakes and the low-stakes person both recall that the bank was open two weeks ago. “Recall” is factive. To be told that they recall that the bank was open two weeks ago is to be told that they both know that the bank was open two weeks ago. But on my view, it’s harder for high-stakes to know that the bank was open two weeks ago than for low-stakes to know this. The more such information one provides, the more one suggests that high-stakes has grounded her beliefs via more secure methods of gathering evidence than low-stakes. That’s a confound we need to pay attention to.

    I do suspect that we need to go to more extreme cases than the bank cases to elicit the intuitions. Josh May and Aaron Zimmerman find effects of stakes. But the effects of stakes are not enough to cause variations in judgments in these ordinary cases. Going to non-ordinary cases may cause variations in judgments. Or at least it does in my undergrads.

  32. I agree with Jonathan W’s point that there’s a, “clearer source of a gap between the psych work and the recent x-phi work: namely, that the Kruglanski et al work is all first-personal, and the x-phi work is third-personal.” (Indeed, the Certain Doubts comment I referred to earlier makes this point specifically.) It might be possible to argue from the first-person psych work to a tension with the third-person x-phi work, or to empirically demonstrate a psychological principle that reveals such a tension. But I think it’s inaccurate to claim that this would defeat the most important conclusions of the earlier x-phi work. Most of that work is explicitly challenging the argument from third-person contrast cases that epistemologists have relied on, as the following quotes demonstrate:

    May et al.

    “What follows is a report of our attempts to experimentally test Stanley’s and Schaffer’s claims about our common-sense intuitions about such cases.”

    Feltz and Zarpentine:

    “In this paper, we apply this methodology to anti-intellectualist claims about ordinary
    knowledge attribution. Our results indicate that some of the empirical claims made by anti-intellectualists are not empirically supported.”

    Neta and Phelan:

    “So the question that we set out to answer was the following: Do our intuitive judgments about cases support the hypothesis that there is a relation between one‘s evidence and the practical costs of being wrong?”

    The claim made by the experimentalists has the following structure: Some epistemologists argue from X and if X then Y to Y; but, in fact, it’s not the case that X; so you don’t get Y from if X then Y. You can’t challenge this kind of argument by saying I’ve got another argument for Y! (Of course, this is all assuming that there aren’t devastating confounds for the X-phi experiments to show it’s not the case that it’s not the case that X, and that Angel’s work doesn’t defeat the previous X-phi case, and so on.)

  33. Mark,

    Those of us in the stakes-sensitivity literature (and no doubt philosophy generally) have done a poor job of separating out what depends on “appeals to intuition” from what depends on theoretical premises. I tried to emphasize in my book how much depends on the Knowledge-Action links. In some sense, this should be clear from the fact that I am upfront that what I regard as certain clear intuitions, such as High attributor-low subject cases, are not captured in my framework, and I regard that as ok, because these are merely intuitions, and not supported by theoretical principles (the knowledge-action links). But I admit that my vocabulary when I’m introducing the bank cases does strongly suggest that I am relying on brute intuitions about bank hours. I regret this. My aim in laying out the cases as I did was rather to show all the different logical possibilities, and use them to draw the relevant distinctions. For example, I use the cases to distinguish what predictions contextualism makes versus what predictions IRI makes. Most crucially, I wanted to draw out the fact that high-attributor/low-subject stakes has a different basis than the rest. The latter does not depend on Knowledge-Action links in the way the rest of the cases do.

    One salutary effect of X-phi is that it has led all of us to be much clearer about when we are relying on theoretical principles and when we are not. Just speaking of myself, I have come to realize that far less of philosophy is based on intuitions than seems to be from the vocabulary we use. And I learned this from thinking through the project of X-phi.

    That said, I still think it’s true that we have stakes-sensitive intuitions. As I said, I use them in my intro epistemology classes (and I bet you do too). Admittedly, I don’t use cases involving banks (and I suspect none of my undergraduates knows what a check is anyway). I use more extreme cases.

    As far as the first-personal/third-personal stuff. There might be something out there in the social psychology literature that is third-personal. Nobody here seems conversant with that literature (I’ve read fewer than half a dozen papers, all from 1987 and before). But even the first-personal stuff requires explanation. Why do people shift their credences when confronted with heightened stakes? At the very least, I can explain why they do.

  34. Just to give the textual evidence supporting Jason’s claim that the intuitions in the cases were never thought to be the deciding factor for anti-intellectualists:

    From Stanley 2005: “The role of intuitions is not akin to the role of observational data for a scientific theory. The intuitions are instead intended to reveal the powerful intuitive sway of the thesis that knowledge is the basis for action… But the VALUE of knowledge is explicable in part by its links to action…” (12)

    From Fantl and McGrath 2003: “We reject evidentialism, but we feel that the above argument is not enough to do the job. The intuitions in the Train Cases, though ultimately correct, are not strong enough to count as data in a decisive argument against evidentialism. Evidentialism is not so easily refuted.” (69)


    “In light of these sorts of evidentialist responses, one cannot refute evidentialism by a simple appeal to the Train Cases. But this is not our plan. Rather, we aim to provide a theoretical basis for rejecting evidentialism by defending a ‘pragmatic’ necessary condition on epistemic justification.” (70)

    So, on Mark’s point, we’re not all coughing up some new argument to flee from the old and refuted case-based argument. The old and “refuted” case-based argument was never the operant one.

    (And I still have yet to see an X-Phi stakes-shifting case in which it is just obvious that the High Stakes subject is irrational to act on the relevant p. To the extent I share the experimental subjects’ tendency to ascribe knowledge to High in those case, I also have a rather strong tendency to think, “Oh just wait until tomorrow. You’ll be fine!” But if there are a number of experimental subjects who have that intuition as well, then their knowledge-ascribing intuitions are irrelevant to anti-intellectualism. How many experimental subjects have the following conjunction of intuitions: “High knows the bank is open tomorrow and it’s nuts for High to wait until tomorrow.” Has this been tested for?)

  35. I didn’t say that you are, “all coughing up some new argument to flee from the old and refuted case-based argument.” I said you can’t criticize a challenge to one kind of argument by pointing out that there are other arguments for the position (even if those arguments were already in circulation). I take it that the case-based arguments were supposed to bear at least part of the load for at least some of the epistemologists (though they clearly weren’t decisive). Why else would so many people have included them in so many papers? It’s a common enough move in philosophy to challenge one argument for a position, even if that challenge (even if successful) doesn’t defeat the position. And, for the most part, this is what the experimental papers on this topic have set out to do. In any case, I like Jason’s concessive point that thinking through the X-phi literature can lead us to be clearer about when we’re relying on intuitions, theoretical principles, etc. If that’s all this work has accomplished, I still think it would be worthwhile.

  36. Ditto to what Mark just said. To just expand on his points a little, it’s pretty clear in at least some authors in this literature (such as Keith) that the intuitions are carrying at least a substantial portion of the evidential load, and while granting Jason’s line that Jeremy quotes, there are also key places in the book where the argument turns at least somewhat on which accounts can do a better or worse job of accommodating which intuitions. (That correct theory here need in no way, as Jason rightly points out, cleave neatly & tightly to the intuitions, does not mean that the presence or absence of the intuitions is evidentially epiphenomenal.)

    I think it may likely be a mistake in these debates to try to speak of anything as the deciding factor. The mode of argumentation is frequently abductive, and in such arguments it may rarely be the case that any one piece of the explananda is crucial on its own, while still being the case that, if enough of the right individual data points get rejected, what will count as the best explanation will thereby shift. So evaluating any piece of the evidence for anti-intellectualism is relevant, even if the evaluation of no individual piece can ever be decisive.

    I think Jeremy has a lovely idea for a further experiment, and I hope that he’ll get working on designing the instrument for it soon! I do think that his comment contains a common misconception here, though, which is that the x-phi people in question take their negative findings to speak directly against anti-intellectualism in some way. I believe that they are offered more in an undermining-a-proposed-argument-for-p way, than an argument-for-not-p way.

  37. I can’t speak for the use of the cases in arguments for contextualism. But whatever use DeRose puts them to is independent of their function in arguments for anti-intellectualism.

    I understand that the X-Phi work on anti-intellectualism should be taken in an undermining-a-proposed-argument-for-p way. But the proposed argument that it takes itself as undermining is explicitly and repeatedly rejected by Fantl and McGrath and seriously hedged by Stanley. It’s not exactly a straw man that the X-Phiers are attacking. But it’s close. And the focus among X-Phiers on that argument without really saying what the other arguments may be gives the impression that that’s really the best anti-intellectualists can do: “maybe, in response to those arguments, the anti-intellectualists can seek to develop others. But we’ll wait and see what those look like.”

    Feltz and Zarpentine do consider the possibility of other arguments (and cite Stanley and Hawthorne 2008) in a footnote. But even they treat these additional arguments as options the anti-intellectualist might turn to if the reliance on intuitions in the stakes-shifting cases runs dry: “In the face of these results, the anti-intellectualist might try to motivate the connection in some other way. If the role of intuitions is to support the thesis that ‘knowledge is the basis for action,’ other arguments may be given to that effect.”

    It’s not just that there are “other arguments” for the position. Nor that the anti-intellectualist is forced to turn to those other arguments in the face of the experimental results. Those other arguments were always the central arguments. The intuitions, at most, were only used to get people in the head of thinking about what the position involved and that it wasn’t initially totally out of line with ordinary thinking. Some of the most prominent advocates for anti-intellectualism were crystal clear about this.

    When X-Phiers cite proponents of anti-intellectualism, they cite three authors: Hawthorne, Stanley, and Fantl and McGrath. The only one of those three who really relies on the cases in the way the X-Phiers discuss is Stanley and he denies that the the cases are data, while Fantl and McGrath outright reject the argument. Is this not worth stating in the X-Phi papers? Would it not moderate the impact that the X-Phi results are being taken to have if it was widely understood that the argument they’re attacking was presented by a single anti-intellectualist, and then only in a hedged way? The X-Phi results are being taken as really important developments in the debate over anti-intellectualism. But the results are only as important as the argument they’re focusing on.

  38. Jeremy, looking closely at what I take to be the most recent version of the Feltz & Zarpentine

    I’d say that they are pretty clear and responsible on just those issues. (See in particular section 7, where they discuss Jason’s uses of the cases. As I noted earlier that he doesn’t use them as data doesn’t mean that he doesn’t use them as evidence in some other way.) I would also add that I don’t think it makes sense to stipulate Keith out of the discussion, since the particulars of some of the cases has played an important role in some of the arguments between contextualism and its rivals in this corner of the literature.

    You ask, “Would it not moderate the impact that the X-Phi results are being taken to have…?”, but, honestly, I don’t see that anyone is taking those results to have much more of an impact than the claim that it is dangerous, at this time, to use ordinary-epistemic-practice-based arguments for IRI. Who do you take to be claiming a bigger impact for it than that? Those arguments appear in prominent enough locations, that this certainly seems like an important enough result to be worth their (and others) having done it.

    One thing that is maybe causing you some consternation is that maybe it seems to you that the x-phi epistemologists are ignoring the Fantl & McGrath-style arguments. But I think that those arguments are just not as clearly amenable to being addressed by x-phi methods as the argument-from-instances-of-our-epistemic-practices is. I haven’t heard anyone anywhere suggest that any of the x-phi results so far do anything to undercut arguments like y’all’s. There’s no reason for y’all not to take these results, indeed, as a vindication of your decision not to go the ordinary-practices route in making your case!

  39. The Feltz and Zarpentine discussion in that version (which is the version I was using) is certainly better than some of the other treatments (though it still overemphasizes, it seems to me, the importance of the case-intuitions). I’m more concerned with the body of the experimental work on anti-intellectualism. Suppose that a number of papers all cite a number of proponents of view X and that all those papers then isolate and criticize a single argument they claim to be supportive of view X without explicitly mentioning that the argument they criticize is limited to only one of the mentioned proponents and without (in most cases) really emphasizing that even that proponent hedges the argument a great deal. Suppose further that all these papers mention that, “in the face of the results”, the proponents of view X can try to motivate X in “some other way” but (in most cases) don’t mention that the original works all presented another way and extolled its virtues over the way criticized. Surely there’s a broad implication here that the proponents of view X don’t really have anything better up their sleeve already.

    Maybe you’re right — maybe people (that is, blog-readers and paper-readers who aren’t themselves contributors to the debate) aren’t taking the results to be more far reaching than I think they should be taken. But just in case, I do think it’s worth pointing out on blogs where the work is discussed just how limited the role of the stakes-shifting cases is in published arguments for anti-intellectualism. You know, just so people don’t get the wrong idea.

    As far as vindication goes, I’ll try to take your advice.

  40. Jeremy, yes, I certainly agree that it’s worth making clear in these fora that there are other important arguments for anti-intellectualism! I think the differences between these arguments I just wanted to make clear myself that the existence of those other arguments doesn’t make for any sort of failure or flaw in the extant x-phi work on the topic, that is aimed at the ordinary-epistemic-practices arguments. And I would also add that, if anyone were to try to infer from these sorts of results to the falsity of anti-intellectualism, I’d agree with you that they would thereby me making a philosophical blunder.

  41. Jeremy,

    On the issue of x-phiers mentioning alternative arguments for anti-intellectualism: I think we were forthright just as Feltz and Zarpentine were. In fact, we provide the quote from Stanley about the intuitions being defeasible and not being data (as you do in comment #34). We do go on to discount this a bit and assume he relies on them to build an argument. But, as Jason mentions, he certainly did come off as having such cases play a fairly substantial evidential role. Moreover, as Jonathan W. points out, even if the role is minor, it’s still a role. And as I mentioned earlier, I’m sure we and readers would find such experiments interesting even Stanley himself doesn’t put as much weight on them. Suppose we never did the experiments and Jason disavowed such cases entirely after the book came out. I’d still think it interesting to test the cases.

    Moreover, in the final section of the paper, after reviewing our results, we write:

    “Thus, some doubt is cast on the cogency of attempts to support such a version of anti-intellectualism by appealing to its capturing common-sense or ordinary linguistic practice with respect to these bank cases. But, of course, our results don’t address other arguments for Stanley’s view, such as appeals to principles connecting knowledge and action.” (p. 272)

    Perhaps we could have drew more attention to the other arguments. And of course it’s fine to just point out in a blog thread the limitations of certain results when those results are under discussion (as you say in comment #39). But Brian’s original post above, and much of the discussion here, is about whether x-phi, specifically the survey methodology, is at all worthwhile in philosophy. So when you point out limitations, it can look like you’re suggesting these limitations bolster the case for the methodology being worthless.

Leave a Reply