I’ve been interested recently in defending a particular norm relating knowledge and decision problems. To set out the norm, it will be useful to have some terminology.

- A
**decision problem**is a triple (S, A, U) consisting of a set of states, a set of actions, and a utility function that maps state-action pairs to utilities. - An agent
**faces**a decision problem (S, A, U) if she knows that her utility function agrees with U about how much she values each state-action pair, she knows she is able to perform each of the actions in A, and she knows that exactly one of the states in S obtains. - A decision problem (S’, A, U’) is an
**expansion**of a problem (S, A, U) for agent x iff S’ is a superset of x, U’ agrees with U on every state-action pair where the state is in S, and the agent knows that none of the states in S’ but not S obtain.

Then I have endorsed the following principle:

Ignore Known Falsehoods. If (S’, A, U’) is an expansion for x of (S, A, U), then the rational evaluability of performing any action φ is the same whether φ is performed when x faces (S’, A, U’) or when she faces (S, A, U).

I’m now worried about the following possible counterexample. Let’s start with two games.

Game One. There are two players: P1 and P2. It is common knowledge that each is rational. Each player has a green card and a red card. Their only move in the game is to play one of these cards. If at least one player plays green, they each get $1. If they both play red, they both get $0. P2 has already moved, and played green.

Game Two. There are two players: P1 and P2. It is common knowledge that each is rational. Each player has a green card and a red card. Their only move in the game is to play one of these cards. If at least one player plays green, they each get $1. If they both play red, they both get $0. The moves will be made simultaneously.

Here’s the problem for Ignore Known Falsehoods. The following premises all seem true (at least to me).

- Games are decision problems, with the possible moves of the other player as states.
- In Game One, it doesn’t matter what P1 does, so it is rationally permissible to play red.
- In Game Two, playing green is the only rationally permissible play.
- If premises 1 and 3 are true, then Game Two is an expansion of Game One.

The point behind premise 4 is that if rationality requires playing green in Game Two, and P2 is rational, we know that she’ll play green. So although in Game Two there is in some sense one extra state, namely the state where P2 plays Red, it is a state we know not to obtain. So Game Two is simply an expansion of Game One.

So the big issue, I think, is premise 3. Is it true? It certainly seems true to me. If we think that rationality requires even one round of eliminating weakly dominated strategies, then it is true. Moreover, it isn’t obvious how we can coherently believe it to be false. If it is false, then rational P2 might play red. Unless we have some reason to give that possibility 0 probability, it follows that playing green maximises expected utility.

(There is actually a problem here for fans of traditional expected utility theory. If you say that playing green is uniquely rational for each player, you have to say that two outcomes that have the same expected utility differ in normative status. If you say that both options are permissible, then you need some reason to say they have the same expected utility, and I don’t know what that could be. I think the best solution here is to adopt some kind of lexicographic utility theory, as Stalnaker has argued is needed for cases like this. But that’s not relevant to the problem I’m concerned with.)

So I don’t know which of these premises I can abandon. And I don’t know how to square them with Ignore Known Falsehoods. So I’m worried that Ignore Known Falsehoods is false. Can anyone talk me out of this?

I’m new enough to all of this stuff that I’m not sure I can provide help but I do have a few questions if you don’t mind.

1.) Did you mean to say S’ is a superset of S instead of a superset of x?

2.) It seems to me like reasoning in this case goes as follows:

a.) The only rational decision is green, so my opponent will pick green, therefore, I can pick either colour.

b.) But by symmetry, my opponent will think the same thing, so they will pick either colour, so my only rational decision is green.

And then the reasoning will cycle again:

a*.) But my opponent will think this and so pick green, so I can pick either colour.

So to my ignorant mind, it seems that what decision is rational depends on the depth to which you expect your opponent to model you.

Further, at either stage, it seems like ignore known falsehoods is not a problem. At stage a.) it

isrational to pick either colour. At stage b.) it isn’t but in this case, the opponent picking red isn’t a known falsehood so S’ isn’t an expansion and so the ignore known falsehood principle doesn’t come into play.(James Joyce has an unpublished paper in response to Andy Egan which talks about how a causal decision theorist should make use of the evidence given to them by their own decision, I wonder if this would have anything to say on how a causal decision theorist would reach an equilibrium in this problme?)

Am I missing some obvious argument that makes picking green flat out and clearly the only rational decision regardless of your modelling of the opponent’s modelling of you?

Definitely on (1)!

On (2), I expect my ‘opponent’ (not the best term really in a perfectly cooperative game) to model me to just the same depth as I model them. And the worry is that there’s no coherent way to do that given Ignore Known Falsehoods.

Here is one coherent way for us to model each other. The only rational play is green. But because it is metaphysically possible for the other person to play red, even though I know they won’t, I shouldn’t simply take it as given that they will play green. I know they will play green, of course, but I’m barred for some reason (I wish I knew what) from using that fact in my reasoning.

I’d like to know another coherent way to model the situation, but I can’t tell what it could be.

Put another way, Ignore Known Falsehoods might hold at every stage, but it doesn’t follow that it holds in the limit. And that’s what matters.

How fine grained are states? Do they just include the players’ choices, or other background facts about the environment?

I was thinking states can be as fine grained as you like them to be. If you make them the wrong granularity, they won’t be useful for modeling purposes since the agent won’t face (in my sense) a decision problem, but you will still have a puzzle.

Suppose we give up on IKF. Then our solution to the puzzle you raise is that in Game Two, P1 is rationally required to play green, even though P1 knows that it doesn’t matter whether she plays green or red. (Assuming a modest closure principle.) In other words, we have to deny:

OK: It’s ok to do anything whose results you know would be ok.

This sounds like a slightly crazy option, too. That makes me think that the interesting puzzle you raise is just an interesting puzzle for everybody, not a distinctive source of pressure on IKF. (Notice, for instance, that denying OK might render your (2) somewhat questionable.)

Right, this is basically a counterexample to the Knowledge-Action Principle. It’s very surprising!

As I understand your argument, in your game one, where the two players play sequentially, the player to move first (P2 above) is not rationally required to play green, because if P2 plays red, the player to play second, P1, can be counted on to rationally play green.

I may be misunderstanding something, but it seems to me that your game two is not an expansion of game one.

Game one has, for the second player (P1), two possible states, S1: P2 has

alreadyplayed red, and P1 knows that P2 has already played red (in which case P1 is rationally required to play green), and S2: P2 hasalreadyplayed green and P1 knows that P2 has already played green (in which case P1 is not rationally required to play either one or the other).It seems to me that neither of these states is available in game two. Likewise, the only state existing for P1 in game two is S: P1 and P2 will

simultaneouslyplay a card, with neither knowing before their play which the other has played. It seems to me that this state does not appear in game one.Thus it seems to me that your premise 1, is the problem. It seems that reducing the definition of the states to merely which card P2 has played/may play, without considering the knowledge P1 has in game one and lacks in game two, does not capture the analysis of the difference between the two games. Since the analysis of game one, deciding that P1 is or is not rationally required to play green, actually uses the knowledge of which card P2 has played, it seems that this in a necessary part of the state description facing P1.

Similarly, the lack of this knowledge plays a part in game two.

If I have missed something important here, sorry.

Well that’s one way to divide up the states, by distinguishing the temporal location of the card play. But it isn’t the only way. We could also say that which state one is in is just a function of which card P2 (timelessly) plays.

States are just modelling devices; we can choose whichever order of granularity suits us best.

More importantly, the way Ignore Known Falsehoods is written, as long as there is one way of dividing up modal space into states that makes Game Two an expansion of Game One, the principle applies. And there is; just carve up modal space according to what P2 does (at some time or other).

Brian writes, in comment #2: “I know they will play green, of course, but I’m barred for some reason (I wish I knew what) from using that fact in my reasoning.” I think the reason for that bar can be seen in the set-up of the case, where our reasoning is explicated along the lines that “if rationality requires playing green in Game Two, and P2 is rational, we know that she’ll play green.” So the premise that rationality requires playing green must already be in place to arrive at the further conclusion (combined with the other premises about P2’s rationality, etc.) that P2 will play green. Without that premise, we simply don’t know what P2 will do. Thus, as soon as we take the question of what rationality requires here to be an open question, we lose all that knowledge, and Game Two’s ‘extra state’ becomes live. (And, of course, if it’s a live possible state that P2 plays red, then P1 can quickly reason from there that she’d best choose green herself.)

P1 is thus never in a position both to know that P2 will play green, and to use that knowledge to infer to an evaluation that rationality requires less of them than originally thought. At most, if P1 puts herself in a situation of reconsidering her belief in the rational demand to play green, then she will also have to thereby reconsider what P2 might do, and then right away restore her original belief that rationality requires both of them to play green.

One consequence of this is that Jonathan’s reasoning isn’t going to go through, I think — it’s not ever going to be true in Game Two that P1 knows that it doesn’t matter whether she plays green or red. One way of putting this is that, if P1 comes to believe that it doesn’t matter, then that will be a decisive undercutting defeater to her belief that P2 will choose green, which was in turn a needed premise in the reasoning that is supposed to secure that it doesn’t matter whether she plays green or red.

None of this saves IKF from the counterexample — in particular, it keeps premise 3 in place — but it does suggest how IKF could be weakened slightly and be rendered immune from such counterexamples. The main idea of IKF may be well-suited for situations where the rational evaluations are a function of some independent knowledge of the set of states that might obtain. But it’s liable to run into trouble when the rational evaluations themselves figure into the production of the knowledge used to determine the set of states that might obtain. So, as a first stab:

(IKF*) If (S’, A, U’) is an expansion for x of (S, A, U)

— and if the knowledge that eliminates the states in U’-U does not depend on antecedent knowledge of the rational evaluability of actions in φ

— then the rational evaluability of performing any action φ is the same whether φ is performed when x faces (S’, A, U’) or when she faces (S, A, U).

I suspect that that could be tightened up substantially; though there are some problematic cases of using antecedent knowledge of such rational evaluations, there should be lots of harmless uses of such knowledge as well.

Right. If P1 could rationally infer that red is acceptable, then she couldn’t know P2 would play green, so couldn’t have inferred that red is acceptable. That seems probably right to me.

But I still think we’re left with an unstable end situation. What does P1 know? Assuming she knows P2 will play green seems to lead to contradiction with some weak extra assumptions, and assuming she doesn’t know that also leads to contradiction with some weak extra assumptions. So some very innocuous looking assumption must be false.

Having said that, I think (IKF*) is probably right, and the difference between (IKF) and (IKF*) is enough to suggest there is a problem for things like the Knowledge-Action Principle, or the Reason-Action Principle.

It seems to me you only get a contradiction from the assumption that P1 does not know what P2 will do. That entails that P2 does not know what P1 will do, and from there, that for either the only rational choice is green, and from there, that P2 will choose green. Since P1 is perfectly rational, she will draw all those inferences, and ends up in a state of knowledge that is contrary to the assumption.

But when we assume that P1 knows that P2 will play green, we don’t actually have the means to derive the contradictory result that P1 also doesn’t know that. P1 has to do something further — and, indeed, something not fully rational — to undermine her epistemic position with regard to what P2 might do. She would need first to take it to be a live possible course of action for her to choose red, to start the train of thought that could potentially throw the ex hypothesi knowledge as to P2’s choice into question. But since she

alreadyknows that green is the only rational move for either player, she is not rationally required to seriously consider choosing red at all. (Note that if P1 were to know that P2 will play green, but on someotherbasis than the mutually recognized perfect rationality of the players, then that part of the reasoning won’t hold. We’d have something equivalent to Game One, with red as a rationally permissible play. There still wouldn’t be a contradiction involved.)So I’m not sure there’s a puzzle left about this particular aspect of the case. One cannot consistently make P1 out as ignorant of what choice P2 will make; but one can consistently make P1 out as knowing that P2 will choose green, as the sole rational play for both players.