The "All Else Equal" Fallacy, again

Here's the entry from the statistical lexicon:

The "All Else Equal" Fallacy: Assuming that everything else is held constant, even when it's not gonna be.

My original note about this fallacy came a couple years ago when New York Times columnist John Tierney made the counterintuitive claim (later blogged by Steven Levitt) that driving a car is good for the environment. As I wrote at the time:

These guys are making a classic statistical error, I think, which is to assume that all else is held constant. This is the error that also leads people to misinterpret regression coefficients causally. (See chapters 9 and 10 of our book for discussion of this point.) In this case, the error is to assume that the walker and the driver will be making the same trip. In general, the driver will take longer trips--that's one of the reasons for having a car, that you can easily take longer trips. Anyway, my point is not to get into a long discussion of transportation pricing, just to point out that this seemingly natural calculation is inappropriate because of its mistaken assumption that you can realistically change one predictor, leaving all the others constant.

I hadn't thought much about this but then I see that Levitt repeated this error in his new Freakonomics book and on his blog, where he writes:

In SuperFreakonomics, far and away the most common subject of emails is drunk walking vs. drunk driving. In particular, every few days someone writes us to tell us that our analysis is wrong because we are comparing the rate of death per mile driven drunk versus the rate of death per mile walked drunk. Sure, they say, drunk walkers get killed more per mile. But since cars travel much faster, per hour, it is safer to drive drunk than to walk drunk.

It is true that if someone held a gun to your head and said, "If you don't walk drunk for an hour or drive drunk for an hour, I will shoot you. You choose whether you would walk or drive," then you might very well want to spend your hour walking drunk. However, in real life, that is virtually never the dilemma you face. Rather, you are drunk in one place and you want to get to another place. The distance you need to cover is what is constant, not the time you will spend traveling.

Thus the per-mile comparison we [Levitt and Dubner] make is the most sensible one.

Well, as the saying goes, saying it don't make it so. I have to admit that I've never been drunk, so I'm not the authority on this topic, but, in general, no, people are not necessarily trying to get from point A to point B. If Point A is 10 miles from Point B, I doubt that many drunk people are going to try to walk from A to B. They might drive from A to B, or they might walk to the bus stop and take the bus, or they might walk to a friend's house, or whatever. Or, if they drive, they might decide to stop at the supermarket at the way home, adding another couple of miles to their trip.

More to the point, the very existence of drunk driving as an option can put you in the situation where you and you car are 10 miles from home, you're drunk, and the most convenient option is to get in the car and try to make it back. (Levitt talks about the cost of taking a cab but not about the time cost required to go back and get the car.) If the only available options were to walk or take the bus, maybe people wouldn't put themselves in this position in the first place--and, in any case, then they could get back home one way or another without having to worry about their car.

In their book, Levitt and Dubner give a very specific example of someone who lives exactly one mile away, but that misses the point, since the real concern about drunk driving is habitual behavior, not this sort of single-play game.

I also think it's a bit irresponsible for Levitt to mention "other risks associated with driving drunk like getting arrested" without mentioning the risk of hurting or killing someone else who happens to be in the path of your car. (He mentions it briefly in the book, but only briefly.) After all, the danger to others is a key reason that it's illegal in the first place!

A meta-lesson: Microeconomics ain't easy, and don't let a regression--or division by a baseline--be a substitute for clear thought. It's a classic error to analyze a decision as if it were a one-time choice, without recognizing the underlying incentives that make the situation come up repeatedly. It's disappointing to see Levitt make this mistake and then see him double down and defend his error.

P.S. The funny thing is that Levitt realizes that many people are bothered by his calculation. He might think a bit harder about why those people might be right! The commenters on Levitt's blog make a bunch of good points, so maybe he'll read these and change his mind.

P.P.S. Taking a cab might not be so safe. I've known a couple people who were seriously injured in cab rides. Also, cabs sometimes run over and even kill pedestrians, and cab drivers can be drunk too! Once when I was living in Chicago I was in a cab whose driver was extremely drunk--I could smell it. Luckily, he must have realized it, and he was driving very very slowly. Typically I'm impatient when I'm in a cab, but this time I wanted him to take as much time as he needed!

P.P.P.S. Just to clarify, I'm not saying that Levitt's analysis is necessarily wrong: something useful can definitely be learned from the comparisons of risks per mile of driving and walking, and comparing at different times of day, different states of inebriation, and so forth. Where I think Levitt is wrong is in his categorical claim that his comparison "is the most sensible one," as if he's the expert here. Sorry, but I think his blog commenters have more sensible things to say on this than he does.

The real issue, I think, is that, when doing "Freakonomics," you can be counterintuitive, or you can be sensible, but it's hard to be both. I mean, sure, sometimes you can be. But there's a tradeoff, and in this case, Levitt is choosing to push the envelope on counterintuitiveness. As long as nobody actually decides to drive drunk because of the book, it's fine, I guess. As I said, I think the actual calculations have some value, as long as you're a bit more careful in the interpretation.

P.P.P.P.S. I've been writing about Levitt so much lately, that I should probably state just one more time that I think his blog is great. Really, how many forums are out there where people can discuss this sort of issue. That particular blog entry has 81 comments, most of which make good points. The Freakonomics blog exposes lots of people to rigorous, data-based thinking and gives a wide audience ot a lot of interesting research. And of course I wouldn't spend my time criticizing (some of) their blog if I didn't think that it was, on balance, a good thing.