A few months ago, I posted about a study showing implicit racial bias in NBA referees' calls. Now it's baseball's turn, because yesterday reports of study by Parsons et al.1 that shows analogous results for home plate umpires began popping up all over the media.
The study is pretty straightforward, though the data analysis must have taken forever. I'll let Parson's et al. tell you what they did:
There are 30 teams in Major League Baseball, with each team playing 162 games in each annual season. During a typical game each team's pitchers throw on average roughly 150 pitches, so that approximately 730,000 pitches are thrown each season. We collected pitch-by-pitch data from ESPN.com for every MLB game in the three years 2004-2006.2 For each pitch we identify the pitcher, pitcher's team, batter, batter's team, pitch count, score, inning, and pitch outcome. We classify each pitch into one of seven mutually exclusive categories: Called strike, called ball, swinging strike, foul, hit into play, intentional ball or hit by pitch. We supplement each pitch observation with game-level information from ESPN.com box scores including the stadium name, home team, away team, team standings, and the identities and positions of all four umpires. In addition, for each pitcher's appearance in each game we collect the exact number of innings pitched and the number of allowed hits, walks, strikeouts, homeruns, runs and earned runs. Finally, for each starting pitcher in each game we collect the game score, a composite index designed to summarize a starting pitcher's performance. (p. 4)
Those three seasons yielded a total of 2,120,166 pitches. I feel sorry for the poor undergrads who had to sit and watch all of those! Forty-seven percent of the pitches were excluded because the batter swang or was hit by the pitch, or the pitch was intentionally thrown for a ball, leaving 53% of 2.1 million pitches to analyze. In addition to looking at whether each pitch was a ball or a strike, they also coded the race of the home plate umpire, batter, and pitcher. Using a regression model (see the basketball post for a brief description of regression) that takes only these into account, they find that when the umpire and pitcher are of the same race, the umpire is about .34% more likely to call a strike. That's only a little more than half a pitch a game, but over all of those pitches, and when you take consider that it's only one source of potential discrimination, it's a potentially important effect.
Interestingly, the effect of same-race umpires goes away when the umpire's calls are evaluated by QuesTec's computerized system (only 35% of MLB ballparks had the system between 2004 and 2006), when crowds were larger, and when there were either two strikes, three balls, or both (meaning that the call could potentially end the bat). Parsons et al. argue that these results suggest that the cost of a decision will affect the influence of implicit racial bias. Specifically, the data implies that costlier decisions will display less bias, a finding that could have a wide range of implications in research on implicit biases.
1Parsons, C.A., Sulaeman, J., Yates, M.C., &Hamermesh, D.S. (Unpublished Manuscript). Strike three: Umpires' demand for discrimination.
- Log in to post comments
A third of one percent? Half a pitch a game? That's nothing. Are you (or the researchers) seriously suggesting that umpires are so racially biased against minority players that they ring them up to the tune of one more strike every other game? That's ridiculous. It's either the most subtle (not to mention mild) form of racism every discovered, or it's an innocent statistical anomaly. Surely it's the latter.
eh? Less than half a pitch a game? i.e. about 1 pitch every second to third game?
I'd like to see the significance tests on that. Since they ignored all the other data they collected, which could also be correlated, this is actually showing pretty unbiased umpiring really. I mean, whether the ump was insulted by the manager the last inning might have something to do with things too, and at a one pitch every 2 or 3 game level, that correlation would be very confounding, to my mind anyway. That does look like an upper limit on what would be visible bias.
A couple things: the difference is, of course significant, but with a sample size like that, any difference would be. But it's important to note that the difference is collapsed across races. It's actually higher for some.
But the most interesting finding is that the effect goes away in high pressure situations.
Chris, just a quick question - isn't the effect going away in costlier decision situations associated with a lack of power (because of, I presume, a radically lower number of observations)...?
Whatever the case, interesting study.
I don't claim to understand the statistical methods used to show this bias. I think the most interesting part of the paper is figure 4 (http://www.eco.utexas.edu/faculty/Hamermesh/Baseball4Authors.pdf), which looks like it says that minority pitchers win 9% more often when there is an umpire of their ethnicity. That's really shocking.
Here's my first question for people who know more about statistical methods than I do. Is 8 minority umpires too few to compare to 85 white umpires and find significant results? They report including 5 black and 3 hispanic umpires. I know they looked at millions of pitches, but they probably looked at about 10 times more pitches called by white umpires, which seems significant, especially when we're talking about less than one-half percent.
Second question: If there was a subtle bias, why would it not show up with batters as well? Why would an umpire unconsciously hurt or help a pitcher yet not do that for a hitter? I guess that's less of a statistical question than a suggestion that I am mystified by the analysis.