gladwell.com

Letting Igons be Igons

 From "Blowing Up," by Malcolm Gladwell, The New Yorker, April 22,2002

More on Quarterbacks

 

A few more thoughts on quarterbacks:

 

There are two separate issues with respect to quarterbacks. The first is whether, historically, NFL teams have done a good job of predicting which college quarterbacks will succeed in the pros. Dave Berri and Rob Simmons’ paper in the Journal of Productivity Analysis (that I relied on in the essay “Most Likely to Succeed” in my new book “What The Dog Saw”) proves pretty convincingly, I think, that the answer is no. One of the best parts of that paper is how Berri and Simmons demonstrate how much NFL teams tend to irrationally over-weight “combine” variables like speed, height and Wonderlic score.

There’s a second wonderful paper on this general subject by Cade Massey and Richard Thaler—Thaler being, of course, one of the leading lights in behavioral economics—called “The Loser’s Curse.” The argument of the Thaler-Massey paper goes something like this (and I encourage anyone who is interested in sports to read the whole thing, because I can’t do it justice here). By looking at the trades that NFL teams make, we can estimate the “market value” of a draft pick. And what we find is that teams place a very high value on high first round picks. The first pick in the draft, they write, has historically been valued as much as “the 10^th and 11^th picks combined, and as much as the sum of the last four picks in the first round.” Then Thaler and Massey calculate the true value of draft picks, using what they call “surplus value.” The key here is that all NFL teams operate under a strict salary cap. So a player’s real worth to a team is the extent to which his performance exceeds the average performance of someone making his salary. And what do they find? That market value and surplus value are radically out of sync: that teams irrationally over-weight the importance of high first round picks. In fact, according to their analysis, the most useful draft picks are in the second round, not the first: that’s where surplus values tend to be highest. Hence the title of the paper: “The Loser’s Curse.” The NFL rewards its weakest teams by giving them the highest draft picks—but those picks are actually not the most valuable picks in the draft.

    It is important to note here that we are talking about relative value. Personnel decisions in the NFL have clear opportunity costs: if you pay $15 million for a quarterback who only gives you $10 million of value, then you hve $5 million less to pay for a good linebacker. As they write: “To be clear, the player taken with the first pick does have the highest expected performance . . . but he also has the highest salary, and in terms of performance per dollar, is less valuable than players taken in the second round.”

What Massey and Thaler are saying, in essence, is that NFL general managers are not rational decision-makers. That’s why I think its so useful in this particular discussion. Those who believe that draft position is a good predictor of quarterback performance are essentially voting for the good judgment of the people who make draft decisions. And what Berri and Simmons in particular—and Massey and Thaler in general—remind us is that that kind of blind faith in the likes of Matt Millen and Al Davis simply isn’t justified. And, by the way, why should that fallibility come as a surprise? We’ve known for a long time that it is not easy to making decisions under conditions of extreme uncertainty. Here is Massey and Thaler from their conclusion:

 

Numerous studies find, for example, that physicians, among the most educated professionals in our society, make diagnoses that display overconfidence and violate Bayes’ rule. The point, of course, is that physicians are experts at medicine, not necessarily probabilistic reasoning. And it should not be surprising that when faced with difficult problems, such as inferring the probability that a patient has cancer from a given test, physicians will be prone to the same types of errors that subjects display in the laboratory. Such findings reveal only that physicians are human.

Our modest claim in this paper is that the owners and managers of National Football League teams are also human, and that market forces have not been strong enough to overcome these human failings. The task of picking players, as we have described here, is an extremely difficult one . . . Teams must first make predictions about the future performance of (frequently) immature young men. Then they must make judgments about their own abilities: how much confidence should the team have in its forecasting skills? As we detailed in section 2, human nature conspires to make it extremely difficult to avoid overconfidence in this task.

   

   This brings up the second question. Is it possible to ever accurately predict which college quarterbacks will succeed in the pros? Both the Thaler analysis and the Berri analysis hold out the real possibility that teams can be a lot smarter than they currently are. The New England Patriots clearly have taken some of Thaler’s lessons to heart, for example. There has also been a real effort by the folks over at Pro Football Outsiders to come up with a more useful algorithm for making quarterback selections. David Lewin’s “career forecast” zeroes in on career college starts and career college completion percentage as the best predictors of professional performance. I took the position in my essay “Most Likely to Succeed” that I didn’t think that quarterbacking (as opposed to other positions on the field) was predictable in this sense—that there is so much noise in the data, and so much variability between the college and professional games—that attempts at rationalizing draft day decisions have real limits. I’m still of that inclination. I’m willing to be convinced, though. I’d love to see more statistically-minded people weigh in on the Lewin analysis, and I’d also like to have a better handle over how the recent innovations in college offenses—particularly the use of ever more aggressive spread formations—affects the accuracy of that algorithm.

 

Pinker on "What the Dog Saw."

Steven Pinker reviewed my new book "What the Dog Saw," in the New York Times Book Review this past Sunday. I sent the following letter to the editor in response:

It is always a pleasure to be reviewed by someone as accomplished as Stephen Pinker, even if—in his comments on “What the Dog Saw” (Nov. 15)—he is unhappy with my spelling (rightly!) and with the fact that I have not joined him on the lonely ice floe of IQ fundamentalism. But since football has been on my mind these days, I do want to make one small observation about his comments.

  In one of my essays, I wrote that the position a quarterback is taken in the college draft is not a reliable indicator of his performance as a professional. That was based on the work of the academic economists David Berri and Rob Simmons, who, in a paper published the Journal of Productivity Analysis,  analyze forty years of National Football League data. Their conclusion was that the relation between aggregate quarterback performance and draft position was weak. Further, when they looked at per-play performance—in other words, when they adjusted for the fact that highly drafted quarterbacks are more likely to play more downs—they found that quarterbacks taken in positions 11 through 90 in the draft actually slightly outplay those more highly paid and lauded players taken in the draft’s top ten positions. I found this analysis fascinating. Pinker did not. This quarterback argument, he wrote, “is simply not true.”

    I wondered about the basis of Pinker’s conclusion, so I e-mailed him, asking if he could tell me where to find the scientific data that would set me straight. He very graciously wrote me back. He had three sources, he said. The first was Steve Sailer. Sailer, for the uninitiated, is a California blogger with a marketing background who is best known for his belief that black people are intellectually inferior to white people. Sailer’s “proof” of the connection between draft position and performance is, I’m sure Pinker would agree, crude: his key variable is how many times a player has been named to the Pro Bowl. Pinker’s second source was a blog post, based on four years of data, written by someone who runs a pre-employment testing company, who also failed to appreciate—as far as I can tell (the key part of the blog post is only a paragraph long)—the distinction between aggregate and per-play performance. Pinker’s third source was an article in the Columbia Journalism Review, prompted by my essay, that made an argument partly based on a link to a blog called “Niners Nation." I have enormous respect for Professor Pinker, and his description of me as “minor genius” made even my mother blush. But maybe on the question of subjects like quarterbacks, we should agree that our differences owe less to what can be found in the scientific literature than they do to what can be found on Google.