A few more thoughts on quarterbacks:
There are two separate issues with respect to quarterbacks. The first is whether, historically, NFL teams have done a good job of predicting which college quarterbacks will succeed in the pros. Dave Berri and Rob Simmons’ paper in the Journal of Productivity Analysis (that I relied on in the essay “Most Likely to Succeed” in my new book “What The Dog Saw”) proves pretty convincingly, I think, that the answer is no. One of the best parts of that paper is how Berri and Simmons demonstrate how much NFL teams tend to irrationally over-weight “combine” variables like speed, height and Wonderlic score.
There’s a second wonderful paper on this general subject by Cade Massey and Richard Thaler—Thaler being, of course, one of the leading lights in behavioral economics—called “The Loser’s Curse.” The argument of the Thaler-Massey paper goes something like this (and I encourage anyone who is interested in sports to read the whole thing, because I can’t do it justice here). By looking at the trades that NFL teams make, we can estimate the “market value” of a draft pick. And what we find is that teams place a very high value on high first round picks. The first pick in the draft, they write, has historically been valued as much as “the 10th and 11th picks combined, and as much as the sum of the last four picks in the first round.” Then Thaler and Massey calculate the true value of draft picks, using what they call “surplus value.” The key here is that all NFL teams operate under a strict salary cap. So a player’s real worth to a team is the extent to which his performance exceeds the average performance of someone making his salary. And what do they find? That market value and surplus value are radically out of sync: that teams irrationally over-weight the importance of high first round picks. In fact, according to their analysis, the most useful draft picks are in the second round, not the first: that’s where surplus values tend to be highest. Hence the title of the paper: “The Loser’s Curse.” The NFL rewards its weakest teams by giving them the highest draft picks—but those picks are actually not the most valuable picks in the draft.
It is important to note here that we are talking about relative value. Personnel decisions in the NFL have clear opportunity costs: if you pay $15 million for a quarterback who only gives you $10 million of value, then you hve $5 million less to pay for a good linebacker. As they write: “To be clear, the player taken with the first pick does have the highest expected performance . . . but he also has the highest salary, and in terms of performance per dollar, is less valuable than players taken in the second round.”
What Massey and Thaler are saying, in essence, is that NFL general managers are not rational decision-makers. That’s why I think its so useful in this particular discussion. Those who believe that draft position is a good predictor of quarterback performance are essentially voting for the good judgment of the people who make draft decisions. And what Berri and Simmons in particular—and Massey and Thaler in general—remind us is that that kind of blind faith in the likes of Matt Millen and Al Davis simply isn’t justified. And, by the way, why should that fallibility come as a surprise? We’ve known for a long time that it is not easy to making decisions under conditions of extreme uncertainty. Here is Massey and Thaler from their conclusion:
Numerous studies find, for example, that physicians, among the most educated professionals in our society, make diagnoses that display overconfidence and violate Bayes’ rule. The point, of course, is that physicians are experts at medicine, not necessarily probabilistic reasoning. And it should not be surprising that when faced with difficult problems, such as inferring the probability that a patient has cancer from a given test, physicians will be prone to the same types of errors that subjects display in the laboratory. Such findings reveal only that physicians are human.
Our modest claim in this paper is that the owners and managers of National Football League teams are also human, and that market forces have not been strong enough to overcome these human failings. The task of picking players, as we have described here, is an extremely difficult one . . . Teams must first make predictions about the future performance of (frequently) immature young men. Then they must make judgments about their own abilities: how much confidence should the team have in its forecasting skills? As we detailed in section 2, human nature conspires to make it extremely difficult to avoid overconfidence in this task.
This brings up the second question. Is it possible to ever accurately predict which college quarterbacks will succeed in the pros? Both the Thaler analysis and the Berri analysis hold out the real possibility that teams can be a lot smarter than they currently are. The New England Patriots clearly have taken some of Thaler’s lessons to heart, for example. There has also been a real effort by the folks over at Pro Football Outsiders to come up with a more useful algorithm for making quarterback selections. David Lewin’s “career forecast” zeroes in on career college starts and career college completion percentage as the best predictors of professional performance. I took the position in my essay “Most Likely to Succeed” that I didn’t think that quarterbacking (as opposed to other positions on the field) was predictable in this sense—that there is so much noise in the data, and so much variability between the college and professional games—that attempts at rationalizing draft day decisions have real limits. I’m still of that inclination. I’m willing to be convinced, though. I’d love to see more statistically-minded people weigh in on the Lewin analysis, and I’d also like to have a better handle over how the recent innovations in college offenses—particularly the use of ever more aggressive spread formations—affects the accuracy of that algorithm.