More on Quarterbacks

 

A few more thoughts on quarterbacks:

 

There are two separate issues with respect to quarterbacks. The first is whether, historically, NFL teams have done a good job of predicting which college quarterbacks will succeed in the pros. Dave Berri and Rob Simmons’ paper in the Journal of Productivity Analysis (that I relied on in the essay “Most Likely to Succeed” in my new book “What The Dog Saw”) proves pretty convincingly, I think, that the answer is no. One of the best parts of that paper is how Berri and Simmons demonstrate how much NFL teams tend to irrationally over-weight “combine” variables like speed, height and Wonderlic score.

There’s a second wonderful paper on this general subject by Cade Massey and Richard Thaler—Thaler being, of course, one of the leading lights in behavioral economics—called “The Loser’s Curse.” The argument of the Thaler-Massey paper goes something like this (and I encourage anyone who is interested in sports to read the whole thing, because I can’t do it justice here). By looking at the trades that NFL teams make, we can estimate the “market value” of a draft pick. And what we find is that teams place a very high value on high first round picks. The first pick in the draft, they write, has historically been valued as much as “the 10^th and 11^th picks combined, and as much as the sum of the last four picks in the first round.” Then Thaler and Massey calculate the true value of draft picks, using what they call “surplus value.” The key here is that all NFL teams operate under a strict salary cap. So a player’s real worth to a team is the extent to which his performance exceeds the average performance of someone making his salary. And what do they find? That market value and surplus value are radically out of sync: that teams irrationally over-weight the importance of high first round picks. In fact, according to their analysis, the most useful draft picks are in the second round, not the first: that’s where surplus values tend to be highest. Hence the title of the paper: “The Loser’s Curse.” The NFL rewards its weakest teams by giving them the highest draft picks—but those picks are actually not the most valuable picks in the draft.

    It is important to note here that we are talking about relative value. Personnel decisions in the NFL have clear opportunity costs: if you pay $15 million for a quarterback who only gives you $10 million of value, then you hve $5 million less to pay for a good linebacker. As they write: “To be clear, the player taken with the first pick does have the highest expected performance . . . but he also has the highest salary, and in terms of performance per dollar, is less valuable than players taken in the second round.”

What Massey and Thaler are saying, in essence, is that NFL general managers are not rational decision-makers. That’s why I think its so useful in this particular discussion. Those who believe that draft position is a good predictor of quarterback performance are essentially voting for the good judgment of the people who make draft decisions. And what Berri and Simmons in particular—and Massey and Thaler in general—remind us is that that kind of blind faith in the likes of Matt Millen and Al Davis simply isn’t justified. And, by the way, why should that fallibility come as a surprise? We’ve known for a long time that it is not easy to making decisions under conditions of extreme uncertainty. Here is Massey and Thaler from their conclusion:

 

Numerous studies find, for example, that physicians, among the most educated professionals in our society, make diagnoses that display overconfidence and violate Bayes’ rule. The point, of course, is that physicians are experts at medicine, not necessarily probabilistic reasoning. And it should not be surprising that when faced with difficult problems, such as inferring the probability that a patient has cancer from a given test, physicians will be prone to the same types of errors that subjects display in the laboratory. Such findings reveal only that physicians are human.

Our modest claim in this paper is that the owners and managers of National Football League teams are also human, and that market forces have not been strong enough to overcome these human failings. The task of picking players, as we have described here, is an extremely difficult one . . . Teams must first make predictions about the future performance of (frequently) immature young men. Then they must make judgments about their own abilities: how much confidence should the team have in its forecasting skills? As we detailed in section 2, human nature conspires to make it extremely difficult to avoid overconfidence in this task.

   

   This brings up the second question. Is it possible to ever accurately predict which college quarterbacks will succeed in the pros? Both the Thaler analysis and the Berri analysis hold out the real possibility that teams can be a lot smarter than they currently are. The New England Patriots clearly have taken some of Thaler’s lessons to heart, for example. There has also been a real effort by the folks over at Pro Football Outsiders to come up with a more useful algorithm for making quarterback selections. David Lewin’s “career forecast” zeroes in on career college starts and career college completion percentage as the best predictors of professional performance. I took the position in my essay “Most Likely to Succeed” that I didn’t think that quarterbacking (as opposed to other positions on the field) was predictable in this sense—that there is so much noise in the data, and so much variability between the college and professional games—that attempts at rationalizing draft day decisions have real limits. I’m still of that inclination. I’m willing to be convinced, though. I’d love to see more statistically-minded people weigh in on the Lewin analysis, and I’d also like to have a better handle over how the recent innovations in college offenses—particularly the use of ever more aggressive spread formations—affects the accuracy of that algorithm.

 

Comments

[Eric Olsen]

fascinating stuff, Malcolm! Now if I'm an NFL owner, I want to know statistically where in the draft are these "value" players? Where am I going to get the most bang for my buck? Whether it's the bottom half of the first round, or the top half of the second round...and then ALWAYS trade down my picks to get more of these "value" positions.

[Smperk]

I can't wait to see what happens to the Patriots with their 3 second round picks next year and if it feeds into Thaler-Massey's research.

[Stan Hansen - Brown Egg Marketing]

"In fact, according to their analysis, the most useful draft picks are in the second round" This just seems statistically unsound, even though I get it. Some teams (i.e. my Bears) often trade away their 1st round picks b/c of this BUT that limits their selection at the best talent. In fact that means that they are usually picking at least the 32nd most talented player. Crazy.

[Paul Orwin]

An important point here is that the GM/owner may have more on his/her mind than maximal player value per dollar. A simple and likely answer here is that they can boost attendance at games the next season with a big name quarterback a lot more than with a very talented left tackle, even though the lineman may be the better choice (cheaper, easier to predict). Trading for an experienced franchise quarterback can be expensive, and not without risks (Jay Cutler; I'm a bears fan, what can I say). So perhaps taking Matt Leinart is good for 3000 extra people in the stands next season, while the sure thing (or two "value players" in the 2nd round) gets you nothing. Of course, winning tends to sell tix too, but now we are back on familiar ground; instant gratification vs. long term strategic thinking. That one, I'm afraid, was settled a long time ago, and not just for football teams.

[Travis Leleu]

@Eric Olsen,

I think even more interesting than looking at surplus value based on selection number would be to cluster the different *types* of players. Looking at on-field position and draft number from a "value" point of view would be very interesting.

I've seen lots of baseball analysis like this, mostly split on hitters/pitchers and college/baseball. For football, it might be worth looking at what conference the guy played in as well (SEC is going to be a lot tougher than some others).

[Norwegian Shooter]

Okay, how about the other criticisms from Pinker?

[test]

Hey Malcolm,
I notice you use the word "algorithm" a couple of times in your post. That reminds me of how in Blink you defined an "algorithm" as an "equation".
Cheers.

[Tony]

I think “The Loser’s Curse” is overvaluing “surplus value”. Getting better players per dollar is not nearly as important in the NFL as the salary cap would make it seems. Great players are what you are looking for in the draft. If you get someone like Peyton Manning, your team always has a chance. I bet there is a huge correlation to wins and great QBs. A high first round pick is valuable because it gives you a shot a great player.

[Mike]

Thaler-Massey basically took the MONEYBALL theory and tried to apply it to football -- and did it poorly.

They implicitly assume that players are traded based solely on performance, ignoring the financial aspects of trades.

Question: Why would the Milwaukee Bucks EVER trade Kareem Abdul Jabbar for 5 stiffs and some cash?

Answer: Player salaries are DEPRECIABLE. Kareem's salary was fully depreciated, so trading him made great sense FINANCIALLY even though it didn't make sense from a performance standpoint.

This simple example shows why you can't determine "market value" by simply looking at the draft position of players invovled in trades - there is a LOT more to it than that.

I'd wager money that Thaler and Massey can't even win their fantasy football leagues, let alone do a good job in drafting NFL quarterbacks.

[Steve Bezner]

Some good thoughts here, a la Moneyball and the Beane approach to baseball.

Obviously statistical approaches are incomplete; the Titans benched Vince Young after an unsatisfying start to his career (Young had a very good completion rate as a college player; not stellar, but very good), but now find themselves starting him again, to much better results. I would be interested to know how the statistical component relates to psychological profiling.

In other words, how much of a "leader" mentality or "play to the whistle" thought process does a QB need in order complement his completion percentage?

The Cowboys lucked into a value with Tony Romo, but have been maddened by his decision-making skills (less so this season, but you get the point).

Also: does D1 vs. D2 or D3 experience factor in?

[Jim_S]

Mr. Orwin is right. The value of a player to a team is not the value of his athletic performance. It is his ability to attract revenue. (and we see this reflected in salaries/endorsements today).

That factor is too big to ignore and is not be included in this theory, so the theory no longer works.

At all.

[Steve Davis]

I was hoping New England would get a little more credit for this from most of the authors involved (I am not a Patriots fan). Much like Bean in Oakland, they adopted this type of reasoning before anyone else ever began contemplation. Beane just had the ego to allow a book to be published chronicling his "genius".

I think a lot of teams have fallen backward into this thought process. It is very difficult for teams in the top 3 to trade down even 1 spot anymore. The cost/benefit is extremely skewed and fueling discussion on a rookie pay scale much like the NBA employs.

The perfect example of this concept is the Arizona Cardinals 2003 draft. They traded the 6th overall pick for, essentially, 2 other first round picks. With those two later picks they drafted Bryant Johnson WR and Calvin Pace DE. Both are no longer with the team after dismal performances for the first 3 years of their career. They did strike gold with their 2nd round pick in Anquan Boldin. It is perfect evidence of the theories presented.

[Maynard Handley]

"The point, of course, is that physicians are experts at medicine, not necessarily probabilistic reasoning. And it should not be surprising that when faced with difficult problems, such as inferring the probability that a patient has cancer from a given test, physicians will be prone to the same types of errors that subjects display in the laboratory. Such findings reveal only that physicians are human."

This is FAR too kind to doctors.
The difference between doctors and other people is that other people accept that they are not experts and outsource the job to someone who is an expert.
Doctors, on the other hand, get all pissy about how no-one else can possibly understand their cool doctor insights and they have to be the ones to do the job, no matter how lousy they are at it. It is doctors who scream bloody murder when it is suggested that they follow checklists, or use expert systems, or do any other damn thing that might improv outcomes. It is doctors that scream about communism and socialized medicine when states or the feds try to get them to base their diagnoses on science rather than voodoo --- 17 years plus to move from medical journals to common practice --- hell of an achievement there. Hippocratic oath my ass -- judge by what these people do, not what they say.

And the US public stands for this. No matter what random self-serving crap doctors throw out to justify their absolute LACK of professionalism, whether it's ridiculous internist hours or their ongoing fight against electronic records, the public accepts it. "So trainee doctors actually make *better* decisions after they have been awake for 40 hours? Hmm, that sounds unlikely, but you have a white coat so it must be true".

Malcolm, people in the media like yourself have a responsibility to call this profession out. I don't give a damn about anecdotes regarding a few saints. Just like your recent article on Atticus Finch, the issue is not the behavior of individuals, the issue is the structure of the system --- a system that most of these people defend, and defend aggressively.
Sure, a few quarterbacks suffer brain damage every year, after being paid a lot of money and having made a choice. What about the *millions* of years of life that are lost and ruined every year through the many many many failings of the US medical system --- the same system that is defended to the death by the doctor's union AKA the AMA? How about you show at least the same level of outrage?

[Steve Sailer]

Malcolm:

But what you published in The New Yorker was something very different than what you now say in this blog post. What you actually published in "Most Likely to Succeed" was:

"This is the quarterback problem. There are certain jobs where almost nothing you can learn about candidates before they start predicts how they'll do once they're hired. ... The problem with picking quarterbacks is that [U. of Missouri quarterback] Chase Daniel's performance can't be predicted. The job he's being groomed for is so particular and specialized that there is no way to know who will succeed at it and who won't. In fact, Berri and Simmons found no connection between where a quarterback was taken in the draft—that is, how highly he was rated on the basis of his college performance—and how well he played in the pros."

Professor Pinker called you on on this, writing in the NY Times: "It is simply not true that a quarter­back’s rank in the draft is uncorrelated with his success in the pros."

You chose to respond to Pinker in an open letter and you chose not to try to laugh it off by saying something like, "Oh, sure, I was just being over-dramatic to hype my article. Of course high draft choices do better than low draft choices, but I was just trying to emphasize that there is a lot of uncertainty."

Instead, you got all sanctimonious in your reply to Pinker.

Now that you are better informed about the the topic you previously wrote about, you should ask The New Yorker to publish a correction of what you wrote.

[Hendrick Lee]

I am not an expert in football but couldn't Thaler's result also indicate that individual players do not matter much? As in, in a bad team, even if top draft players cannot perform well. This would not be difficult to test I think; by measuring the "value" of a player using Thaler's method and how 'value' transitions from one team to another.

[Jim Myers]

One way to improve an algorithm for projecting a college quarterback's success in the pros would be to find out what good NFL quarterbacks excelled at in college. You could make a list of the ten best quarterbacks each year for the past decade, then dig up their college stats and see which area they excelled in. These areas would then be weighted more heavily in choosing which quarterback to draft.

However, I agree that player statistics can only tell you so much in this situation. The thing that matters most is in-person talent scouting, but I think a smart algorithm could be useful in providing another perspective. I'd like to see a study that illustrates how thorough in-person talent scouting relates to NFL franchise success.

And on the topic of recent football trends impacting all this, I'd say the variables are decreasing all the time because the college game is becoming more and more like the pros. College athletes are recruited like pros because now playing in a top program means heavily influencing where they get drafted, which means how much they get paid. College football is a big business and coaches are paid millions to win, so we are seeing more and more pro-style offenses, like the spread and the no-huddle. The game is also played faster and more physically, and teams are under a great amount of personal and career pressure, similar to the pros. All these factors plus the emergence of the wildcat offense in the NFL leads me to believe that teams will have an easier time evaluating which players will be successful in the pros. Look at the first three quarterbacks drafted two years ago: Matt Ryan and Joe Flacco are starters on teams that made the playoffs last year and currently have winning records, and Josh Freeman beat out two other quarterbacks to win a starting job on the awful Tampa Bay Buccaneers.

[Maria]

You are still not addressing the most important part of the criticism: that per-play statistics are the wrong way to look at this problem.

[Christopher Daggett]

@Comments addressing off-field value of players, e.g. ticket & merchandise sales. In the paper, Thaler-Massey state:

"A more subtle argument is that the utility to the team of signing a high draft pick is derived from
something beyond on-field performance. A very exciting player, Michael Vick comes to mind, might help sell tickets and team paraphernalia in a way his performance statitistics do not reflect. We are skeptical of such arguments generally. Revenue from paraphernalia sales are shared equally across all teams (18). Moreover, few football players (Vick may be the only one) are able to bring in fans without successful team performance. But in any case, if high draft-picks had more fan appeal this should show up in their 6th-year contracts, and we find no evidence for it.

(18) All licensing revenues from club names and team colors are split evenly among the clubs as part of NFL Properties; individual player jersey licensing revenues are part of Players Inc group licensing and each player who
has signed a Group Licensing Agreement--approximately 98% of them--gets an equal share of all Inc revenues (after expenses) and the individual player gets compensated based on how many of his jerseys have been sold.”
(Duberstein, 2005)

[Dan]

Malcolm,

I'm disappointed and a little surprised that you actually have the gall to quote a paper (the one by Thaler) that explicitly states that "the player taken with the first pick does have the highest expected performance" and yet not admit that your previous blog post--your overly snarky response to Steven Pinker's criticism of you--was wrong.

It's great that, in the face of high-profile public criticism, you've decided to go and do a little more research into a topic of contention, but I'm baffled by how, after apparently bothering to review the literature a bit more, you nonetheless concluded that that literature which explicitly disagreed with you somehow validated your prior assumptions.

I used to think you were a science writer who merely lacked a bit of rigor and self-discipline, and not the dishonest hack I now think you much more likely to be.

You should be ashamed of yourself. We all make mistakes; what differentiates the honest from the rest is that they have the honor to admit to those mistakes.

[Steve Sailer]

Here is Prof. Pinker's reply to Mr. Gladwell's letter to the editor:

Steven Pinker replies:

"What Malcolm Gladwell calls a “lonely ice floe” is what psychologists call “the mainstream.” In a 1997 editorial in the journal Intelligence, 52 signatories wrote, “I.Q. is strongly related, probably more so than any other single measurable human trait, to many important educational, occupational, economic and social outcomes.” Similar conclusions were affirmed in a unanimous blue-ribbon report by the American Psychological Association, and in recent studies (some focusing on outliers) by Dean Simonton, David Lubinski and others.

"Gladwell is right, of course, to privilege peer-reviewed articles over blogs. But sports is a topic in which any academic must answer to an army of statistics-savvy amateurs, and in this instance, I judged, the bloggers were correct. They noted, among other things, that Berri and Simmons weakened their “weak correlation” (Gladwell described it in the New Yorker essay reprinted in “What the Dog Saw” as “no connection”) by omitting the lower-drafted quarterbacks who, unsurprisingly, turned out not to merit many plays. In any case, the relevance to teacher selection (the focus of the essay) remains tenuous."

http://www.nytimes.com/2009/11/29/books/review/Letters-t-LETSGOTOTHET_LETTERS.html?_r=1

[Marsha Keeffer]

All of this makes me wonder how a player like the iconic Jerry Rice would be rated. His freshman year in professional football was in no way a harbinger of his brilliant career.

[Pro Gnosticator]

"All of this makes me wonder how a player like the iconic Jerry Rice would be rated."

Wild guess? Middle of the first round. #16 overall, if I had to pick. But I guess we'll never know...

[Mark]

"The first is whether, historically, NFL teams have done a good job of predicting which college quarterbacks will succeed in the pros. Dave Berri and Rob Simmons’ paper...proves pretty convincingly, I think, that the answer is no."

I'm sorry, but Berri and Simmons are a disaster. Here's what Berri claims:

"this is because in the NFL...draft position is linked to playing time. And this link is independent of performance."

This is crap. There's no way that a quarterback who's drafted #1 who's bad (Alex Smith) gets to play as much as a quarterback who's drafted #1 who's good (Peyton Manning.)

"Independent of performance" means R^2 = 0 for a guy like Berri who can't do anything but throw horse meat into SPSS and turn the crank.

Controlling for draft position, QB playing time in the NFL is absolutely not independent of performance. I defy Berri to prove that.

[john of sparta]

agreed.
switching sports to make my point:
Dale Earnhardt, Jr. is the most
popular in NASCAR, however he
doesn't win. Jimmy Johnson does
win, and nobody cares. DEJR increases ticket sales, TV revenue, etc. JJ just wins. ZZZZZZ. evaluation based upon WINS doesn't
reveal the Real Goal of $$$$. same
with the NFL/MLB/Xgames/etc.

[Christopher Horn]

One of the problems with the "controlling for draft pick" argument is that it is hard to isolate the impact of the variable. $10 million of value from a $15 million quarterback leaves a team with $5 million less for a linebacker, but $10 million of quarterback value is also better than $5 million - and if the same team has a linebacker earning $2 million but delivering $7 million of value, the effect - in theory - is washed out, which may cancel the impact of losing $5 mil on the QB.

More broadly, it seems like NFL QB is a profession that validates the 10,000 hour rule as much as any other. The iconic example is Peyton Manning, an individual I don't know and for whom I have a ton of respect, but let's face it, based on his preparation, he must be a pretty geeky, studious fellow. In a championship winning sort of way.

Something like that nerdiness, or 10,000 hour-of-preparationness, must be the ticket to success for a QB in the NFL. Perhaps few other QBs prepare as much as Manning. But all successful ones must approach Manning's dedication, given the universality of the 10,000 hour rule.

So taking a classic example: when Jake Delhomme goes undrafted out of Louisiana-Lafayette in 1997, he's just one of dozens of undersized QBs of moderate physical gifts who is essentially overlooked by NFL scouts.

Based on the conclusions in Outliers, I infer from Delhomme's success that, if not quite to Peyton Manning's level of devotion, Delhomme probably is closer to a 10,000 hour QB preparer than the vast majority of small D1 QBs whose careers go nowhere after college.

But how does a scout determine the propensity for 10,000-hour-type preparation? Perhaps one of the linked articles discussed this - from here in the cheap seats, it seems nearly impossible to do.