Pinker on "What the Dog Saw."

Steven Pinker reviewed my new book "What the Dog Saw," in the New York Times Book Review this past Sunday. I sent the following letter to the editor in response:

It is always a pleasure to be reviewed by someone as accomplished as Stephen Pinker, even if—in his comments on “What the Dog Saw” (Nov. 15)—he is unhappy with my spelling (rightly!) and with the fact that I have not joined him on the lonely ice floe of IQ fundamentalism. But since football has been on my mind these days, I do want to make one small observation about his comments.

  In one of my essays, I wrote that the position a quarterback is taken in the college draft is not a reliable indicator of his performance as a professional. That was based on the work of the academic economists David Berri and Rob Simmons, who, in a paper published the Journal of Productivity Analysis,  analyze forty years of National Football League data. Their conclusion was that the relation between aggregate quarterback performance and draft position was weak. Further, when they looked at per-play performance—in other words, when they adjusted for the fact that highly drafted quarterbacks are more likely to play more downs—they found that quarterbacks taken in positions 11 through 90 in the draft actually slightly outplay those more highly paid and lauded players taken in the draft’s top ten positions. I found this analysis fascinating. Pinker did not. This quarterback argument, he wrote, “is simply not true.”

    I wondered about the basis of Pinker’s conclusion, so I e-mailed him, asking if he could tell me where to find the scientific data that would set me straight. He very graciously wrote me back. He had three sources, he said. The first was Steve Sailer. Sailer, for the uninitiated, is a California blogger with a marketing background who is best known for his belief that black people are intellectually inferior to white people. Sailer’s “proof” of the connection between draft position and performance is, I’m sure Pinker would agree, crude: his key variable is how many times a player has been named to the Pro Bowl. Pinker’s second source was a blog post, based on four years of data, written by someone who runs a pre-employment testing company, who also failed to appreciate—as far as I can tell (the key part of the blog post is only a paragraph long)—the distinction between aggregate and per-play performance. Pinker’s third source was an article in the Columbia Journalism Review, prompted by my essay, that made an argument partly based on a link to a blog called “Niners Nation." I have enormous respect for Professor Pinker, and his description of me as “minor genius” made even my mother blush. But maybe on the question of subjects like quarterbacks, we should agree that our differences owe less to what can be found in the scientific literature than they do to what can be found on Google.

Comments

[ali]

astonishing that Pinker would give a dope like Sailor the time of day; i'm sure he will be much embarrassed later.
no doubt Sailor's onslaughts (as per usual) on this blog will soon descend.... my condolences.

[Kristan]

Great rebuttal! And so polite, too.

Also, this is a ridiculously off-topic observation, but I love that you use giant font size. My eyes are so happy right now.

[steve]

Could you rebut Pinker's "IQ over 120 does matter" claim?

There's a lot more data about that than about NFL quarterbacks, and it matters a lot more.

http://infoproc.blogspot.com/2009/11/pinker-on-gladwell.html

[Michele Hush]

You and Pinker are two of my favorite brainiacs. I think it's time for a beer summit (although, were it me, it would be a martini summit).

[Alexis Madrigal]

Pinker's review was jocular, cruel and intended to embarrass instead of enlighten.

Part of it is: haters gon' hate. People like your work, so you've become an easy-as-Al-Gore target for those who deem popularity itself a crime.
The other part is: you get famous and simple mistakes people regularly make become inexcusable. Not to be too nice — after all, lack of humanity is a key attribute for an intellectual — but I want to say, "You know, Malcolm Gladwell goes out and comes into work overtired sometimes, too."

[Ed Cyzewski]

Reading that rebuttal makes me look forward to the book all the more. Very thorough and well-reasoned response that acknowledged some mistakes and challenged Pinker's own mistakes.

[china]

ah, thanks for that. i thought his review lacked intellectual depth but not snobbery.

[twitter.com/1day1brand]

Malcolm, every citizen should have to take a course on basic market research hygiene before they are allowed off their social network.

[twitter.com/bingisser]

I agree with your rebuttal, but am wondering if the statistics also take into account the fact that a player drafted higher will likely be playing for a team that is not as good. Without a good offensive line to block, receivers to catch, and run game to keep the defense guessing, a talented quarterback will not be putting up as big of numbers as they could on a better team.

[Bill]

Hilarious, Malcolm. Well played.

[Steve Sailer]

Dear Malcolm:

I know that you have more profitable things to do with your time than blog, but blogging etiquette is that you post a link to whatever you claim to be refuting.

My first analysis of your NFL quarterback claim is here:

http://isteve.blogspot.com/2009/01/can-you-predict-who-will-be-good-nfl.html

Your readers can see that for all quarterbacks drafted in the 1980s and 1990s, there was a correlation between spot in the draft and career yards, games played, seasons starting, and Pro Bowls.

The differences in NFL career totals between the quarterback taken at the top and bottom of the draft were typically an order of magnitude or more on all those measures.

Steve

[roy]

As Martin pointed out, just selecting the QBs taken in the first 10 picks skews the numbers. These are almost always bad teams, and likely don't currently have a decent QB. That leads to having these QBs starting their rookie years.

The NFL draft is seven rounds, and about half of the starting QBs in the NFL were first round picks.

[Steve Sailer]

Dear Malcolm:

Your fundamental problem, as you admitted a few years ago on your blog, is that you constantly get infatuated with some academic's new ideas. That openness to new ideas is your best feature, but it also leads you to constantly make a fool of yourself in public because you never ask the objects of your latest intellectual man-crushes any tough questions. You don't perform the obvious reality checks.

So, let's do a couple of obvious reality checks on David J. Berri's and Rob Simmons' line of thinking. (Readers can buy access to their latest paper "Catching a Draft" here: http://www.springerlink.com/content/k96t8116v8686350/)

Berri is using a very, very slippery approach.

First, he likes to compare quarterbacks picked in the top 10 draft picks in a year to those picked 11 to 50 or to 11 to 100. (He ignores the many picked below #100, where the accuracy of the draft becomes even more apparent.) But because the teams pick in inverse order of how well they did the previous season, those top ten draft picks are going to, on average, bad teams: the worst 10 teams in the league (leaving out trades of draft choices). In contrast, picks 11 to 50 or 11 to 100 will go, on average, to better teams. All else being equal, it’s easier to be a successful quarterback on a good team than a bad team, if they let you play.

And that's where Berri's other major trick comes in: he wants to measure success on a per play basis, rather than some more useful cumulative measure, such as Pro Bowl selections or career yardage or whatever.

There are obvious problems with measuring success on a per play basis. Obviously, if you’re selected lower in the draft and you turn out be as bad as expected, the coaches don’t let you get many plays.

For example, you can see the career statistics of all quarterbacks drafted since 1980 here: http://www.pro-football-reference.com/draft/QB-1980-now.htm

They’re arranged per draft order for each year. You’ll notice that a high proportion of high draft choices played a lot. Some of the low draft choices played a lot, but a lot of them barely played at all in the NFL: the team didn’t invest much in them, and when they proved in practice, unsurprisingly, to be less than NFL starting quality, they went to the bench or into insurance sales.

So, what you overlooked is that there’s a huge selection bias built into Berri’s measure of success. If a lower draft pick turns out in training camp and in weekly practices to be better than the NFL draft consensus (e.g., Tom Brady), eventually they let him play. But if he is a low draft pick and he doesn't prove to be better than the NFL thought he was, he doesn't get to play much, so he doesn't get much weight in Berri's analysis.

For example, the year Brady was picked 199th, Tee Martin was picked 163rd. In Tee's career, he completed 6 passes in 16 attempts for 69 yards, 0 touchdowns, and 1 interception. That's not terrible, but it's not good enough to be a regular NFL quarterback. In other words, Tee Martin proved to be exactly as mediocre as you would expect a 5th round draft choice to be.

So, Brady's statistics would weight much more heavily on a per play basis than Tee Martin's in Berri's analysis.

So, Berri's methodology rigs the conclusion ahead of time.

The problem with Berri's sleight-of-hand is that there are a lot more Tee Martins than Tom Bradys.

On the other hand, if some NFL team thinks you are such hot stuff that they'll burn a high draft choice and millions of dollars on you because they really, really need a new quarterback right now, well, then they often make you play a fair amount at a young age, even if you aren't ready for the NFL right now, or even if you aren't as good as they thought you were.

Moreover, when those lower drafted quarterbacks did play, they played typically under conditions more fruitful for success per play. Typically, they weren’t thrown in as 22-year-old rookie starters on lousy teams. In their younger years, they probably played against second-string defenses in the last minutes of blowouts. They got a chance to mature on the bench, so they didn't get a lot of plays until they were in their primes. Or the starter went down on a good team, and they stepped into the driver’s seat of a high-powered machine (like Matt Cassell taking over for Tom Brady last year, whose having a harder time this year in Kansas City where he can't just throw the ball in the general direction of Randy Moss.) All this pumps up their per play statistics.

Malcolm, the reason your reputation has plummeted in recent years as your net worth has risen is that you are too trusting of academics. You shouldn't be in awe of David J. Berri, Associate Professor of Economics at Southern Utah University in Cedar City. David J. Berri should be in awe of you, who is likely the highest paid print journalist in America. You should make Professor Berri prove his theories to you by subjecting his ideas to rigorous reality checks.

Steve

[MattMc]

Martin and Roy's points are well taken. Per play performance would amplify the distortion of higher draft picks going to worse teams, since those players would typically start younger. I would also suspect a selection bias, in that later round draft picks have to show really well in camp to even make the squad. On the other hand, the early picks are already signed to multi-year contracts, so they stick around regardless.

That said, your overall point is valid. It's not possible to predict performance without a lot of error. It's a fun data set to look at, and I enjoy the research, even if it's not quite as strong of a point as it seemed to be.

To me, it's not a question of sources and ad hominem barbs, it's simply about controlling for numerous variables.

[R.J. Lehmann]

I can think of absolutely no good reason to use a "per play" statistic over career statistics in evaluating whether a draft selection was a good one. Comparing the first ten picks of a draft (about one-third of the first round) to the next 80 picks (2/3rd of the first round, all of the second round, and just over half the third round) also just seems like a completely arbitrary dividing line. A quarterback taken anywhere in the first round would be a "high" pick by definition. One taken in rounds two or three would be considered a "project" -- one with a weakness of some kind, but since plenty of other productive players are still available in those rounds, still one that possesses significant potential. It's the quarterbacks taken toward the end of the draft that would considered total gambles, and it's an inordinately rare thing that such gambles pay off.

[Steve Sailer]

What Berri is doing, in effect, by using his "per-play" measure is comparing quarterbacks taken at the top of the draft (most of whom get a lot of plays in the NFL) to those taken lower in the draft who turned out to be surprisingly better than expected, and thus got a lot of plays. He's essentially leaving out of his analysis all those lower drafted quarterbacks who turned out to be as mediocre as expected and thus didn't get many plays in the NFL.

In other words, Berri's methodology is pre-rigged to produce the conclusion that Malcolm likes.

[Theresa]

Malcolm said: "Sailer, for the uninitiated, is a California blogger with a marketing background who is best known for his belief that black people are intellectually inferior to white people."

I've read a lot of Sailer's stuff on IQ, and I think I can safely say that nowhere does he claim that "black people are intellectually inferior to white people."

What he does point out is that pretty much all IQ research shows that blacks ON AVERAGE (i.e. as a GROUP) have a lower AVERAGE IQ than whites. (He also points out that whites have a lower average IQ than East Asians.)

Your statement makes it sound as though Steve thinks that ALL black have lower IQs than ALL whites. I'm sure that he does not think that -- nor does any psychometrician worth his salt. Simply because that is not what the evidence tells us.

[Eric]

Sailer makes 'The Bell Curve' type of arguments. To imply that because of this, his observation on quarterbacks, based on data he clearly presents, is incorrect, is irrelevant ad hominem.

[Pierre]

Writing about eigenvalues and referring to them as 'igon values' is not a simple spelling error. It clearly demonstrates that the author has never read a single page on the topic!

[M.E.]

Steve has clearly exerted quite a bit of thought on his rankings. Why not attempt a rebuttal? Calling him a racist - as if it matters in a NFL statistics discussion is weak.

[Steve Sailer]

Malcolm's statement in the New Yorker is quite uncompromising:

"This is the quarterback problem. There are certain jobs where almost nothing you can learn about candidates before they start predicts how they'll do once they're hired. ... The problem with picking quarterbacks is that [U. of Missouri quarterback] Chase Daniel's performance can't be predicted. The job he's being groomed for is so particular and specialized that there is no way to know who will succeed at it and who won't. In fact, Berri and Simmons found no connection between where a quarterback was taken in the draft—that is, how highly he was rated on the basis of his college performance—and how well he played in the pros."

I've looked at the 278 quarterbacks drafted in the 1980s and 1990s, and the correlations between draft pick and various career statistics:

Draft and Passing Yards: r = -0.48
Draft and Years Starting: r = -0.48
Draft and Games Played: r = -0.52
Draft and Pro Bowls: r = -0.33

So, the correlations between draft picks and Pro Bowls turn out to be weaker than many other measures, but it's still quite noticeable in real life.

[Jim]

ali nailed it: "no doubt Sailor's onslaughts (as per usual) on this blog will soon descend.... my condolences."

100% classy and predictable, mr. sailer - comment and blog away into the void...4 comments in an hour? wow.

[D]

Malcom, I think you should take up some of Sailer's public writings on IQ, crime (etc), and race if you're going to say such inflammatory things. For some reason I highly, highly doubt we'll be seeing you enter that territory. At best, we'll see you call ihim on "IQ fundamentalist" (oooooh!) as an excuse not to enter a realm where he has TONS of data on his side and you do not.

[NN]

As I've said on Steve's blog, the word misspelling can refer to two very different phenomena: typos and ignorance of subject matter. If you spelled "Des Moines" as "Des Miones", you've made a typo. If you spelled it as "Damoyn", you revealed to your audience something very interesting about you as a person.

Ignorance of mathematics is not a sin of course, but pontificating on topics that involve quantitative analysis while being THAT ignorant of math is kind of funny.

[M Stein]

"I have not joined him on the lonely ice floe of IQ fundamentalism"

Physics Professor Steve Hsu has highlighted the flawed nature of Gladwell's claim about iq over 120 being meaningless.

"What Pinker refers to as the major claim of Outliers: IQ above 120 doesn't matter, is easily shown to be false. Randomly selected eminent scientists have IQs much higher than 120 and also much higher than the average science PhD (120-130); math ability within the top percentile measured in childhood is predictive of future success in science and engineering; advanced education and a challenging career do not enhance adult IQs relative to childhood IQ.

So, accomplished scientists tend to have high IQs, and their IQs were already high before they became scientists -- the causality is clear. 10,000 hours of practice may be necessary but is certainly not sufficient to become a world class expert.

I recently remarked to a friend that many aspects of psychometrics which were well established by the 1950s now seem to have been completely forgotten due to political correctness. This leads to the jarring observation that recent social science articles (the kind that Gladwell is likely to cover) are sometimes completely wrong headed (even, contradicted by existing data of which the authors are unaware) whereas many 50 year old articles are clearly reasoned and correct. The data I cite in the links above comes from the Roe study of eminent scientists and the Terman longitudinal study of gifted individuals, both of which were conducted long ago, and the SMPY longitudinal study of mathematically precocious youth, which is ongoing. I've interacted with many social scientists whose worldview is inconsistent with the established results of these studies, of which they are unaware."

http://www.technologyreview.com/blog/post.aspx?bid=354&bpid=24410