Pinker on "What the Dog Saw."

Steven Pinker reviewed my new book "What the Dog Saw," in the New York Times Book Review this past Sunday. I sent the following letter to the editor in response:

It is always a pleasure to be reviewed by someone as accomplished as Stephen Pinker, even if—in his comments on “What the Dog Saw” (Nov. 15)—he is unhappy with my spelling (rightly!) and with the fact that I have not joined him on the lonely ice floe of IQ fundamentalism. But since football has been on my mind these days, I do want to make one small observation about his comments.

  In one of my essays, I wrote that the position a quarterback is taken in the college draft is not a reliable indicator of his performance as a professional. That was based on the work of the academic economists David Berri and Rob Simmons, who, in a paper published the Journal of Productivity Analysis,  analyze forty years of National Football League data. Their conclusion was that the relation between aggregate quarterback performance and draft position was weak. Further, when they looked at per-play performance—in other words, when they adjusted for the fact that highly drafted quarterbacks are more likely to play more downs—they found that quarterbacks taken in positions 11 through 90 in the draft actually slightly outplay those more highly paid and lauded players taken in the draft’s top ten positions. I found this analysis fascinating. Pinker did not. This quarterback argument, he wrote, “is simply not true.”

    I wondered about the basis of Pinker’s conclusion, so I e-mailed him, asking if he could tell me where to find the scientific data that would set me straight. He very graciously wrote me back. He had three sources, he said. The first was Steve Sailer. Sailer, for the uninitiated, is a California blogger with a marketing background who is best known for his belief that black people are intellectually inferior to white people. Sailer’s “proof” of the connection between draft position and performance is, I’m sure Pinker would agree, crude: his key variable is how many times a player has been named to the Pro Bowl. Pinker’s second source was a blog post, based on four years of data, written by someone who runs a pre-employment testing company, who also failed to appreciate—as far as I can tell (the key part of the blog post is only a paragraph long)—the distinction between aggregate and per-play performance. Pinker’s third source was an article in the Columbia Journalism Review, prompted by my essay, that made an argument partly based on a link to a blog called “Niners Nation." I have enormous respect for Professor Pinker, and his description of me as “minor genius” made even my mother blush. But maybe on the question of subjects like quarterbacks, we should agree that our differences owe less to what can be found in the scientific literature than they do to what can be found on Google.

Comments

[B]

Wow, I can't help but feel that my respect and admiration of Dr. Pinker has diminished a little since he so blithely replied to your honest inquiry with, based on our common analysis, a quick google-search result.

[Theresa]

African said: "Malcolm you've just been bombed by a blog-full of racist Sailerites with nothing better to do with their free time."

Hey African! You must be one of those politically correct Gladwellians. Not so busy yourself today I see. ;-)

[Dan]

There may well have been flaws in Pinker's critique, but your letter to the editor does a poor job of showing them, Mr. Gladwell.

Whatever Sailer's views are on race and IQ--I'll have to take your word that they're as offensive as you claim--they bear no relevance to his claims (made, to great length, in the comments on this post) about football.

If you had a legitimate argument with which to respond to Prof. Pinker, I'd hope you'd have done so. Even someone who is completely ignorant of the meaning of an eigenvalue surely knows what an ad hominem is.

(As an aside, a few years ago, after reading your book "Blink," I sent you an email noting some logical discrepancies I saw within it. You politely responded, with a letter I very much appreciated, saying that you intended to clarify and correct them in a later edition. I'm disappointed that now, a few years and a booming public reputation down the road, you've apparently forgotten how to take honest criticism.)

[Marcus]

Steve Sailer's background or reputation is irrelevant here. The issue is that the numbers are both publicly available and obvious. Sailer simply crunched them, and anyone in this comment thread could do the same. There is clearly a very significant relationship between draft position and QB success in the NFL. That's just a fact, and Steve Sailer's racist reputation shouldn't get in the way of seeing a clear fact.

In the age of the internet and cheap computing power, it should no longer be acceptable to swallow a golly-gee-whiz statistic that is obviously cooked. Berri's per-play statistic is terrible from a statistical sense -- it would get him marked down in any introductory econometrics course because it deliberately selects and weights the data in order to distort it to prove the author's "counterintuitive" point. Gladwell is too trusting of his interview sources and doesn't apply enough due diligence himself.

[burger flipper]

Don't make us sick Megan McArdle on you again, Mal.

[Brown Bourne]

http://brownbourne.wordpress.com/2009/11/17/pinker-and-the-brain/

[african]

Hey Malcolm, it seems that you have an infestation of "Sailerites." Give it about a week or two and that'll clear right up.

[Robert Bloomfield]

Let me get this straight. Pinker says "it simply isn't true that there is no correlation between quarterback quality and draft ranking." You make fun of him for this. But the paper you cite says this in the abstract:

"We find only a weak correlation between teams’ evaluations on draft day and subsequent quarterback performance in the NFL."

So if you indeed said or implied that there was no correlation, you said something that simply isn't supported by the paper you claim to base your results on.

[tm]

It's apparent that aside from not having a copy editor to ferret out your glaring spelling errors, you don't know how drafts work. In the NFL, the draft order is generally determined by how well (or poorly) teams did in the prior season. Thus, really bad teams will be drafting first, and the super bowl champion near the end of the first round. Yes, there is the usual horse trading of draft picks that complicates things, but in general, teams that suck draft first.

That dynamic means bad teams are desperate for a big change to turn around their fortunes. Which typically means drafting a QB. Who will then be thrown into a situation that is by definition dysfunctional. Because the team is losing not (only) because they lack a competent QB, but because they lack other key personnel: the five guys whose job it is to give a QB time to throw the ball or a decent running back or a few good wide receivers. Or a general manager/owner that's incapable of filling those positions with competence.

To be surprised that a first round QB pick on an atrocious team can't put up the kind of numbers a QB who is brought into a decent team that carefully grooms him to step onto the field when the time is "right" is ludicrous. Of course, the poor first round pick is going to be sacked, intercepted and otherwise earning an outrageous salary to be pummeled each week on the field and off. It isn't indicative of scouting, rather it is more a management problem. NFL teams are no different than most businesses after a "quick fix" to what ails them.

Also, with scouts their value is not in identifying the obvious first round choices, but in uncovering the hidden talents, the next Brett Favre out of a sea of mediocre looking QB prospects.

There are a few maxims in football that are relevant here. First, that QBs get all the glory and all the blame, which your quarterback essay and ensuing discussion again proves true. Second, and more important to the overall success of a team, football is a game decided in "the trenches". That is, in most games, the least glamorous positions are key to the outcome of the game. And no position is more glamorous than quarterback.

Thanks to the mistakes found in this book, you'll be in a good position to draft a top copy editing prospect.

[Amy Chapman]

You and Pinker are engaged in the old nature nurture debate. I have been disappointed by your recent books because I believe you now start with a subconscious premise, that nuture is the more powerful force, then seek the proofs which sustain your beliefs. It's a common problem. You began, I believe, as someone with a more open mind. Society in general fantastically over estimates the role of nurture and underestimates that of nature. Pop culture favors the self-help model, even when it is inadequate. As a mother who has tried to influence her children all their lives, and a daughter whose father suffered from mental illness, I have come from painful experience to believe that our genetic destiny is greater than any of us would like to admit. Obviously I wish your research supported my pre-existing beliefs! But the best course would be if you could try to banish from your mind the desired outcome of your research so you might accept studies that contradict preconceived notions.

[Bob Roper]

Blacks are less intelligent then Caucasians and Asians are more intelligent then Blacks and Caucasians. On average of course. There are plenty of black people that are smarter then some Asians or White people The IQ data is out there. Why are people so afraid of it.

[Steve Sailer]

Through 2008, among quarterbacks drafted from 1980-1999, top ten draftees averaged 2,975 pass attempts in their careers. Quarterbacks drafted 11th to 100th averaged 1,470 attempts, a little less than half as much. And quarterbacks drafted 101st or higher averaged only 387 attempts.

So, Berri is more or less throwing away the lousier half of the sample of quarterbacks drafted 11th-100th (and totally ignoring all the quarterbacks drafted after 100) and comparing them to all the quarterbacks drafted in the top ten.

When you actually count everybody drafted, you get the following figures for career yardage (through 2008):

Drafted
Mean Yards Median Yards
Top 10 20,296 18,148
11-100 10,099 3,881
101+ 2,614 0

The differences between the mean and the median (50th percentile) point out that the higher drafted players tend to be safer bets. The quarterback at the 50th percentile among the top ten draftees of his year went on to have a fairly impressive NFL career, throwing for 18,148 yards. (The median top ten quarterback of 1980-1999 in career yardage was Jim McMahon, who led the Chicago Bears to the 1985 Super Bowl title.)

In contrast, the 50th percentile of the 11th to 100th picks of his year only accumulates 21% as much career yardage. The median quarterbacks of the 11-100 group are Mark Herrmann and Chuck Long.

And the 50th percentile of 101st plus picks never completes a pass in the NFL).

So, the top ten quarterbacks drafted in the Eighties and Nineties tended to be safer bets, which has its value. Investors want to both maximize return and minimize risk.

http://isteve.blogspot.com/2009/11/gladwell-strikes-back.html

[JD]

Re: Quarterback draft location. If you compared draft spots in groups of 10. e.g 1-10 vs. 11-20 or 31-40. Which group of 10 has the best performing pros. I would guess 1-10.

[Cedar]

Just because a book sells millions of copies does not make it science, or true (e.g. pseudoscience is actually a more profitable industry than science). Just because a person writes on a blog read by hundreds of people does not make it false.
Your unwillingness to engage in the substance of the argument proves Pinker's point. Issues like selection bias and the logic of statistical analysis are critical elements of science, and science training. Resorting to ad hominem attacks on the "bloggers," then oversimplifying their case for aggregate statistics, and the problems with only looking at the top ten choices shows that you don't have the statistical and methodological chops.
I have to agree with Pinker that what most disappoints me (and other social scientists I have spoken to) about your work is not the misspelling, but the disregard for the conventions of statistical and methodological reasoning when it doesn't suit the purposes of your narrative. Unfortunately for your readers, this results in a distorted view of science as populated by minor geniuses with interesting anecdotes, rather than a collection of data and evidence organized and interpreted according to agreed upon rules of methodological evaluation. These rules are very seldom sexy or interesting, and rarely come with anecdotes of their own, but they are critical for succeeding at understanding complex phenomena in the behavioral sciences.

[Ross]

I haven't read through the comments, but neither Pinker nor Gladwell seem to consider that all drafting teams are not created equal. A quarterback drafted in the high teens or especially the twenties of any round is very likely being drafted to a strong team. Moreover, if the team has finished with a record good enough to earn a late draft spot, AND that team still needs a quarterback (one of the most vital positions for team success), the likelihood of that quarterback being surrounded by considerably better-than-average NFL talent is high (as are the chances that the QB will not be immediately thrown into action). In that scenario, the QB's chance for success would certainly seem much higher than a more talented quarterback drafted to a team with a terrible offensive line or an incompetent receiving corps or no running game. This observation could certainly be incorporated into the models by considering not just draft position but quality of drafting team.

[Just an observation]

It looks like pinker/sailer have won this round.

DING DING

[dts]

Pierre's comment above is worth noting. Simple spelling mistakes happen, calling eigenvalues "Igon Values" is something different. It's like writing a history of British rock and referring to the influential band The Hoo. It's hard to believe Gladwell has any idea at all what he's talking about if he takes a *fundamental* technical term and spells it in a way that is totally, laughably, nonsensically wrong. This kind of mistake could only be made by someone who has zero familiarity with the concept.

[dts]

Pierre's comment above is worth noting. Simple spelling mistakes happen, calling eigenvalues "Igon Values" is something different. It's like writing a history of British rock and referring to the influential band The Hoo. It's hard to believe Gladwell has any idea at all what he's talking about if he takes a *fundamental* technical term and spells it in a way that is totally, laughably, nonsensically wrong. This kind of mistake could only be made by someone who has zero familiarity with the concept.

[Mark]

It surprises me that Gladwell has any fans left. Remember the question of hockey player birthdates? Well, it's not true of star players in the NHL. Remember when he told us someone hired a WNBA player to coach a 12-year-old girls basketball team and she found a way to exploit inefficiencies in their game? Amazing. And now he's parroting some bad analysis of NFL quarterbacks from Berri, who has embarrassed himself before with his own stupidity and arrogance. When will the madness stop? Gladwell's a good writer - why can't he be bothered to write about something that actually matters?

[Munroe Ross]

Yesterday I wrote a completely inoffensive comment encouraging Malcolm to take it in good humor and to learn about signal detection theory, yet my comment was deleted. Why? Just how sensitive is your skin, Mr Gladwell?

[Munroe Ross]

whoops... didn't realise there was a page 2 and page 3 to the comments. my apologies.

[disappointed in Gladwell]

Well, you have to give MG credit for one thing: he may be a talented-but-sloppy writer who knows little about quantitative analysis, but he is certainly good at attracting smart comments. This thread comprises one of the most thorough, decisive, comprehensive, well-reasoned, bullet-proof eviscerations of a writer's miscast arguments I've ever seen. It should be used as a case study in how to refute poor reasoning and bad writing.

[Couch scientist]

Igon, before dismissing sailer as a racist, why don't you revisit your earlier work which explained black atheletic success on greater genetic variability. I found that work to be racist, as it implied one race (which we know don't really exist) had a genetic propensity to succeed that other races don't. I don't write you off just because you're a racist.

[Dan]

@disappointed in Gladwell:

Gladwell is, I think, a very good writer. That's part of the problem.

I suspect the degree of specialization in the western education system (or professional sphere) is such that there are few who are both as compelling writers as Mr. Gladwell is and have the familiarity with statistics and social science that Professor Pinker has. As a result, the pop science shelves are populated by a mixture of abstruse, overly technical writing by professional scientists and easy to read, charming, and utterly convincing hokum by excellent writers who lack the faintest understanding of the scientific method.

That Gladwell is one of the more popular purveyors of bad science doesn't mean he's uniquely responsible for the phenomenon--it simply means he's particularly talented at it.

Gladwell's writing talent--his ability to engage the reader, to make even the most esoteric facts seem interesting, and the most mundane articles seem worthy dinner party conversation--is something I envy. But if I had to give up all my rationality or understanding of science--or, in a less favorable interpretation, simply give up any pretenses I might have of honesty--I don't think I'd find that a fair bargain.

[Steve Sailer]

Malcolm Gladwell makes more money than me. I'm jealous.