gladwell.com

The Perfect and the Good

I wrote a piece for the The New Yorker a few weeks ago about a group of people who have created a neural network that predicts (or tries to predict) the box office of movies from their scripts. (It's not up on my site yet, but will be soon).

The piece drew all kinds of interesting responses, a handful of which pointed out obvious imperfections in the system. Those criticisms were entirely accurate. But they were also, I think, in some way beside the point, because no decision rule or algorithm or prediction system is ever perfect. The test of these kinds of decision aids is simply whether--in most cases for most people--they improve the quality of decision-making. They can't be perfect. But they can be good.

In "Blink," for instance, I wrote about the use of a decision tree at Cook County Hospital in Chicago to help diagnose chest pain. Lee Goldman, the physican who devised the chest pain decision rule, says very clearly that he thinks that there are individual doctors here and there who can make better decisions without it. But nonetheless Goldman's work has saved lots and lot of lives and millions and miillions of dollars because it improves the quality of the average decision.

Is the average movie executive better off with a neural network for analyzing scripts than without it? My guess is yes. That's why I wrote the piece. I think that one of the most important changes we're going to see in lots of professions over the next few years is the emergence of tools that close the gap between the middle and the top--that allow the decision-making who is merely competent to avoid his errors to be reach the level of good.

I think the same perspective should be applied to the basketball algorithms I've been writing about. It is easy to point out the ways in which either Hollinger's system or Berri's system fail to completely reflect the reality of what happens on the basketball court. But of course they are imperfect: neither Berri or Hollinger would ever claim that they are not. The issue is--are we better off using them to assist decision-making that we are making entirely judgements about basketball players using conventional metrics? Here I think  the answer is a resounding yes. (Keep in mind that I live in New York City and have had to watch Mr. Thomas bungled his way toward disaster. I would think that.)

And the reason that lots of smart people, like Berri and Hollinger and others, spend so much time arguing back and forth about different variations on these algorithms, is that every little tweak raises the quality of decision-making in the middle part of the curve just a little bit higher. That's a pretty noble goal.

That said, here are the latest updates on the Hollinger-Berri back and forth. And remember. I don't think this is a question of one of them being wrong and the other right. They are both right. It's just that one of them may be a little more right than the other.

Here we go. First Hollinger's response, courtesy of truehoop.com, (an excellent site by the way.)

And then. Berri's response.

NBA Metrics Continued....

I've long been a fan of John Hollinger, who writes about basketball for espn.com, in large part because of Hollinger's statistical system for analyzing NBA players. Hollinger calls it PERs, and I like it chiefly because I'm in favor of any system that tries to improve on what I think are  our woefully inadequate intuitive judgments of basketball ability.

Now David Berri--whose book, "The Wages of Wins", I wrote about a few months back--has critiqued  Hollinger's methodology.

Berri's argument is quite simple. As those of you who have read "Wages of Wins" know, Berri's big problem with the way we judge pro basketball players is that we over-rate the importance of how many points a player scores, and vastly under-rate the importance of things like turnovers, rebounds, and shooting percentage.

Now comes Berri's critique of Hollinger: he says that Hollinger makes the same mistake. Here's the critical section:

In discussing the NBA Efficiency metric – which the NBA presents at its website – I argued that this measure fails to penalize inefficient shooting. The regression of wins on offensive and defensive efficiency reveals that shooting efficiency impacts outcomes in basketball. The ball does indeed have to go through the hoop for a team to be successful.

The same critique offered for NBA Efficiency also applies to Hollinger’s PERs, except the problem is even worse. Hollinger argues that each two point field goal made is worth about 1.65 points. A three point field goal made is worth 2.65 points. A missed field goal, though, costs a team 0.72 points.

Given these values, with a bit of math we can show that a player will break even on his two point field goal attempts if he hits on 30.4% of these shots. On three pointers the break-even point is 21.4%. If a player exceeds these thresholds, and virtually every NBA played does so with respect to two-point shots, the more he shoots the higher his value in PERs. So a player can be an inefficient scorer and simply inflate his value by taking a large number of shots.

I'd be interested to see how Hollinger replies to this.

As I recall from the last time I posted on Berri, some readers have a problem with Berri's conclusions, mostly because his system ends up highly valuing players like Ben Wallace and Dennis Rodman and Kevin Garnett and dismissing the value of players like Allen Iverson.  But the more Berri's fleshes out in arguments, the more convinced I become.

If you're a skeptic, I urge you to start reading Berri's blog.

One more point: one of the fascinating things about this argument is how similar it is to the argument currently going on in medicine about "clinical" versus "acturial" decision-making.  One study after another has demonstrated that in a number of critical diagnostic situations, the unaided judgment of most doctors is substantially inferior to a diagnosis made with the assistance of some kind of algorithm or decision-rule. Doctors don't like to admit this. But it happens to be true.

A lot of the huffing and puffing about Berri's ideas, it strikes me, is just basketball's version of the same defensiveness and close-mindedness.