NBA Metrics Continued....

I've long been a fan of John Hollinger, who writes about basketball for espn.com, in large part because of Hollinger's statistical system for analyzing NBA players. Hollinger calls it PERs, and I like it chiefly because I'm in favor of any system that tries to improve on what I think are  our woefully inadequate intuitive judgments of basketball ability.

Now David Berri--whose book, "The Wages of Wins", I wrote about a few months back--has critiqued  Hollinger's methodology.

Berri's argument is quite simple. As those of you who have read "Wages of Wins" know, Berri's big problem with the way we judge pro basketball players is that we over-rate the importance of how many points a player scores, and vastly under-rate the importance of things like turnovers, rebounds, and shooting percentage.

Now comes Berri's critique of Hollinger: he says that Hollinger makes the same mistake. Here's the critical section:

In discussing the NBA Efficiency metric – which the NBA presents at its website – I argued that this measure fails to penalize inefficient shooting. The regression of wins on offensive and defensive efficiency reveals that shooting efficiency impacts outcomes in basketball. The ball does indeed have to go through the hoop for a team to be successful.

The same critique offered for NBA Efficiency also applies to Hollinger’s PERs, except the problem is even worse. Hollinger argues that each two point field goal made is worth about 1.65 points. A three point field goal made is worth 2.65 points. A missed field goal, though, costs a team 0.72 points.

Given these values, with a bit of math we can show that a player will break even on his two point field goal attempts if he hits on 30.4% of these shots. On three pointers the break-even point is 21.4%. If a player exceeds these thresholds, and virtually every NBA played does so with respect to two-point shots, the more he shoots the higher his value in PERs. So a player can be an inefficient scorer and simply inflate his value by taking a large number of shots.

I'd be interested to see how Hollinger replies to this.

As I recall from the last time I posted on Berri, some readers have a problem with Berri's conclusions, mostly because his system ends up highly valuing players like Ben Wallace and Dennis Rodman and Kevin Garnett and dismissing the value of players like Allen Iverson.  But the more Berri's fleshes out in arguments, the more convinced I become.

If you're a skeptic, I urge you to start reading Berri's blog.

One more point: one of the fascinating things about this argument is how similar it is to the argument currently going on in medicine about "clinical" versus "acturial" decision-making.  One study after another has demonstrated that in a number of critical diagnostic situations, the unaided judgment of most doctors is substantially inferior to a diagnosis made with the assistance of some kind of algorithm or decision-rule. Doctors don't like to admit this. But it happens to be true.

A lot of the huffing and puffing about Berri's ideas, it strikes me, is just basketball's version of the same defensiveness and close-mindedness.

Comments

[Petey]

"A lot of the huffing and puffing about Berri's ideas, it strikes me, is just basketball's version of the same defensiveness and close-mindedness."

Or, y'know, it could stem from some fundamental misunderstandings in Berri's worldview...

Saying Berri's metrics are better than Hollinger's metrics is like saying John McCain has a better plan for Iraq than Donald Rumsfeld does. The important thing to take away is that they're both wrong, not that one is 8% correct while the other is 12% correct.

[Me Myself and Eugene]

First of all: loved your spiel at the New Yorker Festival. Thank them, will ya, for streaming that. Or posting it. Or whatever it is that the kids are calling video on demand these days.

Second: I'm not buying the argument for efficiency, because I thinking they're tipping the scale in the wrong direction. Mr. S. O'Neil, for instance, is not a very efficient shooter, but he tips in a lot of his own missed shots. For that tip in, I don't think he gets a rebound credit, and he gets two points for two shots. His shooting percentage is artificially high, as he takes close in shots, but the bottom line is that no matter how many tips it takes, his team scored on that trip down the floor.

You could say the same for three point specialists. There's the three pointer that you take when you're open, and then there's the three pointer you take because you see that one of your guys has boxed out underneath and is therefor in good rebounding position, dictating you take a quick shot. The result of the missed three pointer might still be two points for your team, and in the long run that will balance your missed shot.

Think about hockey's plus / minuses. No matter how many assists, penatly minutes, whatever you get, it all boils down to: did your team score more than the other team when you were on the ice?

I think any measuring stick that takes a league-wide average and then applies it to a team's particular system will always be off by a few points. The key is, how does that player fit into the system? When we can measure that, then we're in business.

[Ben]

Malcolm,

Please read this great discussion and critique of the Wages of Wins system by Dan Rosenbaum, a leading member of the advanced basketball statistics field. It's a long thread, but it also includes at the end a discussion of this specific post.

http://sonicscentral.com/apbrmetrics/viewtopic.php?t=877

The specific discussion relating to PER is here:
http://sonicscentral.com/apbrmetrics/viewtopic.php?t=877&postdays=0&postorder=asc&start=151

[Dane Cao]

Trying to find a scientific measure of basketball abilities is like searching for holy grail to me.And the most important thing is,what's the point of chasing it?People watch bastetball games to have fun and if they think some guys rock,so be it.Why try to tease out rules out of everything?

[Dane Cao]

Trying to find a scientific measure of basketball abilities is like searching for holy grail to me.And the most important thing is,what's the point of chasing it?People watch bastetball games to have fun and if they think some guys rock,so be it.Why try to tease out rules out of everything?

[Petey]

"Why try to tease out rules out of everything?"

Because it's fun.

The problem is that, unlike with baseball players, reducing individual basketball players to metrics provides more noise than signal.

[Petey]

Malcolm,

I'm a Berri skeptic BECAUSE I read his blog.

I'd strongly second Ben's above recommendation that you check out Dan Rosenbaum's well-informed thoughts on Berri.

When you start to probe deeper into this stuff, you'll discover that many of Berri's claims are truly laughable.

And more broadly, you'll discover that in basketball metrics, there really is no "THE math". Hollinger's PER and Berri's WP are both arbitrary models based on arbitrary assumptions. Unlike baseball, the dynamics of basketball preclude the existence of even a vaguely "correct" model.

One crucial difference between Hollinger and Berri for me is that Hollinger doesn't make the type of insupportable claims for his model that Berri does. That should reflect negatively on Berri's overall credibility.

Berri is a marvelous publicist, but that doesn't mean you ought to take his pronouncements at face value. He's more than slightly clueless about both the intricacies and limitations of basketball metrics.

[Holger]

Berri's metric (as well as a lot of others) is that basketball is inherently difficult to measure. Position has a great impact on the statistics as have teammates. Basically the question is: If Ben Wallace shoots 60% from the field, then why isn't he getting 20 shots per game game rather than Rip Hamilton who shoots maybe 45%?

Because if Ben Wallace shot 20 shots per game he wouldn't shoot 60% anymore.

[JohnMcG]

I would be equally skeptical of the research if it concluded that Allen Iverson was far and away the best player in the NBA.

I think what gets me about a lot of these number-crunchers is the lack of humility. The Boston Red Sox win a World Series with a sabermetric savvy GM (and the second highest payroll in the major leagues by a healthy margin), and the statheads put out a book about how they invented a new way of winning. There was no book last year about how the White Sox invented a new way of winning last year, and I suspect there won't be one about the Cardinals this year, even though they won championships with even more meager payrolls.

And in this case, Berri crunches his numbers and concludes that the best player by far, is a player on a team that has missed the playoffs the last several years in the team sport in which an individual player can have the greatest impact. Rather than reconsider, he publishes a book about how everyone else is wrong and he's right.

It's probably true that we could benefit from a more scientific approach to evaluating players. It's probably also true that some resistance to it is not entirely rational. But it would be nice if the prophets of these methods displayed some grounding in common sense and actual wins and losses.

[R. Porter]

In my view, it's not just Berri et. al.'s basketball metrics that are off. I blogged (http://coyotesqrl.blogspot.com/2006/09/gift-of-good-data-analysis.html) a while back on their chapter on the impact of work stoppages on sports. I contend that their input data is garbage, because they only look at attendance itself and not all-media viewership. Now, it might be nigh-impossible to accurately measure the effect of stoppages on viewership due to the increase in entertainment choices and the whims of casual fans; however, completely ignoring that data paints a very incomplete picture.

[neil]

Could someone point me to the source for the claim that

"one study after another has demonstrated that in a number of critical diagnostic situations, the unaided judgement of most doctors is substantially inferior to a diagnosis made with the assistance of some kind of algorithm or decision-rule."

Did Malcolm write about this specifically?

[David Lewin]

I strongly recommed you read Dan Rosenbaum's critiques of Berri's work (at the site a previous commneter mentioned). Dan Rosenbaum is a professor of economics at UNC-Greensboro and consults with an NBA team. He makes two major points, one that Berri's metric does no better than NBA efficiency at predicting how teams do with given players on the court, and two, that Berri made a major fundamental error in the possessions formula he used to derive his system.

[Sam Craven]

It would be nice if people who post here actually watched basketball, knew the rules or looked for 5 seconds at the stats.

1. Me, Myself and Eugene:
SHAQ has a career 57.9% Field Goal percentage. If that isn't efficient I don't know what is...
Also, a tip in DOES count as a rebound. Sure he can't shoot far from the basket - but he doesn't.

2. Hockey's plus minus system can still hide a bad player if he is on a line with top performers. Also, +/- doesn't come into play on the powerplay, so someone who is a PP specialist won't have a great +/- no matter how many times he scores.

3. Holger: "If Ben Wallace shoots 60%". Thanks for making me laugh. Furthermore - Big Ben has NEVER shot more than 11 FG attempts in a game, probably because he has no offensive game whatsoever.

4. NBA basketball puts such a huge emphasis on points per game because that is what puts people in the stands. NCAA vs. NBA basketball really comes down to the emphasis on scoring. One of the many reasons why there aren't that many people who love NCAA & NBA basketball - most love one and hate/ignore the other...

[Petey]

Hollinger responds:

http://www.truehoop.com/leaguewide-issues-55650-john-hollinger-responds-to-david-berri-and-malcolm-gladwell.html

[Aaron]

neil,

There is a part about the use of algorithms in medical decision making in the book Blink.

It was about diagnosing heart attacks in Chicago and how having too much information actually hurts doctors' ability to diagnose patients correctly. Using an algorithm to decide which factors were most important, they were able to be a lot more efficient, both in terms of money and lives saved.

[DavidEzra]

Although a lot can be gained by stats, I'd be currious to know how much the "intangibles" factor in?

More specifically, I'm especially interested in an athlete's ability to take his fellow teammates to the next level because he (she) is on the floor. (According to Hollinger's metrics, this athlete's teammates would be "credited" for his/her "intangibles.")

It never ceases to amaze me how teams that have all the talent in the world (according to our "infallible stats") get schooled by teams made up of players that, at least on paper, should still be in their diapers.

(What I'm suggesting is that "stats" in and of themselves are not good / bad or right / wrong. But rather, they can become good / bad or right / wrong depending upon how they are interpreted. Would a player be arrogant enough to claim that his/her "statisical rating" was entirely attributed to him / herself? Would a manager "cut" a player because his/her "statistical rating" was less than desirable, but his "presence on the floor" in elevating fellow players more than compensated for his/her "biased metric?")

[Me Myself and Eugene]

"It would be nice if people who post here actually watched basketball, knew the rules or looked for 5 seconds at the stats."

It would be nice if more people could explain their points without attacking others. But if that was the case, I'd be in Canada.

Anyway, I got a good yuk out of your attempt to call Shak "efficient." If the NBA ever started calling offensive fouls and enforced the three second rule, then we'd see how efficient he really is. For crying out loud, his seven year old daughter fixed his free throw shooting for, what? about twelve games, but he couldn't stick with it for the whole season. Bend your knees, she said, and that by itself worked miracles, until he slipped back to his old habits.

[Dan Rosenbaum]

Malcolm,

I am puzzled as to why you have hitched your wagon to the Wages of Wins horse. Kevin Pelton, for example, is one of the best writers out there on basketball statistics, and I think he is well-respected by pretty much anyone who has come into contact with him.

Dave Berri, on the other hand, has made a new and interesting contribution to the basketball statistics field, and I recommend that people read his book. But Dave is no Kevin Pelton. Kevin brings a depth of understanding of baskeball statistics and an ability to cut to the heart of an issue that few people possess - even those of us with much more formal statistical training.

Now, I have laid out my arguments against Wins Produced in two threads at APBRmetrics.

http://sonicscentral.com/apbrmetrics/viewtopic.php?t=877

http://sonicscentral.com/apbrmetrics/viewtopic.php?t=1002

For those without the time to read these very long threads, here is a synopsis of my arguments.

Wins Produced is a metric that (a) professes to be regression-based, but is only marginally so, (b) misapplies its own logic when it derives (rather than estimates) its linear weights, (c) proposes explaining team wins as a barometer when ANY metric with a team adjustment (no matter how bizarre) would explain team wins just as well, and (d) only performs microscopically better than points per game at explaining how teams do when particular line-ups are on the floor.

I am not arguing here for points per game as a metric, but it just seems that if Wins Produced was worth so much attention, it would explain substantially more than points per game. And it's far from clear that it does.

Call me crazy, but I don't think that is too high of a bar to ask a metric to jump over.

And check out Kevin Pelton's writings. I don't think I have ever read anything of Kevin's where I haven't learned something. And I cannot say that about everything that I read.

[Petey]

"Wins Produced is a metric that ... proposes explaining team wins as a barometer when ANY metric with a team adjustment (no matter how bizarre) would explain team wins just as well"

That, I think, is the crucial point that Malcolm hasn't understood so far.

Berri is selling his system as based on some "true" weighting of statistical attributes, when in reality, his weighting is totally arbitrary.

[db]

I must object to the description of medical diagnosis. Decision aids help physicians make less mistakes (of discharging heart attack patients from the ER), but they do not make diagnoses. I will admit that the difference here is a bit semantic, but I think we need accuracy in this claim.

I have blogged on this issue today - 7 deadly causes of chest pain

[JC]

APBRmetrics crowd,

This attack is getting old, and it's become annoying. Scrap the old APBRmetrics thread, start over, and make your point...that is, if you think there is something worthwhile in there. You should be able to do so succinctly and without generalizing. Don't point to a thread. I've read it, and it's rambling, full of inaccuracies, and very long. Please pull out the critiques that you think are so damning (don't generalize) that Berri has not responded to and present them--don't just say X is wrong, prove it. Don't point to credentials. Berri has better academic credentials than Rosenbaum and also has done consulting work in the NBA. That doesn't mean that Berri's work ought to be taken at face value or accepted any more than we should accept Rosenbaum's opinion because of his credentials. The argument matters. If you want to fight a pointless credential war, you have already lost.

I am glad that Berri, et. al. has stirred debate, but the response of his critics has been so nasty and without substance that I believe now, more than I did when I first read the book, that his system is the best available. Berri has responded numerous times in public, and many more in private, yet the critics won't stop with the same old arguments. Is his system perfect? No, and I don't think he claims it is. Is it progress? Absolutely.

The internet is a great place for strangers to get together and hate someone, but let's stop pretending this critique is anything more than a reverse fan club. It's time to rise above that if you with to be taken seriously. If you think you have a valid critique, you should submit it to the Journal of Sports Economics, where it will be reviewed by anonymous referees.

[Dan Rosenbaum]

This is so repetitive. Time after time after time I have (publicly or privately) made an argument, presented actual empirical results to support the argument, and then I get the following response.

(a) XXX is a better academic economist than you are.

(b) Your arguments are too silly to respond to.

(c) Publish your ideas in a respected academic journal and we will pay attention to them.

Dave Berri, in essence, makes this same argument in his blog. This parallels what his private exhanges with me.

http://dberri.wordpress.com/2006/11/26/answering-a-critic/

I guess it is good that Dave is kind of responding, but this mostly seems to be an attack of my academic credentials.

Which is too bad. I would find a discussion of the ideas much more interesting.

[Aaron]

db,

In your blog post you call it a "diagnostic test". I guess it does seem like semantics to me.

Dan Rosenbaum,

Can you show me where in the blog post you refer to Dave Berri says any of the three things you mention? I really don't see him saying anything you accuse him of. (Maybe JC does, but not Berri.) Where does he say anything about credentials?

Also, he mentions that your critique seems to boil down to his model not accurately predicting yours. This seemed accurate to me. Care to respond?

[JC]

I did not make point A. I was countering the credentialing in this comment (http://gladwell.typepad.com/gladwellcom/2006/11/nba_metrics_con.html#comment-25664423) I then said the following:

"That doesn't mean that Berri's work ought to be taken at face value or accepted any more than we should accept Rosenbaum's opinion because of his credentials. The argument matters."

I never said point B. I asked that the critique be restated rather than being fed a generalization of a long and rambling chat room thread. The argument is very complex, and any worthwhile critique should be able to be expressed in a single coherent document. I've read through it, and read Berri's response, and I think Berri is right. Convince me otherwise.

As to point C. You are an academic economist. Berri, et. al. are academic economists. You disagree with an argument they have made. Isn't publishing a response in the arena where sports economists are supposed to converse, and vetted by a third party, a fair suggestion? If you don't want to, that's fine, but it seems awfully strange that you would refuse to do so.

[Dan Rosenbaum]

I think it useful to ask whether a player rating can explain how a team does when a particular player is on the court. And it would be better if we took out the effect of teammates and opponents.

I don't know what is so controversial or complicated about that. Yes, ratings can come from producing such a statistic, but that is hardly "my" model. And to be frank, the ratings from that model often can be pretty awful.

But that is beside the point. The important thing is that the statistic that meausures how the team does when a particular player is on the court, holding the effect of teammates and opponents constant, is not biased for or against PER, Wins Produced, points per game, or any other box score statistics-based metric we use. Yes, it is measured with error, but with large enough samples, that is not a big problem.

Because it is measured with error, it often is not a good player rating. But as a barometer of player ratings (and for lots of other uses), it seems pretty useful to me.

PER, Wins Produced, NBA Efficiency make all kinds of assumptions about how certain types of players affect (or don't affect) their teammates. With adjusted plus/minus statistics, we can test those assumptions empirically.

I guess I could write this up in a 20-page paper and see if it made any more sense to people than it does in a handful of paragraphs. But I am afraid that if I cannot get my point across in a handful of paragraphs, 20 pages probably is not going to help me that much.

At the very least, I learn much quicker whether or not what I am saying is being communicated well or poorly. And at the end of the day, I just am not sure there are that many academic economists who really care about this topic to this level of detail. There are, however, lots of non-academics who do.