« Degree of Difficulty | Main | The Perfect and the Good »

Comments

Petey

"A lot of the huffing and puffing about Berri's ideas, it strikes me, is just basketball's version of the same defensiveness and close-mindedness."

Or, y'know, it could stem from some fundamental misunderstandings in Berri's worldview...

Saying Berri's metrics are better than Hollinger's metrics is like saying John McCain has a better plan for Iraq than Donald Rumsfeld does. The important thing to take away is that they're both wrong, not that one is 8% correct while the other is 12% correct.

Me Myself and Eugene

First of all: loved your spiel at the New Yorker Festival. Thank them, will ya, for streaming that. Or posting it. Or whatever it is that the kids are calling video on demand these days.

Second: I'm not buying the argument for efficiency, because I thinking they're tipping the scale in the wrong direction. Mr. S. O'Neil, for instance, is not a very efficient shooter, but he tips in a lot of his own missed shots. For that tip in, I don't think he gets a rebound credit, and he gets two points for two shots. His shooting percentage is artificially high, as he takes close in shots, but the bottom line is that no matter how many tips it takes, his team scored on that trip down the floor.

You could say the same for three point specialists. There's the three pointer that you take when you're open, and then there's the three pointer you take because you see that one of your guys has boxed out underneath and is therefor in good rebounding position, dictating you take a quick shot. The result of the missed three pointer might still be two points for your team, and in the long run that will balance your missed shot.

Think about hockey's plus / minuses. No matter how many assists, penatly minutes, whatever you get, it all boils down to: did your team score more than the other team when you were on the ice?

I think any measuring stick that takes a league-wide average and then applies it to a team's particular system will always be off by a few points. The key is, how does that player fit into the system? When we can measure that, then we're in business.

Ben

Malcolm,

Please read this great discussion and critique of the Wages of Wins system by Dan Rosenbaum, a leading member of the advanced basketball statistics field. It's a long thread, but it also includes at the end a discussion of this specific post.

http://sonicscentral.com/apbrmetrics/viewtopic.php?t=877

The specific discussion relating to PER is here:
http://sonicscentral.com/apbrmetrics/viewtopic.php?t=877&postdays=0&postorder=asc&start=151

Dane Cao

Trying to find a scientific measure of basketball abilities is like searching for holy grail to me.And the most important thing is,what's the point of chasing it?People watch bastetball games to have fun and if they think some guys rock,so be it.Why try to tease out rules out of everything?

Dane Cao

Trying to find a scientific measure of basketball abilities is like searching for holy grail to me.And the most important thing is,what's the point of chasing it?People watch bastetball games to have fun and if they think some guys rock,so be it.Why try to tease out rules out of everything?

Petey

"Why try to tease out rules out of everything?"

Because it's fun.

The problem is that, unlike with baseball players, reducing individual basketball players to metrics provides more noise than signal.

Petey

Malcolm,

I'm a Berri skeptic BECAUSE I read his blog.

I'd strongly second Ben's above recommendation that you check out Dan Rosenbaum's well-informed thoughts on Berri.

When you start to probe deeper into this stuff, you'll discover that many of Berri's claims are truly laughable.

And more broadly, you'll discover that in basketball metrics, there really is no "THE math". Hollinger's PER and Berri's WP are both arbitrary models based on arbitrary assumptions. Unlike baseball, the dynamics of basketball preclude the existence of even a vaguely "correct" model.

One crucial difference between Hollinger and Berri for me is that Hollinger doesn't make the type of insupportable claims for his model that Berri does. That should reflect negatively on Berri's overall credibility.

Berri is a marvelous publicist, but that doesn't mean you ought to take his pronouncements at face value. He's more than slightly clueless about both the intricacies and limitations of basketball metrics.

Holger

Berri's metric (as well as a lot of others) is that basketball is inherently difficult to measure. Position has a great impact on the statistics as have teammates. Basically the question is: If Ben Wallace shoots 60% from the field, then why isn't he getting 20 shots per game game rather than Rip Hamilton who shoots maybe 45%?

Because if Ben Wallace shot 20 shots per game he wouldn't shoot 60% anymore.

JohnMcG

I would be equally skeptical of the research if it concluded that Allen Iverson was far and away the best player in the NBA.

I think what gets me about a lot of these number-crunchers is the lack of humility. The Boston Red Sox win a World Series with a sabermetric savvy GM (and the second highest payroll in the major leagues by a healthy margin), and the statheads put out a book about how they invented a new way of winning. There was no book last year about how the White Sox invented a new way of winning last year, and I suspect there won't be one about the Cardinals this year, even though they won championships with even more meager payrolls.

And in this case, Berri crunches his numbers and concludes that the best player by far, is a player on a team that has missed the playoffs the last several years in the team sport in which an individual player can have the greatest impact. Rather than reconsider, he publishes a book about how everyone else is wrong and he's right.

It's probably true that we could benefit from a more scientific approach to evaluating players. It's probably also true that some resistance to it is not entirely rational. But it would be nice if the prophets of these methods displayed some grounding in common sense and actual wins and losses.

R. Porter

In my view, it's not just Berri et. al.'s basketball metrics that are off. I blogged (http://coyotesqrl.blogspot.com/2006/09/gift-of-good-data-analysis.html) a while back on their chapter on the impact of work stoppages on sports. I contend that their input data is garbage, because they only look at attendance itself and not all-media viewership. Now, it might be nigh-impossible to accurately measure the effect of stoppages on viewership due to the increase in entertainment choices and the whims of casual fans; however, completely ignoring that data paints a very incomplete picture.

neil

Could someone point me to the source for the claim that

"one study after another has demonstrated that in a number of critical diagnostic situations, the unaided judgement of most doctors is substantially inferior to a diagnosis made with the assistance of some kind of algorithm or decision-rule."

Did Malcolm write about this specifically?

David Lewin

I strongly recommed you read Dan Rosenbaum's critiques of Berri's work (at the site a previous commneter mentioned). Dan Rosenbaum is a professor of economics at UNC-Greensboro and consults with an NBA team. He makes two major points, one that Berri's metric does no better than NBA efficiency at predicting how teams do with given players on the court, and two, that Berri made a major fundamental error in the possessions formula he used to derive his system.

Sam Craven

It would be nice if people who post here actually watched basketball, knew the rules or looked for 5 seconds at the stats.

1. Me, Myself and Eugene:
SHAQ has a career 57.9% Field Goal percentage. If that isn't efficient I don't know what is...
Also, a tip in DOES count as a rebound. Sure he can't shoot far from the basket - but he doesn't.

2. Hockey's plus minus system can still hide a bad player if he is on a line with top performers. Also, +/- doesn't come into play on the powerplay, so someone who is a PP specialist won't have a great +/- no matter how many times he scores.

3. Holger: "If Ben Wallace shoots 60%". Thanks for making me laugh. Furthermore - Big Ben has NEVER shot more than 11 FG attempts in a game, probably because he has no offensive game whatsoever.

4. NBA basketball puts such a huge emphasis on points per game because that is what puts people in the stands. NCAA vs. NBA basketball really comes down to the emphasis on scoring. One of the many reasons why there aren't that many people who love NCAA & NBA basketball - most love one and hate/ignore the other...

Aaron

neil,

There is a part about the use of algorithms in medical decision making in the book Blink.

It was about diagnosing heart attacks in Chicago and how having too much information actually hurts doctors' ability to diagnose patients correctly. Using an algorithm to decide which factors were most important, they were able to be a lot more efficient, both in terms of money and lives saved.

DavidEzra

Although a lot can be gained by stats, I'd be currious to know how much the "intangibles" factor in?

More specifically, I'm especially interested in an athlete's ability to take his fellow teammates to the next level because he (she) is on the floor. (According to Hollinger's metrics, this athlete's teammates would be "credited" for his/her "intangibles.")

It never ceases to amaze me how teams that have all the talent in the world (according to our "infallible stats") get schooled by teams made up of players that, at least on paper, should still be in their diapers.

(What I'm suggesting is that "stats" in and of themselves are not good / bad or right / wrong. But rather, they can become good / bad or right / wrong depending upon how they are interpreted. Would a player be arrogant enough to claim that his/her "statisical rating" was entirely attributed to him / herself? Would a manager "cut" a player because his/her "statistical rating" was less than desirable, but his "presence on the floor" in elevating fellow players more than compensated for his/her "biased metric?")

Me Myself and Eugene

"It would be nice if people who post here actually watched basketball, knew the rules or looked for 5 seconds at the stats."

It would be nice if more people could explain their points without attacking others. But if that was the case, I'd be in Canada.

Anyway, I got a good yuk out of your attempt to call Shak "efficient." If the NBA ever started calling offensive fouls and enforced the three second rule, then we'd see how efficient he really is. For crying out loud, his seven year old daughter fixed his free throw shooting for, what? about twelve games, but he couldn't stick with it for the whole season. Bend your knees, she said, and that by itself worked miracles, until he slipped back to his old habits.

Dan Rosenbaum

Malcolm,

I am puzzled as to why you have hitched your wagon to the Wages of Wins horse. Kevin Pelton, for example, is one of the best writers out there on basketball statistics, and I think he is well-respected by pretty much anyone who has come into contact with him.

Dave Berri, on the other hand, has made a new and interesting contribution to the basketball statistics field, and I recommend that people read his book. But Dave is no Kevin Pelton. Kevin brings a depth of understanding of baskeball statistics and an ability to cut to the heart of an issue that few people possess - even those of us with much more formal statistical training.

Now, I have laid out my arguments against Wins Produced in two threads at APBRmetrics.

http://sonicscentral.com/apbrmetrics/viewtopic.php?t=877

http://sonicscentral.com/apbrmetrics/viewtopic.php?t=1002

For those without the time to read these very long threads, here is a synopsis of my arguments.

Wins Produced is a metric that (a) professes to be regression-based, but is only marginally so, (b) misapplies its own logic when it derives (rather than estimates) its linear weights, (c) proposes explaining team wins as a barometer when ANY metric with a team adjustment (no matter how bizarre) would explain team wins just as well, and (d) only performs microscopically better than points per game at explaining how teams do when particular line-ups are on the floor.

I am not arguing here for points per game as a metric, but it just seems that if Wins Produced was worth so much attention, it would explain substantially more than points per game. And it's far from clear that it does.

Call me crazy, but I don't think that is too high of a bar to ask a metric to jump over.

And check out Kevin Pelton's writings. I don't think I have ever read anything of Kevin's where I haven't learned something. And I cannot say that about everything that I read.

Petey

"Wins Produced is a metric that ... proposes explaining team wins as a barometer when ANY metric with a team adjustment (no matter how bizarre) would explain team wins just as well"

That, I think, is the crucial point that Malcolm hasn't understood so far.

Berri is selling his system as based on some "true" weighting of statistical attributes, when in reality, his weighting is totally arbitrary.

db

I must object to the description of medical diagnosis. Decision aids help physicians make less mistakes (of discharging heart attack patients from the ER), but they do not make diagnoses. I will admit that the difference here is a bit semantic, but I think we need accuracy in this claim.

I have blogged on this issue today - 7 deadly causes of chest pain

JC

APBRmetrics crowd,

This attack is getting old, and it's become annoying. Scrap the old APBRmetrics thread, start over, and make your point...that is, if you think there is something worthwhile in there. You should be able to do so succinctly and without generalizing. Don't point to a thread. I've read it, and it's rambling, full of inaccuracies, and very long. Please pull out the critiques that you think are so damning (don't generalize) that Berri has not responded to and present them--don't just say X is wrong, prove it. Don't point to credentials. Berri has better academic credentials than Rosenbaum and also has done consulting work in the NBA. That doesn't mean that Berri's work ought to be taken at face value or accepted any more than we should accept Rosenbaum's opinion because of his credentials. The argument matters. If you want to fight a pointless credential war, you have already lost.

I am glad that Berri, et. al. has stirred debate, but the response of his critics has been so nasty and without substance that I believe now, more than I did when I first read the book, that his system is the best available. Berri has responded numerous times in public, and many more in private, yet the critics won't stop with the same old arguments. Is his system perfect? No, and I don't think he claims it is. Is it progress? Absolutely.

The internet is a great place for strangers to get together and hate someone, but let's stop pretending this critique is anything more than a reverse fan club. It's time to rise above that if you with to be taken seriously. If you think you have a valid critique, you should submit it to the Journal of Sports Economics, where it will be reviewed by anonymous referees.

Dan Rosenbaum

This is so repetitive. Time after time after time I have (publicly or privately) made an argument, presented actual empirical results to support the argument, and then I get the following response.

(a) XXX is a better academic economist than you are.

(b) Your arguments are too silly to respond to.

(c) Publish your ideas in a respected academic journal and we will pay attention to them.

Dave Berri, in essence, makes this same argument in his blog. This parallels what his private exhanges with me.

http://dberri.wordpress.com/2006/11/26/answering-a-critic/

I guess it is good that Dave is kind of responding, but this mostly seems to be an attack of my academic credentials.

Which is too bad. I would find a discussion of the ideas much more interesting.

Aaron

db,

In your blog post you call it a "diagnostic test". I guess it does seem like semantics to me.

Dan Rosenbaum,

Can you show me where in the blog post you refer to Dave Berri says any of the three things you mention? I really don't see him saying anything you accuse him of. (Maybe JC does, but not Berri.) Where does he say anything about credentials?

Also, he mentions that your critique seems to boil down to his model not accurately predicting yours. This seemed accurate to me. Care to respond?

JC

I did not make point A. I was countering the credentialing in this comment (http://gladwell.typepad.com/gladwellcom/2006/11/nba_metrics_con.html#comment-25664423) I then said the following:

"That doesn't mean that Berri's work ought to be taken at face value or accepted any more than we should accept Rosenbaum's opinion because of his credentials. The argument matters."


I never said point B. I asked that the critique be restated rather than being fed a generalization of a long and rambling chat room thread. The argument is very complex, and any worthwhile critique should be able to be expressed in a single coherent document. I've read through it, and read Berri's response, and I think Berri is right. Convince me otherwise.

As to point C. You are an academic economist. Berri, et. al. are academic economists. You disagree with an argument they have made. Isn't publishing a response in the arena where sports economists are supposed to converse, and vetted by a third party, a fair suggestion? If you don't want to, that's fine, but it seems awfully strange that you would refuse to do so.

Dan Rosenbaum

I think it useful to ask whether a player rating can explain how a team does when a particular player is on the court. And it would be better if we took out the effect of teammates and opponents.

I don't know what is so controversial or complicated about that. Yes, ratings can come from producing such a statistic, but that is hardly "my" model. And to be frank, the ratings from that model often can be pretty awful.

But that is beside the point. The important thing is that the statistic that meausures how the team does when a particular player is on the court, holding the effect of teammates and opponents constant, is not biased for or against PER, Wins Produced, points per game, or any other box score statistics-based metric we use. Yes, it is measured with error, but with large enough samples, that is not a big problem.

Because it is measured with error, it often is not a good player rating. But as a barometer of player ratings (and for lots of other uses), it seems pretty useful to me.

PER, Wins Produced, NBA Efficiency make all kinds of assumptions about how certain types of players affect (or don't affect) their teammates. With adjusted plus/minus statistics, we can test those assumptions empirically.

I guess I could write this up in a 20-page paper and see if it made any more sense to people than it does in a handful of paragraphs. But I am afraid that if I cannot get my point across in a handful of paragraphs, 20 pages probably is not going to help me that much.

At the very least, I learn much quicker whether or not what I am saying is being communicated well or poorly. And at the end of the day, I just am not sure there are that many academic economists who really care about this topic to this level of detail. There are, however, lots of non-academics who do.

The comments to this entry are closed.

My Photo

Bio

  • I'm a writer for the New Yorker magazine, and the author of four books, "The Tipping Point: How Little Things Make a Big Difference", "Blink: The Power of Thinking Without Thinking" and "Outliers: The Story of Success." My latest book, "What the Dog Saw" is a compilation of stories published in The New Yorker. I was born in England, and raised in southwestern Ontario in Canada. Now I live in New York City.

    My great claim to fame is that I'm from the town where they invented the BlackBerry. My family also believes (with some justification) that we are distantly related to Colin Powell. I invite you to look closely at the photograph above and draw your own conclusions.

My Website

Books

  • What the Dog Saw

    buy from amazon

    Outliers

    buy from amazon

    buy from amazon UK

    Blink

    buy from amazon

    buy from amazon UK

    Tipping Point

    buy from amazon

Recent Articles

Blog powered by Typepad