Power Law Distributions

There are, I think, no end to the examples out here of the power-law problems I discussed in "Million-Dollar Murray" last week in the New Yorker. Medical care is obviously a great example. Here's another, from Richard Perez-Pena of the New York Times, who writes:

Reading [your] piece on the homeless took me back to 1996, when, as a reporter, I covered the collision of two commuter trains in New Jersey, in which three people died.  It quickly became clear that the engineer driving one of those trains - who was one of those killed - had run a red light.  New Jersey Transit disclosed that over his career, he had been cited four times for "serious" moving violations, including two for running red lights, which might be the most dangerous infraction of all. 

What we did not know—and they would not tell us—was whether four was a lot or a little.   So we demanded to see the safety records (names redacted) of all the engineers at New Jersey Transit, Metro-North Commuter Railroad, and Long Island Rail Road—about 1,000 in total. After months of wrangling, we got them.

How many engineers had more than four serious violations than the four this man had? Only one, who had five.
How many had the same number of serious violations, four, that this man had? Two.
How many had three serious violations? Two.
How many had run more red lights than this man had?  None.

The engineer who had probably caused this very rare fatal accident had what may have been the single worst safety record of 1,000 people driving commuter trains in the metropolitan area.  Only five were even in his dubious ballpark. The great majority of engineers had no safety violations at all.

And here's the kicker: about 8 percent of the engineers accounted for about half of all the violations. Yet not only were they allowed to continue working, they were under no particular scrutiny.  Worse still, these three railroads actually did more to police the engineers than most railroads, and more than was required by federal law.  The results made me think of the Christopher Commission report, which I had also covered, and the finding that 44 of 7,000 officers caused the bulk of the problems.

Comments

[Bill Hooker]

Welcome to the "blogosphere" (yeah, I know), and thank you for the "Million Dollar Murray" piece. I wonder whether what you are describing are actually power laws or some other kind of tail-heavy distribution (see, e.g., [http://www.cscs.umich.edu/~crshalizi/weblog/232.html] or [http://www.pnas.org/cgi/content/abstract/79/10/3380] , but don't ask me to explain the math!), but I suspect it may not matter for practical purposes as much as the general shape of the distribution.

[clarke ching]

Hi Malcolm,

Glad to see you are blogging.

This comment isn't related to the blog entry but I couldn't find your email address.

I have a suggestion: one your blog could you make your paragraphs shorter? It makes them much easier to read online - compared to when on paper.

Cheers
Clarke Ching
a Kiwi in Scotland

[Bill Hooker]

P.S. if you leave html turned off in your comments, how about a warning notice at the top of the comment form?

[David Sifry]

Malcolm,

I'm a huge fan, it is great to read your blog. "Tipping Point" and the essays therin were a big reason why I started Technorati and built it the way I did - to see if I could test the theories you described so eloquently and also figure out what else was going on in this massive open agora known as the Blogosphere.

Anyway, I just wanted to contact you and share my thanks. If you're ever in San Francisco, I'd love to give you a tour of Technorati and introduce you to the folks here...

Dave (dsifry AT technorati DOT com)

[Peter]

Train operators on the New York subway who run red lights face much more serious discipline. If a T/O is still in the probationary period when the violation occurs, he or she is likely to be fired; if the T/O is past the probationary period, one violation usually results in a period of suspension without pay, while the second one is a firing offense.
It's worth noting that the commuter RR's are under the jurisdiction of the Federal Railway Administration while the subway, being non-interstate, is not. So here is a situation in which federal regulations are actually more leinent than an unregulated entity's internal disciplinary rules. I suppose there's a lesson in that.

[Arnie McKinnis]

So much for the common notion of 80/20 - it seems the real rule should be 5/95

Welcome to the blogsphere!!

[The Wordyeti]

I'm not sure what you're struggling towards here; I think that it's something akin to the Bell Curve theory of organizational effectiveness - which states that there are a couple of super-performers on the high end, that know and do their own jobs and everyone else's.

Then there are the great mass in the middle, just kinda punching the clock and getting through the day.

And at the bottom are the screw-ups, known in the police departments here in Southern California as "slugs" ... these are the guys who create 90% of the problems.

BTW - as you might have surmised from my post, yes, I work in&around law enforcement here in L.A., mostly trying to help the 50 or so Police Chiefs in Los Angeles County figure out what to do about the slugs. It is not as easy as the movies and TV shows would have you believe. There are all kinds of books/seminars on the subject, with names like "Managing the Marginal Employee" and whole departments of government/industry have had to be set up (that wonderful Orwellian phrase "Risk Management") to figure out how to keep the people you want to do the job from blowing a fuse and burning down the goddam house instead.

Anyway - it's great to see that you've put up a blog - I've used the stuff you wrote in "Blink" about Abner Diallo and the 47 shots several times - and in fact, if you look at the recent OIR report on the LA Sheriffs infamous Compton shooting of a couple months back, you'll see that many of teh same factors that led to the NYPD capping an innocent man were at work again here. Plus ca change...

[Phil Gerbyshak]

Welcome to the blogosphere Malcolm! We're glad to have you as part of the community, and if your blog is anything like your articles and books, your blog will be GREAT!

[Account Deleted]

Dave LaF: You wrote, "I'm not sure what you're struggling towards here; I think that it's something akin to the Bell Curve theory of organizational effectiveness - which states that there are a couple of super-performers on the high end, that know and do their own jobs and everyone else's." I think that's precisely what he's not saying.

In the Bell Curve, you have two ends and general tapering to the worst or best. In fact, the power distribution shows that it's not gentle. There's extreme failure, then very moderate underperformance, and then general competence. Extreme failure is a very small number, moderate underperformance higer, and then competence the largest. A Bell Curve predicts a large amount of competence (or average behavior) with a relative small standard deviation. Power Law predicts enormous standard deviation for a very small number.

Malcolm, what about extreme success? Not mentioned in the article, but it's part of David Sifry's recent State of the Blogosphere report over at Technorati. The top blogs achieve enormously higher links-in.

[Lance Tracey]

Malcolm, hello, being profiled with long hair or following hush puppy sales don't compare to you entering this community. Hearty welcome to you for hanging out. LT
You'll never see feedback like this (not mine) ever.

[Lance Tracey]

Malcolm, hello, being profiled with long hair or following hush puppy sales don't compare to you entering this community. Hearty welcome to you for hanging out. LT
You'll never see feedback like this (not mine) ever.

[Lance Tracey]

Malcolm, hello, being profiled with long hair or following hush puppy sales don't compare to you entering this community. Hearty welcome to you for hanging out. LT
You'll never see feedback like this (not mine) ever.

[Lance Tracey]

Malcolm, hello, being profiled with long hair or following hush puppy sales don't compare to you entering this community. Hearty welcome to you for hanging out. LT
You'll never see feedback like this (not mine) ever.

[Nancy Lebovitz]

I'm delighted to see that you've got a blog.

OK, you don't want Million Dollar Murrays and you can't afford to give everyone an apartment. What's a sensible policy? I've
been trying to figure it out and I don't have an obvious answer. If a homeless person's medical bills go above some amount, they get an apartment? Maybe an apartment for anyone who's been on the streets for at least five years?

[curmudgeon]

Correct me if I'm wrong on the math, but if you fire these guys, you end up with the same situation, at a lower frequency. If you before you had one accident every 5 years, now you'll have one every 10, and someone involved will have a poor record. If you flunk out the bottom part of the curve, you'll still have a similar curve afterward.

The trick is picking the spot on the curve that appropriately protects the public. If you're a pilot, any significant transgression will end your career. That makes sense in jobs where public safety and confidence is at issue, perhaps including train engineers.

But in addition to potentially arbitrarily ending the career of someone who is qualified and trained, it also means higher salaries to retain people in a less forgiving job, more training, shorter work hours, and retiring them at 60.

And if inappropriately applied (forced retirement at 60?) where does this approach become profiling, the subject of another recent article?

[Ben Hyde]

Hope you find the blog a fun add-on.

Something I wrote about your last two peices: http://enthusiasm.cozy.org/archives/2006/02/problem-cases/

These power-law distributions, their everywhere and it's facinating how hard it is to integrate that realization into behavior and policy.

[Bill Seitz]

I drew the opposite, and more optimistic, conclusion from your article.

You find it disappointing that a blue-ribbon panel finds such simple narrow causes (compliance, a few hard-cases) of big problems.

I think that's a *good* thing, it means maybe a lot of problems (involving human behavior) that we see as big problems can really be attacked by focusing on a small number of hard-cases. They may be hard to solve, but they're easier to study, easier to experiment with, easier to tailor solutions to. And you don't need to start a new bloody government agency.

http://webseitz.fluxent.com/wiki/z2006-02-25-GladwellPowerLawPublicPolicy

[paul francis]

Hi Malcolm - great to see you have started your own blog. "The Tipping Point" was the inspiration for starting our business - Zealot. I'm curious. What was your "Tipping Point" in starting your blog (after resisting for so long)?

[Avi Solomon]

Do you know about Pareto's Law?

[Charles Tran]

Wow, that was a very powerful story. Your writing skills are amazing and I'm sure I will continue to enjoy your blog. I loved both Tipping Point and Blink.

That being said, I hope this blog serves as a catalyst for your next book/endeavor.

Fan #10348940928342892891

[zp]

hi all. i wanted to voice my thanks to gladwell for inviting comments. i'm a fan and critic of the new yorker and i enjoy alex ross's blog a lot, but i'm always sad the comments are closed. my thoughts on the power law and/or homelessness policies article are at my blog, but the comments here have been very interesting . . .

[Amy Stewart]

I think there's an interesting convergence between this story and "Blink." I once worked at a place where we had a very scary, dangerous incident involving a worker who'd gone over the edge. As we were meeting later to talk about how to prevent such things in the future--and considering broad, cumbersome policies that would not really address the problem--I said, "Come on, how many of you are really surprised that she was the one to lose it? We all know off the top of our heads who the loose cannons are around here. At any given time, you could line all our employees up from most at-risk to least at-risk, and we'd all pretty much agree on the ranking." I was trying to make the point that we all knew who the troublemakers were without any fancy reporting systems or sophisticated analysis, and we should focus on them, not everybody else.

[Tobi Keefe]

I loved the article - it occurs to me there's another group this law fits - physicians/surgeons sued for malpractice. The studies seem to show there is a small group that is responsible for most of the big problems. The fact that this group is not monitored effectively contributes to a big problem.

[Jim Caserta]

There is some logic and math behind taking a normal distribution and turning it into a power law. I think it is how you define what you're looking at. If you take a "Normal" group being scored on a task from 1 to 5, and then say that any score below 2 yields a terrible result, and the closer you are to a score of one, the more likely you are to get the terrible result, that should give something close to a power law. It's also comparing the presence of a low-probability event, to a complete absence. There is going to be a difference between two cops who have no excessive force complaints, perhaps a large difference. The same logic can be applied to good outcomes also - there are a small number of scientists/engineers that account for a majority of the major advances.

Great article.

[Shahar Even-Dar Mandel]

First of all, it's great to find you decided to have a blog.
As for the power-law distribution, I must say I was really puzzled since I find your examples to be perfect examples of normal distributions, with some artificial, but probably insignificant cutoff at one end. Lacking the full data I cannot draw a curve but I'm rather sure that examining the number of traffic violations you'll find a narrow normal distribution centerd a bit above zero, however, since normal distribution gives some probability to any value, you will find, with very low probability drivers with 4 traffic violation or more (the same applies to the LA cops, homelessness might be a bit different). The artificial cutoff comes from the fact that while the mathematical gaussian of normal distribution does stretch out to negative values as well there are no drivers with less than zero traffic violations. The fact that those few drivers at the tail are responsible for most of the trouble does not, in any way, change the disribution of traffic violations.
As for the homeless, they are, to begin with, not really a diverse group, so you might expect to see another distribution. However, looking at some home-ownership distribution you'll probably find a perfectly normal, and rather broad distribution, centered somewhere between ownership of a house and renting one with practically infinite tails having real-estate moguls at one end and Murray at the other.
What really troubled me when reading Million Dollar Murray was the notion that you consider homelessness as a problem mainly due to Murray and his similars, due to "aggressive panhandling" and their cost to the public, whereas, at least from some point of view, the problem of homelessness is, first and foremost,a problem of the homeless. A person who uses shelters sporadically, and for short periods, probably manages to find some temporary jobs in between and/or sleep at some relative's house. Taking Murray off the street will probably make it much easier for you and me to walk the streets with less fear and guilt, it might be a solution to what you think of the the "homelessness problem", but it does not solve the problem of the homeless.