Forensic analysis redux

I take it, from  many of the comments on my last post, that virtually no one bought my idea for loosing the forensic economists on sports records.  So let's try again. Hopefully round two will be a bit more convincing.

       It seems to me that there are two problems here. First, from everything we know it is very difficult for the drug police to catch cheaters. Bonds and Tim Montgomery and Marion Jones got caught because Trevor Graham ratted on them, not because the steriod police were on top of the problem. We have no idea how many other people out there are using chemicals that we simply don't yet have a test to detect.  Second--and here I strongly disagree with some readers--peformance enhancing drugs work. They confer an enormous advantage. They allow an athlete to train so much harder than he or she  otherwise could that they can turn mediocre athletes into very good athletes, and very good athletes into legends. "Game of Shadows" makes, I think, an overwhelming case that drugs allowed Bonds to essentially double his annual home run output, and turned Tim Montomgery from an also-ran into a world record holder.

     Given these two facts, what do you do? That's why I think you have to bring in the forensic economists.  Is this a perfect solution? Of course not. It does introduce a subjective element. It is also possible that we will occasionally see real genius and mistakenly call it cheating. But no administrative system is flawless. Juries send innocent people to jail all the time. But we put up with that because the alternative--no legal system--is a lot worse. What if--in sports like baseball and track and field and swimming--we had a record-review board. We assembled a panel of experts who reviewed the circumstances under which the record was set, physiological evidence from the athlete himself or herself, and statistical evidence about the plausibility of the performance. Beamon would pass. FloJo would not. Bonds would not--and nor, I would wager, would McGwire or Sosa. The point is that we have to do something, otherwise records will cease to mean anything at all.

     Incidentally, in  "Baseball Between The Numbers," by the Baseball Prospectus team, there is a very nice essay by Nate Silver doing exactly this: using forensic tools to try and gauge the extent of steriod cheating in major league baseball. It's worth a read. Silver concludes, interestingly, that the overall amount of cheating seems to be quite small, and confined largely (at this point) to mediocre players trying to get a little edge, not superstars trying to become mega-superstars. I find that very reassuring, not just because it says that Bonds is  an anomaly, but because it reminds us of just how useful statistical analysis can be.

Comments

[Kaan]

This may seem like a bit of an odd argument to make, but here goes: we should not consider sports-related drug use a problem. Perhaps it would be better to accept drug use as the baseline.

As you pointed out, we have no idea how many other people out there are using chemicals that we simply don't yet have a test to detect. And just as the officials are working hard to detect new drugs, the drug makers are producing newer, undetectable drugs. This raises an important question. If we had a test today that could accurately detect all drugs (in all sports), would it be fair for the athletes of today and tomorrow to compete against the records that were set during a time when (we assume) some number of athletes were using drugs, but we weren't testing them?

Drug use or not, technology has improved through the years. Golf balls fly better, bicycles weigh less, and most sports gear is stronger and better than ever before. Even if we could eliminate drug use entirely, old records would still be incomparable to new records. The equipment, techniques, and training have all improved.

In some sports, such as cycling, many people (including current and former pros) claim that most of the pros cycling field is doping anyway. It seems silly to penalize one rider here, another there, if they're all doing it.

I assume athletes have been using performance-enhancing drugs for longer than we know, and it seems impossible that the testing labs can ever catch up with the newest drugs. So why not consider performance-enhancing drugs the baseline?

[Vidar Masson]

I think any discussion on freakish world records must include Michael Johnsons amazing 200 meter world record (19.32 seconds).

[Max]

I wonder what a statistical analysis of older records might reveal? Like, for example, Maris's 61 HRs? Obviously, enhancement drugs were nowhere near as advanced or directly beneficial; but in Jim Bouton's Ball Four there's a lot of talk about "greenies" and amphetamines. Maris's record-breaking season saw him him almost twice as many homers as in any other season besides the following. I don't want to posthumously accuse Maris of cheating, but would an analysis of that record nullify it? How far back would we go if we were to institute such a panel of experts?

[conchis]

A couple of points.

(1) Trying to discover people who have cheated in attaining records seems to me to be a somewhat differenct prospect from detecting everyday, run-of-the-mill cheating, because the performance of your non-cheaters is also, by definition, rather statistically unlikely. I realise that's not fatal, but it does make the task rather more difficult. Even more so when you think that if enough people are doping, it won't get found out, because it won't be out of the ordinary. (You may also just change the game to one of doping just enough to appear normally-abnormal. Maybe that wouldl still count as a win. I don't know.)

(2) Loss functions have to feed into this. My gut says that its better to allow someone an unwarranted record, than to strip someone of a warranted one. Perhaps that doesn't make sense, when you consider that allowing an unwarrranted record is stripping someone else of a warranted one. But I think there's a qualitative difference between disqualifying someone and allowing them to be beaten unfairly.

Moreover, because sports records by their nature are statistically unlikely, when you draw a false positive, it's probably going to be one of the most "deserving" winners that feels the brunt. That sits uneasily with me.

I think it's also worth bearing in mind that, by and large, those found guilty of statistical cheating have very little chance of proving their innocence, if falsely accused. Winning more obviously wouldn't help, because you've argued that that's a sign of cheating. (Although that sort of requirement seems ripe for gaming: if you want to dope and keep your record, make it look like a lucky day, and shave your performace in other cases.) By contrast, there's always the possibility that we'll catch the real cheats in the future with better drug tests. (If you're worried it won't be detectable, save blood/urine samples in case better tests are developed later. you can always strip records later.)

[Adam]

So if all new records are reviewed for statistical plausibility, what will happen to athletes who are approaching record levels?

I think that knowing any record they set will be reviewed might change the athlete's incentives.

For example, a player who wants to set a record and has no problem using steroids to get to that level could plan his performances to pass the statitical analysis. You might say that these cheaters aren't that clever or patient, but I don't think evidence indicates cheaters aren't patient or clever.

Another example is an athlete who is improving honestly. If he is approaching record level performances, he will be aware that if his improvement doesn't pass statistical analysis, his record will be revoked. This will probably be a dis-incentive to that athlete's best performance. I don't think that this is an acceptable result.

In light of these two examples I don't see this solution as acceptable.

Perhaps the answer to the steroid problem is simpler though.

If we're serious about addressing the issue, I suggest we take a look at all suspected cheaters minus the sports in which actually testing is a universal policy. I am not convinced that testing has been used to its potential in many sports and organizations.

Furthermore, I see that in Major League Baseball, a player must be test positive on three seperate occasions to be banned from the game, and this is in light of a much stricter policy that is as old as November 2005.

[Overworm]

"there is a very nice essay by Nate Silver doing exactly this: using forensic tools to try and gauge the extent of steriod cheating in major league baseball."

That is a pertinent topic for a baseball GM who is trying to determine whether or not to offer a player a $100M contract. It's not such a pertinent topic for hardly anyone else.

I think it's a poor analogy to compare cheating in baseball to criminal acts. Society accepts, however begrudgingly for some of us, the possibility that a few innocent people may be imprisoned because the alternative of allowing thousands of violent people to roam the street is not an acceptible alternative.

On the other hand, when the alternative to using drug tests to catch steroid users is to falsely indict and punish innocent overachievers based on the subjective application of statistical trends, I think we've gone too far. Especially when the alternative simply results in shifting some random names around in a record book.

There's no societal greater good to be garnered by ruining the careers of athletic overachievers in an attempt to punish baseball cheaters. There is significant societal good generated by accidentally imprisoning a few innocent men along with the hundreds of thousands of guilty ones. There is a huge difference between the two.

Cheers,
Overworm

[Mark Winburn]

One difficulty in introducing a review of the stats is the allowance for anomalies. Go through a prolonged difficulty of, say, a divorce, and you could find yourself in either the hitting season of your career or the longest slump only to find the next year back to ho-hum normalcy.

Another is the very definition of "enhancing" drugs. Like Robin Williams (and Eddie Izzard before him) joked, marijuana isn't much of an enhancement (except for cancer treatment and glaucoma), but as a Tourette's sufferer, I take drugs that "normalize" me, allowing me to concentrate and excel at sports, while the same drug leaves others catatonic. Caffeine is allowed by many sports although a cyclist was reportedly tossed when after a sprint he -- through cups of coffee, NoDoze, suppositories, et al -- had the equivalent of some 40-50 cups of coffee in just 5 minutes. Sure, that's off the charts but the point is easy to see: where do you draw the line when each drug affects different people in different ways?

Physiologically speaking, Tim Howard, who has Tourette's, apparently manages to play for Man U without medication although the twitches associated with the condition are like having twice the twitch muscles of others. Ditto Jim Eisenreich. (The Baseball Hall of Fame talk about Jim's "handicap" but in the world of TS, one has to tread carefully and wonder if in some cases this isn't an advantage.) This added quickness could be "dumbed down" with medication, but are we to create a world of Harrison Bergerons to have a "fair" game? I say this because in the world of enhancing drugs, we're also faced with Harrison Bergerons ... in reverse: those with shortcomings are driven to be the best no different than those born with gifts -- better vision, cardio, old-fashioned prodigies. Larry Bird did it the honorable way, with hard work. But he was 6'9". I could work twice as hard but I'll never gain an inch. So, what are we doing? Barring the hard work, we're rewarding those in society who are, well, oddities.

On last point -- work ethic. Tiger Woods, Larry Bird are two who come to mind with peerless work ethic and a clean stream of urine. What strikes me as odd is the rumored increased work ethic of steroid users. "Why take them if you're not going to use them?" They're juiced, they work harder, they stop juicing, they still work hard. The stats peak, they taper off a bit, but in the end they've raised their level of the game, and be extension the game as a whole.

This is not to say I approve of or otherwise endorse performance-enhancing drugs. But the field is not level to begin with, either. The former we can ban, investigate, find a new drug, ban it. Rinse, repeat. And the battle will rage on forever like Iraq, always with a new twist, excuse, and mea culpa. In a similar vein (sorry) we must face the ups and downs of psychology and physiology and simply shrug and the inequities while applauding the gifts. So let's do that. Let's applaud the gifts, be grateful that they enrich the lives of others. As for the drug users, no, appointing a United States Handicapper General is not the answer. It's better to battle the drugs and their users as best we can but when it comes to the stats, let's embrace the magic. Did Sosa and McGuire juice? Sure. Did we let it happen? Sure. In so doing, we're all guilty. And I'm not endorsing collective sin when I say let's revel in it. After all, it's just a game.

[Chris]

It's sad, but I think maybe the best thing to do is to stop caring about sports. It's just a game after all.

Either that, or this plan:

1) keep urine and blood samples in escrow
2) instead of paying players, fund annuities for them (including endorsements -- all income)
3) if at a later date, the samples test positive, revoke their annuities.

[Jayakumar]

> 2) instead of paying players, fund annuities for them

This line of thought feels wrong to me. Would we do the same to Fortune-500 CEOs who are using "performance-enhancing drugs" to focus better on their business? Or I guess the proper analogue is "results-enhancing accounting"? Should their pay be put into annuities as well? Should we apply this same statistical analysis to businesses?

[fred]

"...otherwise records will cease to mean anything at all."

Records really don't mean much anyway, once comparisons are made outside of an athlete's contemporary peers. As has been pointed out numerous times, players in each era have different advantages. Those advantages are generally reflected in the record books. So-called "all time" records have very little meaning for all time.

What I find interesting about the Bonds' issue is how upset people are, and the drastic measures, as I must classify loosing the forensics economists on the record books, they're proposing to eliminate the "problem." I tend to think that most of the discontent with Bonds' steriod use is related to the fact that he appears to be a first-class jerk, to put it politely. Of course few people will admit that Bonds' character or off-field behavior has anything to do with their own outrage or assessment of the situation, but those who know a bit of social psychology, which must include you Mr. Gladwell, know better.

[rone]

The point is that we have to do something, otherwise records will cease to mean anything at all.

I think you're exaggerating here. Every record has a context, and the context must always be taken into account. Wilt Chamberlain's 100-point night wouldn't've happened without his teammates constantly feeding him, for example. Cy Young's 508 victories are unassailable because the game has changed.

What's important here isn't the records; it's the players' health. If it's evident that enhancer science advances far more quickly than detection science, then it is crucial that all sports determine what safe levels are and make them known. History shows that Prohibition didn't work, and that it can't work. Let's learn from that.

As for the records, they seem to be more of an American obsession. I'd bet that the average soccer fan doesn't know Pelé's career goals or Maradona's career caps. The important thing is the game. Everything else is just details. Steroids and post-steroid enhancers are bad because the anti-drug culture leads to their misuse and abuse, and thus destabilize the level playing field. That's what needs to be examined and fixed.

[Jonathan]

Would anyone attempt to break a record if it meant several weeks or months of suspicion?
Would Olympic medals be handed out only after an investigation into every record?
Would people still feel the same way about watching sports if they knew that any record-breaking performance would immediately arouse suspicion?

I'm reminded of the way that engagement in politics has fallen since the media assumed that anything a politician said was a lie and since scepticism became the norm.

I agree with the logic behind the argument here, but I can't see it having a positive effect. Instead, as someone mentioned, it'll encourage the idea that you can cheat your way towards a record, then lay off the drugs while you wait for the clinching performance.

[michael]

The assumption of your argument is that the steriod police want to catch athletes violating drug policies. Looking at track and field and the NFL, that assumption is false.

A better way to deter athletes from using steriods and other illegal drugs? Enforce the laws of the nation and not the policies of the sports. If Barry Bonds spent six months or a year in the California penal system because he possed and used steriods, then a lot of athletes would stop using the junk.

[Nancy Jane]

Why is the use of performance-enhancing drugs seen as cheating? As the person with Tourette's mentioned, we use drugs to correct problems that interfere with our performance -- I take antihistimines, which I believe are banned in the Olympics, so that I'm not affected by allergies.

The real problem is that most of the drugs in question are dangerous to the health of the people who take them. Banning them because they are dangerous is reasonable -- if hard to enforce -- and also protects the athletes who are not willing to risk their future health. It certainly isn't fair that people who are sensible about risks to their health are put at a competitive disadvantage.

But what happens when the science gets sophisticated enough that the drugs aren't dangerous to most people? Will they still be banned at that point? I rather suspect, given the money invovled, that they will eventually be required.

[Jay Matthews]

I'm reminding of the starting gun in a sprint race. This may have been mentioned in one of the other posts, but I didn't see it. Think of how the false start in a sprint is measured: even if a runner starts after the gun, if he or she starts so quickly that we can forensicly conclude that the gun was anticipated, it is declared a false start. So comparing performance to what is reasonably possible is not brand new or out of the question.

[Jay Matthews]

I'm reminded of the starting gun in a sprint race. This may have been mentioned in one of the other posts, but I didn't see it. Think of how the false start in a sprint is measured: even if a runner starts after the gun, if he or she starts so quickly that we can forensicly conclude that the gun was anticipated, it is declared a false start. So comparing performance to what is reasonably possible is not brand new or out of the question.

[Rob Meyer]

What would this say to children who are chasing a dream? If their performance is significantly better enough than their peers and those who came before them that it must have been obtained by cheating?

This turns -everyone- who turns in a statistically improbable performance into a suspect doesn't it? Bonds has only made himself a suspect through his own behavior. If his record was squeaky clean no one would be talking about this.

We should let suspicion fall to those who warrant it because of their actions. I realize that instances of sports figures way, way statistically over-performing that aren't drug assisted might be extremely rare, but that occasional reach beyond what people thought was possible is why most people watch sports.

[Ron]

Should we include vision correction to improve eyesight to better than 20/20 and "Tommy John" surgery in the absence of a prior injury in the same "negative" category as anabolic steroids? Both are unnatural, unnecessary and potentially risky.

[Phil Aaronson]

I remember watching the Women's Olympic triathalon (Sydney) with my wife on TV. A complete unknown ripped through the field on the last couple km of the run. Even the TV announcer was was doubting the performance. When it gets to that point, your sport is in deep deep trouble. My wife and I turned off the TV disgusted.

Forensic analysis of performance doesn't have to be some scary thing. Done right, it should be a confirmation of the obvious.

[John S]

I think something that has been largely ignored in this discussion is what a statistical analysis would mean about the effects of records. The entertainment value of sports-particularly records- is to see something amazing and unexpected. A forensic analysis of records would essentially define the realm of possibility in sports-thus tainting all records in advance. An athlete defying logic and probability should be rewarded and praised, not suspected and investigated. It is not a matter of "innocent until proven guilty"; it is a matter of declaring some accomplishments impossible in advance.
Kirk Gibson's home run was not statistically probable, Michael Jordan beating the Jazz with the flu is not statistically probable, Sandy Koufax's dominance for three years was not statistically probable.
It is not statistically probable that a team could go 86 years without winning a World Series.
And these are all things that make sports great. If we let forensic economists limit what we expect, we might as well stop playing and watching. If we won't let our expectations be exceeded, why bother?

[Lester Spence]

Let's say that the data on point shaving is right. Five out of every 100 games are likely fixed.

You cannot use that piece of data to do ANYTHING with a single game.

Similarly, using large N datasets to predict general tendencies in athletic performance can't be used (correctly) to say anything about a single performance--Jordan's 63 against the Celtics for example.

[was]

Biggest problem with your idea is it's too easy to game the system: Athletes will start gradually consuming illegially performance enhancements as a teenager, so there's no unusually large performance jump anywhere. By the time they're 20, they'll be fully juiced up, leaving nothing for your analysts to find.

[Zach]

(Cross posted from Baseball Primer)

At one time, I had hopes of creating a steroid suspicion index. Bonds's career takes off so abruptly I figured there was a fair chance of finding an unambiguous signal.

The trouble is, if you're not Bonds, normal season-to-season variation and the aging process makes it hard to identify the onset of steroids. I imagine it's even harder if the player starts using during the season, uses different types of steroids, etc.

Basically, I gave up after I couldn't find anything in McGwire, Palmeiro, or Pudge Rodriguez's careers that signalled the onset of steroids strongly enough for me to be happy. All of these guys subsequently had corroborating external evidence.

If Gladwell is serious about this, I'd suggest he take a long, hard look at McGwire's career. If he can look at McGwire's career and give firm dates for when McGwire was on PEDs, he will have done better than I could. Bear in mind that McGwire's production is expected to fluctuate broadly because he was injured so often.

Th other issue, particularly relevant with respect to McGwire, is how do you tell anything useful about a player who spent his entire career on PEDs? If you have any hopes at all of finding a PED signal, your best chances are to look as close as you can to the onset of use.

Game of Shadows actually gave a much better steroid suspicion index based on body mass, height, and body mass percentage. Assuming you could get accurate information, using that index would give you much better information than anything guerilla economists could provide.

[Zach]

Incidentally, although I haven't read the Prospectus book, I think you could profitably ask yourself whether their methods are sufficient to identify McGwire and Palmeiro's careers as being influenced by steroids. If they can't, you might want to doubt their conclusions.

[Chuck]

I am very wary of any idea that might be described as discouraging to humanity.

Human beings are often strivers, but we also have an inherent tendency towards fear, laziness and self-centeredness. As a result, many humans need an example to encourage them to strive.

The system you suggest would seem to put at risk those historical honest strivers who might serve as examples.

Furthermore, a system that is thought of as holding even the possibility of punishing honest strivers will be crushing to the human spirit, even if only to a small piece of that spirit.

I am not a sports fan, but even I can be inspired by those who reach new heights in physical accomplishment using nothing but their muscles, sinews, and determination.

I fear that a scheme such as you propose would lead to a homogenization of major sport that would temper humanity's potential for inspiration.