« Medical Blogging | Main | The Single Greatest Advance in Human History »

The Numbers Game

Ben_3Ben Bryner -- In the days leading up to the election, we in the US are used to hearing a lot about traditional American concepts like apple pie and, of course, baseball (my apology up front if this post is irrelevant to you). So this op-ed by Oakland A's manager Billy Beane, former Speaker Newt Gingrich, and Sen. John Kerry that compares health care to major league baseball is in keeping with the theme. The main point of their article is that the health care system would benefit from a new statistics-heavy approach. Their analogy is "sabermetrics," which is the approach to baseball decision-making by relying on statistics, many of them complicated measures derived from several other data points. I don't understand all the statistics, but it's best known for enabling the managers of small-market teams to post wins out of proportion to their low payrolls. There's obviously a lot more to it (here or in the most famous book on the topic, Moneyball). It's an interesting idea, and I definitely think medicine needs more evidence-based care.

But one problem with this line of reasoning is that sabermetrics usually have something to do with runs or wins; when you get down to it, these are the only numbers that really matter in baseball. It seems like the same would be true in medicine, but you have to consider a lot of other things, too. When you're making a decision about a surgical procedure, you have to consider more than just survival, you have to consider recovery times, disability, etc. And while the costs in baseball are relatively simple to unravel, tracking down the true cost of a given procedure is incredibly difficult.

That's not to say this is impossible; plenty of people are working on these statistics (and I imagine the article annoyed some of them). It's not like we have these huge piles of statistics lying around and just need someone to devise new stats to interpret them; usually the problem is getting reliable health care data (or getting the money to collect that data). Collecting data about a baseball game is one thing; what they're suggesting is the equivalent of collecting data on everybody in the stands: How old is each spectator? How many nachos did each person eat? How many times did each person wave a big foam finger? It gets out of control fast.

Strike two is that the analogy breaks down when you consider the people involved. While the sabermetrics for a given player may be interesting to all sorts of people, managers would seem to be the only people who stand to gain any concrete benefit from calculating them. And as far as I can tell, there's no real equivalent to a manager in medicine. (The baseball itself that gets hit all around the park could be the med student, and the bases that get stomped on could be interns, but that's all I can see.) The doctor, who they argue would be the main beneficiary of these stats, often has plenty of useful statistics at his or her disposal. The problem is that they aren't making the decision; they're trying to help another person, the patient, make a decision, and that often is where problems occur. A patient doesn't necessarily understand the numbers and doesn't necessarily care, and the doctor can't explain them all in a fifteen-minute visit. Health care decisions aren't as simple as deciding when to steal a base or when to recruit a certain shortstop. A large portion of the decisions a doctor makes in a given day are affected by so many different factors that it seems impossible to ever design a study addressing that situation.

And it's not as if a numbers-driven approach has never been tried. I think the closest medical equivalent of a sabermetric is the quality-adjusted life year (QALY), which tries to quantify disease burden by assigning a year lived with a certain disease a value between 0 and 1 that represents the relative quality of life with that condition. Calculating them requires several assumptions, which means it can never truly mean the same thing to two different people, and when the state of Oregon tried to use QALYs to ration out Medicaid dollars it didn't work out. Basically, Oregon gathered a bunch of people together and asked them to calculate QALYs and similar values for various medical services. They ranked each medical service based on those values, and then the plan was to pay for the interventions from the top of the list working down until there was no more money left. But the list of approved and denied services seemed arbitrary, and Oregon was forced to shy away from many of the uncomfortable implications of this kind of decision-making (more here).

I'm not saying evidence or statistical analysis isn't a good idea. I'm all for taking good ideas about health care from other disciplines. One of the reasons that health care problems are so intractable is the lack of applicability of solutions from these other arenas, and that's something we have to overcome. But all the same, medical sabermetrics aren't the home run that Beane, Gingrich and Kerry are hoping for. Fundamental systemic changes need to be made to make the system safer, cheaper, and more fair. Stats will be critical in guiding those changes, but the stats themselves aren't the solution. And I’m open to other baseball-related suggestions: roving beverage vendors in the clinics, a seventh-inning stretch during a long operation, nacho cheese dispensers in the nurses’ stations –- I’m all for them. Play ball!

October 29, 2008 in Ben Bryner | Permalink

Comments

The comments to this entry are closed.