Most medical studies are wrong

“Science is a noble endeavor, but it’s also a low-yield endeavor,” Dr. John Ioannidis told The Atlantic recently.

It may be the truest statement yet made on medical research.

It’s a story flying around the medical community today, although it’s based on a single five-year old study, from a team of Greek researchers headed by Ioannidis, titled simply Why Most Published Research Findings are False.

So why is it news now? Possibly because Ioannidis became division chief of the Stanford Prevention Research Center in September. The picture is from the press release.

What Ioannnidis and his Greek team did was a statistical analysis of major medical research papers, finding out that many of the most popular had results that were later reversed after further study.

This goes beyond the placebo effect we talked about last November. It turns out that critics of medical research are right:

• The smaller the study the more likely it’s wrong.
• The smaller the effect seen in a study, the more likely it’s wrong.
• The more variables tested in a study the more likely it’s wrong.
• The more flexible designs, definitions, and outcomes are, the more likely it’s wrong.
• The greater the financial interest in being right the more likely it’s wrong.
• The hotter the field of research, the more likely it’s wrong.

Much of this is stuff your earth science teacher should have told you in junior high school. Narrow the variables. Narrow the scope. Tighten the design so only what you’re testing varies. Take your biases out of the equation.

Improving the scientific method won’t be easy. Biases like money, fame, and power are enormous. Ioannidis’ prescription is to not fall for any single study, to accept that being wrong is not a bad thing, and to have many people re-test any conclusion before acting on it.

This won’t result in perfection. Many, many studies were done on antidepressants like Zoloft, and millions of prescriptions were written, before it was found that in most cases it was no better than a placebo. The same is true with advice on mammograms and PSA tests for cancer. Or ideas like doing puzzles to fend off Alzheimer’s.

What feels right, what’s backed by big money or major institutions, is often wrong. Test it again, and again. Don’t jump to conclusions.

Pretty comforting, actually.

What I find most comforting here is that the bleeding edge health care practiced in America may be wasting money to no good effect. Progress in clinical practice may slow as a result, but more people will be helped for less money.