How much can we rely on medical experts?

"He charges that as much as 90 percent of the published medical information that doctors rely on is flawed."

That is from a recent article by David H. Freedman in the Atlantic Monthly sent to me by my friend April Harding.   The "he" is Professor John Ioannidis, who has done as much research on this topic as anyone in the world.   Here are some other key quotes from the article:

... he was struck by how many findings of all types were refuted by later findings.

“I realized even our gold-standard research had a lot of problems,” he says. Baffled, he started looking for the specific ways in which studies were going wrong. And before long he discovered that the range of errors being committed was astonishing: from what questions researchers posed, to how they set up the studies, to which patients they recruited for the studies, to which measurements they took, to how they analyzed the data, to how they presented their results, to how particular studies came to be published in medical journals."

... 80 percent of non-randomized studies (by far the most common type) turn out to be wrong, as do 25 percent of supposedly gold-standard randomized trials, and as much as 10 percent of the platinum-standard large randomized trials...

 “You can question some of the details of John’s calculations, but it’s hard to argue that the essential ideas aren’t absolutely correct,” says Doug Altman, an Oxford University researcher who directs the Centre for Statistics in Medicine.

If between a third and a half of the most acclaimed research in medicine was proving untrustworthy, the scope and impact of the problem were undeniable. That article was published in the Journal of the American Medical Association.

Sometimes people contact me to say they think I should emphasize more the importance of experts in development.  I often respond by saying that the issue is in development is finding what works rather than relying on people who have credentials based on their degree or who they work for (see Saundra Schimmelpfennig's excellent post on this).  By contrast ( I used to say) relying on credentials made sense in certain fields where there was a clear link between credentials, knowledge, and outcomes such as structural engineering and medicine.

This article makes me think that I should re-think my reference to medicine - a field that needs to re-examine its own standards of proof and good practice.  It does not mean that I will stop going to the doctor when I need to.  It does mean that I will ask a lot of questions (something fortunately encouraged by my current doctor).

Does this have anything to do with philanthropy and development?  I am not sure, but I did like this quote from another researcher, Athina Tatsioni:

“Usually what happens is that the doctor will ask for a suite of biochemical tests—liver fat, pancreas function, and so on,” she tells me. “The tests could turn up something, but they’re probably irrelevant. Just having a good talk with the patient and getting a close history is much more likely to tell me what’s wrong.”