The Certainty of Uncertainty in Medicine

 George D. Lundberg, MD

Medscape

Gerry Gajadharsingh writes:

 Research, that old chestnut! It’s refreshing to see a medic challenging the commonly accepted belief, that much of medicine is based on sound scientific research. The bit that really made me laugh the most was

 “Many of us will recall being told by some professors in medical school that half of what we were being taught was wrong but that the professor (and the poor students) could not know which half.”

 All research is open to bias and I am as guilty as any by choosing to blog about articles and research that interests me and fits in with my philosophical view of life and health.

 The challenge is that patients live in the real world and often present with multiple symptom/problems, affecting different body and mind systems with multiple aetiologies. Research prefers to look at 1 factor and then jump to conclusions, eg eating fat makes you fat (something taken as the “truth” for over 20 years, backed up by “extensive research”, only for more recent research to suggest this is far from proven, with the likely culprit more likely to be sugar.

 I remember trying to get my research study “An Integrated Osteopathic Approach to Human sub-fertility) though the approval and ethics committees as part of my MPhil/PhD, over 10 years ago. They didn’t like the integrated bit and wanted me to focus on 1 thing. Amazing that Integrated Medicine is now the buzzword, apparently everyone now practices in an “integrated way”. By the way I managed to persuade them, developed a very interesting diagnostic and treatment protocol, which I still use for my sub-fertile couples.

 So it would be great if ALL clinicians were open to the FACT that there is a lot that we don’t know about illness and disease and don’t slavishly go on about how conventional medicine practices “evidence based medicine” when it is evident (to many people) that this evidence is not nearly as good as it sometimes makes out.

 

 

Do you remember the old cartoon that shows a corporate CEO diving under his desk or running out the back door when told by the office receptionist that “60 Minutes is here to talk”? That is the way American academics may feel when they learn that John Ioannidis may be examining their published work. The bigger they are, the more scared they become—and with good reason.

John Ioannidis (Disclosure: we are members of the same department at Stanford University) has made a career of questioning conventional wisdom and medical dogma that is based upon research, often by “good investigators” at “the best institutions,” funded by the biggest agencies, and published in “the best journals.” He has found much of these research findings to be either partially or completely false.

His 2005 paper, “Why Most Published Research Findings Are False” is in the open-access journal PLOS Medicine. It has become the most accessed article in the history of the Public Library of Science, known as PLOS. John argues persuasively that many forms of bias operate in affected domains, fields, cultures, people’s heads, reward systems, and ecologic groups. This plays out in small sample sizes; excessive flexibility in designs, definitions, outcomes, and analytic modes; and financial and other conflicts of interest. The mathematics, the psychology, and a misunderstanding or deliberate ignoring of the overriding importance of positive predictive values negates the validity of most—and he means most—published research.

John Ioannidis is not a one-hit wonder. He has published and lectured many times, but often along the same lines.

The Conventional Wisdom Bias

Now there is a new book, called Willful Ignorance; The Mismeasure of Uncertainty by prominent statistician Herb Weisberg of the firm Causalytics in Needham, Massachusetts.

I recommend this book to all serious medical thinkers who are concerned about the frequency with which we in medicine simply do not know the explanation for any particular disease, how to prevent it, how to diagnose it, how to treat or manage it, and how our “conventional wisdom” is so often turned upside down by new findings.

In some ways, there is nothing new about this. Many of us will recall being told by some professors in medical school that half of what we were being taught was wrong but that the professor (and the poor students) could not know which half. I have lived a life in medicine long enough to fully appreciate the wisdom of that professor, even though I thought at the time that the statement was made largely in jest.

Weisberg points out the vast amount of research that is being mass-produced and how little useful insight is generated by it. In addition, he emphasizes the gulf between quantitative research methodology results and clinical practice. One of the problems is the lower priority given to attempting to replicate published research, instead believing that an issue is settled. This bias applies to investigators, funding agencies, the academic reward system, peer reviewers, and medical editors.

How to Fix the Mess

Back to Ioannidis. In 2014, John helpfully proposed how to fix the mess. He is not opposed to research and he values scientific studies that have ultimately turned out to be valid and have benefitted the human condition, but he still alleges that an estimated “85%” of research resources are wasted.

Here is Ioannidis’s description of research practices that may help to increase the proportion of research findings that are indeed true:

  • More large-scale collaborative research;
  • Adoption of a culture of replication;
  • Registration of studies, protocols, analysis codes, datasets, raw data, and results;
  • Sharing of data, protocols, materials, software, and other tools;
  • Enhanced practices of reproducibility;
  • Containment of conflicts of interest of sponsors and authors;
  • More appropriate statistical methods;
  • Better training of the scientific workforce in methods and statistical literacy;
  • Standardization of definitions and analyses;
  • More stringent thresholds for claiming discoveries or “successes”;
  • Improvement of standards of study design;
  • Improvements in peer review, reporting, and dissemination of research results.

Okay, but that is a load of change for people, processes, institutions, finances, education, readers, promoters, and organizations—an ingrained, self-perpetuating culture. Lots of luck with that. But the alternative is continued “willful ignorance” just as Weisberg says. Let’s get on with this sea change. Thank you, John and Herb.