Evidence: It Worked for My Aunt Tillie Is Not Enough

Dr. Harriet Hall

I had a friend who used all kinds of questionable treatments including homeopathy. I asked her how she decided what to try. She said if a friend told her something had worked for him, and if it didnt seem dangerous, she would try it. That was all the evidence she needed. She didnt care about scientific evidence because she said, Science doesnt know everything. Comedian Dara Briain had the perfect answer to that: Science knows it doesnt know everything; otherwise, itd stop. But just because science doesnt know everything doesnt mean you can fill in the gaps with whatever fairy tale most appeals to you. When Oprah Winfrey told Jenny McCarthy that experts said there was no scientific evidence that vaccines caused autism, Jenny retorted, My science is named Evan, and hes at home. Thats my science.

There is a huge disconnect between what science-based medicine calls evidence and what alternative medicine and the general public call evidence. They are using the same word, but speaking a different language, making communication next to impossible.

First, there is no such thing as alternative medicine. There is only medicine that has been tested and proven to work and medicine that hasnt. If a treatment currently considered to be alternative were adequately tested and proven to work, it would be incorporated into mainstream medical practice and could no longer be considered alternative. It would become just medicine. So-called alternative medicine can be defined as medicine that isnt supported by good enough evidence to earn it a place in mainstream medicine.

Alternative medicine, along with complementary and alternative medicine (CAM) and integrative medicine, is not a meaningful scientific term, but a marketing term created to lend respectability to things that we used to call by less respectable names like quackery, folk medicine, and fringe medicine. It puts everything outside of science-based medicine under a single umbrella, including things that might work but havent been adequately tested, things that have been tested and proven not to work but that are still being used (like applied kinesiologya bogus muscle testing procedure used by many chiropractors), and fanciful things that couldnt possibly work, like homeopathy. It fosters the misleading impression that all these things might be equally valid.

There is no such thing as alternative medicine. There is only medicine that has been tested and proven to work and medicine that hasnt.

Science-based medicine has one rigorous standard of evidence, the kind of evidence government agencies require before they allow a pharmaceutical to be sold. CAM has a double standard. They gladly accept a lower standard of evidence for treatments they believe in. However, I suspect they would reject a pharmaceutical if it were approved for marketing on the kind of evidence they accept for CAM.

When science-based medicine evaluates the evidence for alternative medicine, here is what it concludes:

  • Acupuncture is just a theatrical placebo.
  • Homeopathy not only doesnt work but couldnt possibly work.
  • Chiropractic is essentially physical therapy contaminated with bogus diagnostic and treatment methods.
  • Energy medicine is fantasy medicine dressed up with sciencey-sounding words like quantum and frequencies.
  • Naturopathy claims to stress prevention and to consider the whole patient, but thats just what every good doctor does. It offers natural treatments that have not been tested or that have been proven not to work, like homeopathy. What naturopaths do that is good is not special and what they do that is special is not good.
  • Herbal medicine is plausible: after all, half of our prescription drugs came from plants. But every herbal remedy must be tested individually to determine if it is effective and safe; and statistically the great majority of promising remedies fail testing.

So while science says the evidence is lacking, CAM says there is plenty of evidence:

They cite preclinical studies in animals and test tubes, but scientists recognize that even the most promising preclinical studies can be misleading and must be confirmed in humans.

They cite case reports, but we know that case reports are only useful as indicators of what science should evaluate with controlled studies.

Sometimes they can cite clinical studies in humans, but those studies are often flawed. They may be poorly designed or lack a control group, they may be contradicted by other studies, they may be based on such extraordinary claims that we would have to have extraordinary evidence to accept them, and they dont build on each other to create a coherent body of evidence.

They cite doctors who say, In my experience, X is effective. Mark Crislip, one of my colleagues on the Science-Based Medicine blog, says in my experience are the three most dangerous words in medicine, because personal experience is so very compelling and so often leads to false beliefs.

They rely heavily on testimonials and anecdotal evidence. Scientists know that the plural of anecdote is not data; no matter how many testimonials you accumulate, they cant ever prove that the treatment works. Think of all the testimonials over the centuries for balancing the humors with bloodletting.

CAM believers find their evidence very convincing, and it is hard to explain why we dont and how their kind of evidence has been shown over and over to lead people to false conclusions.

Scientists know that the plural of anecdote is not data; no matter how many testimonials you accumulate, they cant ever prove that the treatment works.

You cant blame people for accepting that kind of evidence. They are only doing what comes naturally. Science doesnt come naturally to humans. Evolution equipped us with thinking processes that facilitated survival on the African savannah. Our ancestors had only two sources of information: their own experience and the experiences of other people. If someone said Dont eat those berries; they are poisonous, there was no way to verify the information; but listening to others was likely to help keep you alive. Humans became very adept at pattern recognition. If a pattern of shadows in the bushes looked sort of like a lion, it was safer to assume it was a lion and run away. The cost of over-interpreting patterns was small, while missing the pattern of a real lion could be fatal. Our ancestors learned to make quick decisions: if it was really a lion, they needed to run away now. And emotions served as motivators: fear made them run away faster.

Those thinking processes helped our ancestors survive in a prehistoric environment, but they have become a handicap in the modern world. Today we have more sources of information, but our minds still work the old way. We prefer stories to studies, anecdotes to analyses. We see patterns where none exist. We jump to false conclusions based on insufficient evidence. Emotions trump facts. If your neighbor had a bad experience with a Toyota, youre likely to remember his story and not buy a Toyota even if Consumer Reports says its the most reliable brand. That isnt logical, but humans are not Vulcans. When we act illogically, were just doing what evolution has equipped us to do. It takes a lot of education and discipline to overcome our natural tendencies, and not everyone can do it.

Ray Hyman is a psychologist and one of the founders of modern skepticism. When I asked him why some people become skeptics and others dont, he said he thinks skeptics are mutants: something has evolved in our brains to facilitate critical thinking. Is it nature or nurture? Or both? Maybe life experiences also incline some of us to think more critically.

So what kind of evidence should persuade us? Its tricky, because, as John Ioannidis has shown us, many published research findings are wrong. Early pilot studies are overturned by larger, better-designed studies. Studies are influenced by researcher bias. Some are even fraudulent. With over 23,000 scientific journals and 700,000 papers published every year, its easy to find a study to support almost any belief.

In this context, there is a hierarchy of evidence:

  • Basic science.
  • Test tube studies (in vitro).
  • Animal studies (in vivo).
  • Case reportsof a single patient.
  • Case seriesreporting on a number of patients.
  • Case control studies (example: comparing people with and without lung cancer to see if there are more smokers in the group with lung cancer).
  • Cohort studies (example: following people who smoke and who dont smoke over a period of time to see which group develops more lung cancers).
  • Epidemiologic studies (example: studying whether people in countries with more smokers develop more lung cancers. These studies can show correlations, but they cant determine causation. Countries with more smokers might have other confounding factors that predispose to lung cancer).
  • Randomized controlled trials (RCTs).
  • Systematic reviews and meta-analyses that evaluate all the published evidence pro and con.

The best studies are large, randomized, placebo-controlled, and double- blinded; but that isnt always necessary or even possible. We dont need an RCT to know that parachutes save lives. Its unethical to knowingly endanger a control group by denying them effective treatment. (Anyway, who would volunteer to jump without a parachute?) We didnt need RCTs to figure out that smoking causes lung cancer because there is an overwhelming body of corroborating evidence from several different kinds of research.

Evidence matters. Science works. Its a collaborative, self-correcting enterprise. It never relies on a single study. When a finding is true, it will be corroborated by other studies, and a consensus will eventually build based on a cohesive body of evidence. False findings are eventually overturned and discarded. We can trust the scientific method; we cant trust anecdotes. Sorry, Aunt Tillie!

http://www.skeptic.com/reading_room/it-worked-for-my-aunt-tillie-is-not-evidence/

Courtesy: Skeptic