What celebrity gossip, fad diets, and evidence-based practices have in common
In my last column, “Ducks, Data, and Evidence-Based Politics,” I shared how critical evidence of results will be for organizations that want to survive in the new era of permanently tight budgets. I’m not rescinding that advice, but this column will serve as a reminder of the worrisome limitations associated with a rigorous focus on evidence.
Much of the substantiation for evidence-based practices comes from academic literature and major studies by prestigious organizations, often under contract to the federal government. That may be problematic, according David H. Freedman, author of the new book, Wrong: Why Experts Keep Failing Us—And How to Know When Not to Trust Them.
One of the best ways Freedman illustrates his point is with the story of John Ioannidis, a doctor who methodically reviewed hundreds of published medical studies while serving appointments at the National Institutes of Health and Johns Hopkins University in the mid-1990s.
According to Freedman, “Ioannidis did indeed spot a pattern—a disturbing one. When a study was published, often it was only a matter of months, and at most a few years, before other studies came out to either fully refute the finding or declare that the results were ‘exaggerated’ in the sense that later papers revealed significantly lesser benefits to the treatment under study. Results that held up were outweighed two-to-one by results destined to be labeled ‘never mind.’”
These studies, Freedman writes, “exhibited the sort of wrongness rate you would associate more with fad-diet tips, celebrity gossip, or political punditry than with state-of-the-art medical research.”
Wisecracks about political punditry aside, this has disturbing implications for the trend toward evidence-based practices, particularly when they have budgetary implications.
Freedman points out several sources of error, including:
Which Numbers Count? Numbers imply objectivity, but the choice of which numbers to use is subjective. This is evident in, for instance, rankings of colleges and universities. Each source ranks schools differently based on which data sets they examine and how they are prioritized.
|
Problems of Determining Cause and Effect. This is the same conundrum as the chicken and the egg. Here’s another: does the rooster’s crowing cause the sun to come up?
Animal Studies. Many studies are conducted on rats instead of humans. Yet, there is a significant difference between the two.
Publication Bias. It’s important to know who funded a study. Even if there is no issue with sponsorship, the journals that publish study results are themselves sometimes biased toward studies that show positive results. Researchers, many of whom are subject to “publish or perish” pressures, are less likely to be rewarded for studies that produce no results. Even though studies that verify existing results may actually contribute more to understanding the issues, replication studies almost never receive funding.
Difficulty Challenging Common Wisdom. People often point to the “wisdom of crowds” when observing collective decision making; however, too little attention is paid to the “idiocy of crowds.” Group think is a powerful force and researchers are not immune to it. Those who challenge dominant or majority opinions can find themselves marginalized.
Shaken Confidence
Given all these causes of error, Freedman argues that far too much confidence is placed in expert opinion.
This panoply of wrongness begs the question: Is there any hope for finding the truth? Of course. One way is to simply recognize all of these problems and work to rectify them.
And, appropriately given the book’s subject matter, Freedman leaves one final piece of humble wisdom. He devotes an entire appendix to explaining the ways that his book may be, well … wrong.
Patrick Lester is senior vice president for public policy for the Alliance and UNCA. He has a master’s degree in public policy from Georgetown University. His past experience includes director of public policy for United Way of America, aide for President Bill Clinton’s Domestic Policy Council, senior policy analyst for the Maryland Association of Nonprofit Organizations, and legislative director for the Coalition of Human Needs. He can be reached at 202-429-0400, ext. 15, or by e-mail. |  |
View the archive of Capitol Connection columns or the archive for all columnists.
Printer-friendly version