The most recent issue of the Journal of the American Medical Association reports a study that is shocking and I’ll bet you dollars to donuts that it won’t get much attention. Just read the abstract:

Persistence of Contradicted Claims in the Literature

Athina Tatsioni, MD; Nikolaos G. Bonitsis, MD; John P. A. Ioannidis, MD
JAMA. 2007;298(21):2517-2526.

Context Some research findings based on observational epidemiology are contradicted by randomized trials, but may nevertheless still be supported in some scientific circles.

Objectives To evaluate the change over time in the content of citations for 2 highly cited epidemiological studies that proposed major cardiovascular benefits associated with vitamin E in 1993; and to understand how these benefits continued being defended in the literature, despite strong contradicting evidence from large randomized clinical trials (RCTs). To examine the generalizability of these findings, we also examined the extent of persistence of supporting citations for the highly cited and contradicted protective effects of beta-carotene on cancer and of estrogen on Alzheimer disease.

Data Sources For vitamin E, we sampled articles published in 1997, 2001, and 2005 (before, early, and late after publication of refuting evidence) that referenced the highly cited epidemiological studies and separately sampled articles published in 2005 and referencing the major contradicting RCT (HOPE trial). We also sampled articles published in 2006 that referenced highly cited articles proposing benefits associated with beta-carotene for cancer (published in 1981 and contradicted long ago by RCTs in 1994-1996) and estrogen for Alzheimer disease (published in 1996 and contradicted recently by RCTs in 2004).

Data Extraction The stance of the citing articles was rated as favorable, equivocal, and unfavorable to the intervention. We also recorded the range of counterarguments raised to defend effectiveness against contradicting evidence.

Results For the 2 vitamin E epidemiological studies, even in 2005, 50% of citing articles remained favorable. A favorable stance was independently less likely in more recent articles, specifically in articles that also cited the HOPE trial (odds ratio for 2001, 0.05 [95% confidence interval, 0.01-0.19; P < .001] and the odds ratio for 2005, 0.06 [95% confidence interval, 0.02-0.24; P < .001], as compared with 1997), and in general/internal medicine vs specialty journals. Among articles citing the HOPE trial in 2005, 41.4% were unfavorable. In 2006, 62.5% of articles referencing the highly cited article that had proposed beta-carotene and 61.7% of those referencing the highly cited article on estrogen effectiveness were still favorable; 100% and 96%, respectively, of the citations appeared in specialty journals; and citations were significantly less favorable (P = .001 and P = .009, respectively) when the major contradicting trials were also mentioned. Counterarguments defending vitamin E or estrogen included diverse selection and information biases and genuine differences across studies in participants, interventions, cointerventions, and outcomes. Favorable citations to beta-carotene, long after evidence contradicted its effectiveness, did not consider the contradicting evidence.

Conclusion Claims from highly cited observational studies persist and continue to be supported in the medical literature despite strong contradictory evidence from randomized trials.
Author Affiliations: Department of Hygiene and Epidemiology, (Drs Tatsioni, Bonitsis, and Ioannidis) and the Department of Dermatology (Dr Bonitsis), University of Ioannina School of Medicine; and the Biomedical Research Institute, Foundation for Research and Technology-Hellas (Dr Ioannidis), Ioannina, Greece; Institute for Clinical Research and Health Policy Studies, Department of Medicine, Tufts University School of Medicine, Boston, Massachusetts (Drs Tatsioni and Ioannidis).

Even if you are not trained as a statistician you get the drift from the article title. Doctors persist in believing disproven claims. Let’s drill down a bit here.

The researchers in this article draw a distinction between information generated from “epidemiological” studies versus information generated from “randomized controlled trials (RCT).” These researchers put forth the claim that RCTs are the gold standard and if epi studies conflict with the experiments, science goes with the experiments. (If you want all the pain and glory of learning about these things you can start with this chapter from the Complete Idiot’s Guide to Persuasion on research methods. You’ll like it. Really. It won’t hurt a bit.)

To demonstrate their point, the researchers use the example of Vitamin E studies. The initial findings of a positive effect for using Vitamin E came from a couple of epi studies published in the early 1990s. (I remember reading the NEJM one and using it in, of all things, a large lecture introductory media effects class – the kids were bedazzled.) An epi study is a type of survey where you ask a lot of people (millions sometimes), “Do you take Vitamin E?” and then wait 10 years and ask all the same people, “Are you still alive?” Now, of course, you ask a lot more than the Vitamin E question and you also ask for other interesting outcomes besides mortality, but you get the point.

Experimenters, by contrast, take a slightly different approach. They will get a smaller group of people (sometimes less than one hundred), give everyone a pill, but some folks are randomly assigned a real Vitamin E pill while the others get sugar pill, a placebo. The researchers then watch everyone for awhile and see if anyone dies or gets sick or files a lawsuit.

Experimenters typically mistrust epi researchers while epi researchers tend to view themselves as the true Children of Science. Let’s just say that there is a tension between the RCT folks and those epi mavens. Oddly enough, I am an experimentalist and I can also play statistics like chords on a guitar, so guess where my biases lie. Back to the opera.

We have frightening evidence that medical researchers and physicians and nurses and all those people who went to Schools of Medicine or Public Health, keep on believing things that are untrue. And not just believing, they keep on citing them, teaching them, defending them.

There is likely to be a brief debate about this article, especially from the epi community. Please make sure I’m not on the jury for this one because I’ve already tangled with the epi folks. They require budgets slightly larger than the national debt, typically find piddling effect sizes expressed in those lame 1.1 relative risk ratios, and then they think they’ve discovered blue sky, white snow, and a free lunch . . . not to be harsh or judgmental or unfair or wicked or even accurate about it.

Now, the thoughtful among you might be wondering what all of this has to do with persuasion besides Steve whaling away on some group of people you’ve never met.

Here: if scientists get it wrong and persist in getting it wrong, what explains their beliefs and behavior?

Persuasion theory and research.

Not health theory and research.

This research study demonstrates a common finding in persuasion research on biased, central route processing. Smart, thoughtful people will try to make the data fit their hypothesis even when it is apparent that the data don’t fit.

This is another way of saying that scientists and physicians and their ilk are human after all and behave according to the same rules of psychology that all us mere mortals obey. We tend to believe we are smart, consistent, and justified and we tend to make data fit our key beliefs more often than we make our beliefs fit “scientific reality.”

As I mentioned at the top of this post, it’s unlikely that this JAMA story will get much attention despite its critical importance. Here’s a quick illustration.

On the same day that JAMA published this research article, the New England Journal of Medicine published a study on childhood obesity and adult heart disease. Using the Google news aggregator, I found that in the two days following these publications the childhood obesity study has been picked up in 350 different web sources, the majority of which are standard news sites like the New York Times or the Washington Post. By contrast, in the two days since the publication of this scientific bias study, the Google news aggregator lists fewer than a dozen sites carrying a story about it.

What’s interesting to me about this comparison is that the childhood obesity study is an epi-style method that finds about the same effect size as the Vitamin E studies. As we’ve seen with those epi studies on Vitamin E, we need to be extremely careful in accepting the conclusions until we get stronger evidence from experimental studies (RCTs).

In an earlier post I noted an odd persuasion outcome. Pfizer, an evil pharma and master manipulator of the universe, was experiencing signficant sales declines. Given the reputation many people give to pharmas and their fabulous persuasion skills, I noted that it was therefore peculiar that Pfizer was having trouble. If you’re that good, how can sales decline so badly?

Yesterday, the Wall Street Journal ran a comprehensive story about the continuing pharma woes and not just Pfizer. If you don’t have a subscription, you won’t be able to read this story, so let me quote the lead paragraphs.

Big Pharma Faces Grim Prognosis

Over the next few years, the pharmaceutical business will hit a wall.

Some of the top-selling drugs in industry history will become history as patent protections expire, allowing generics to rush in at much-lower prices. Generic competition is expected to wipe $67 billion from top companies’ annual U.S. sales between 2007 and 2012 as more than three dozen drugs lose patent protection. That is roughly half of the companies’ combined 2007 U.S. sales . . .

First, let me again note as I did in the earlier post, I am not gloating about this economic disaster. Lots of families are going to face some difficult challenges because of this and that’s bad. I would prefer that it didn’t happen.

The larger point, however, is how can this occur? A 50% decline in sales among some of the greatest persuasion manipulators this side of the tobacco companies?

My argument is that Big Pharmas are not and never were strong persuasion agents. That is, they were not particularly good at using words to change the way people think, feel, or behave. They were great at research, distribution, and, most importantly, patents. As the WSJ story makes clear, now that patents are expiring and generics can flood the market, the Big Pharmas will deflate.

I would also argue that the vaunted image of the tobacco companies as persuasion masters (past and present) is also overblown. Hey, how good does your message have to be when the product is also addictive? Merely giving away the product for free is sufficient to create later sales and that requires no persuasion skill.

The same reasoning applies with Pharmas. When no one else has that Pill, you don’t need a Genuine Expert like me to devise persuasion strategy and tactics. Just publicize it and make sure it is available for distribution and you’re rich.

Quick recap:

1. Nope, no gloating here; failure is bad and it hurts and I don’t take any pleasure in it.

2. Don’t confuse success with persuasion skill. Simply because They are Smooth Talkers who make a lot of money, you shouldn’t assume They acquired that money because they are Smooth Talkers.

3. Remember the Rules. Power corrupts persuasion. Pharmas (and Big Tobacco) had a lot of Power (patents, addiction) and achieved their success through that Power even if they also did some Persuasion, too.

Finally, if you’re a Pharma person reading this, good luck to you. Stay optimistic. You have a lot of skill, just keep after it.