Healthy Influence – Persuasion Blog

Persuasion is always about “something” and can never be in a Seinfeld episode about “nothing.”  Right now, a great “something” is anything scientific about health and safety.  You give me a scientific finding about heart attacks, cancer, or erectile dysfunction and baby fire up the Cascade, cause we’re gonna makes some change!  Stated more prudently, science plus persuasion equals change (and maybe fame and fortune).

Of course, the science has to be good because if you persuade on bad science, you make a lot of people confused and maybe worse off.  So, just to be on the safe side, only use persuasion on scientific health and safety research published in premier sources.  And, didja know, there’s even an association of the editors of such elite outlets who work together to develop and enforce strong standards of science for their publications, so silly social scientists like myself who don’t know adenine from zirconia can rely upon the quality of the source to find strong science.

So goes the theory anyway.

If you read this blog you’ll find occasional posts where I throw up my hands over weaknesses in medical researchers and wail over their sins because it makes my job tougher.  You can’t sell that bathtub if the pill doesn’t get you dirty, you know.

Well, here’s Yet Another Reason why persuading with science is such a chore.
The Journal of the American Medical Association offers a study of the registration history for randomized controlled trials that publish in “high impact” journals.  According to the abstract:

Results Of the 323 included trials, 147 (45.5%) were adequately registered (ie, registered before the end of the trial, with the primary outcome clearly specified). Trial registration was lacking for 89 published reports (27.6%), 45 trials (13.9%) were registered after the completion of the study, 35 (10.8%) were registered with no or an unclear description of the primary outcome, 39 (12%) were registered with no or an unclear description of the primary outcome, and 3 (0.9%) were registered after the completion of the study and had an unclear description of the primary outcome. Among articles with trials adequately registered, 31% (46 of 147) showed some evidence of discrepancies between the outcomes registered and the outcomes published. The influence of these discrepancies could be assessed in only half of them and in these statistically significant results were favored in 82.6% (19 of 23).

Conclusion Comparison of the primary outcomes of RCTs registered with their subsequent publication indicated that selective outcome reporting is prevalent.

What this means for everyday people is that the majority of randomized controlled trials (RCT) whose results are published in top medical journals are playing fast and loose with the standard of the scientific method and the editorial standards of those top journals.  If you’re not in the business, you may not see the drama, so let me deconstruct.

When you do an RCT and register it, it means that you are publicly announcing in advance, before you do the experiment, what you expect to happen.  In Las Vegas we call this placing your bet.  In Vegas, you place your money on the table then they deal the cards or spin the wheel and you get what you get.  If you’re doing really good science, you know what you are looking for, why you are looking for it, how to look for it, and you will make the bet BEFORE the experiment.

Except in over half the cases of RCTs published in top journals, the researchers did not register the study.  In other words, they did not put their money on the table, but took the cards or the spin of the wheel and THEN they made the bet.

Now, very quickly, failure to register does not mean the researchers falsified data.  It does mean that they may change their bet after they’ve seen the spin of the wheel or the throw of the dice.  Then, after the result, they offer an explanation of the results which is like throwing the dice, seeing an eight, then betting on eight.

The scientific danger here is that researchers are taking data that was originally designed in a specific experiment to test a specific hypothesis, but because the data did not work that way, they reanalyze the data from a slightly different perspective (e.g. look at only men over 55 married to blonde women thirty years younger) and, lo and behold, they find a positive outcome for a little blue pill.  They then write up the experiment from this perspective, omitting mention of all the men under 55 or not married to younger blonde women, and send this good news to the great journal.

Editors at these top journals apparently are not checking on researchers and are assuming that high quality folks who went to the same schools they did are playing according to Hoyle, which as this abstract demonstrates, appears to be an assumption that is wrong more than half the time.  Boola-boola, old chum.

This is a world class crazy maker for a persuasion guy like me and, of course, for other scientists.  When folks design an experiment one way, check the data, find failure, reanalyze to find success, then write up the success omitting the failure, we’ve got bad science that no one knows about.  All of the benefits of experiments, particularly randomization and control, disappear, but since the researchers are less than forthcoming with the details, no one knows this.

The worst practical implication of this finding is how it corrodes trust.  Who’s deceiving?  Over half the cases checked in this study showed some type of a failure to register.  Now, it’s possible that the researchers still did the study they promised, so who knows?  And, worse still, you don’t know who’s registering and who is not registering.  This is something the editors are supposed to handle and they clearly are not.

Further, many science editors and researchers are clamoring like serfs in a Frankenstein movie on torchlight parade at Castle Pharma searching for the monster of Scientific Fraud for Profit.  It appears while these folks are sharpening their pitchforks many of their peers are playing games with data that would lead to the death penalty if found anywhere at Castle Pharma. (Note:  I am not influenced by any pharma in any way and I do not take little blues pills because Melanie has another intervention that works.)

Whenever I consider any persuasion play I am always worried about the TACT we target.  Is this a good thing?  Is this going to work?  If we change the TACT will people be okay?  When scientific researchers deceive about the nature and quality of the research, it rots the credibility of the science and may offer snake oil in a little blue pill – and that’s best case.  It could kill people, you know.  When you alter an experiment in these ways, you turn it into observational research.  Think about it.