Healthy Influence – Persuasion Blog

Good science often provides both good news and bad news in the same moment and a report by Dan O’Keefe and Jakob Jensen in the Journal of Communication illustrates this.  O’Keefe and Jensen conducted a meta-analysis of the research literature investigating the effects of message framing on health risk behavior.  The good news is that they found something.  The bad news is that the something they found is next to nothing.

At the outset we need to get on the same page with some key words.  First, a meta-analysis is a quantitative review of the literature.  Instead of doing a narrative review where you read everything on a topic, think hard about it, and draw conclusions, a meta-analysis collects all the quantitative data on a topic, analyzes the combined results, and then draws conclusions (quantitative and qualitative).  Well done metas can make a huge contribution to research by providing a large scale perspective with properly applied analysis.  So, a meta is a like a review of the lit, just with numbers instead of words.  It is a well established method that can be abused, but when done right, is a good thing.

Second, message framing is a persuasion tactic that provides information against different backgrounds.  We can say, for example, that health tests like mammograms are helpful either because:  1) the test can detect cancer early when it is easier to treat (gain frame) or 2) if you don’t test, you may not detect cancer early when it is easier to treat (loss frame).  In both message frames, the same claim is made:  Get a mammogram.  However, the gain frame focuses upon advantages while the loss frame keys on disadvantages.  Framing is usually conceptualized within Prospect Theory which is part of the Nobel-prize winning work of Dan Kahneman and Amos Tversky.  You might also know about framing, Prospect Theory, or Kahneman and Tversky through the recent book, Nudge, by Richard Thaler and Cass Sunstein.  Nudges are operationalized through “choice architecture” with message framing being a type of choice design.

Third, Nudge is the current rage for nuanced intellectuals pursuing public policy, particularly with the Federal government.  President Obama has nominated Cass Sunstein to run an important regulatory office in the Executive branch and Sunstein has made no secret of his desire to use persuasion tactics like Nudge and framing to influence citizens to make smarter choices.  While there are many elements to Nudging Public Policy, tactics like framing are part of the Standard Operating Procedure.

Fourth, according to theory, with a particular kind of behavior – disease detection – loss frames should work better than gain frames.  It’s a long argument, you need a good theory background, and it is accessible to anyone willing to die in the library for it, so go to it, otherwise you have to take this claim on its face.  Thus, when Dr. Sunstein becomes the Regulatory Czar, he might propose to write all Federal communications about women getting mammograms with loss frames to encourage more women to get the test.  He might do the same for men and tests for prostate cancer.  With anything that involves disease detection, he might require wording in the loss frame because of the theoretical advantages, plus some good experimental research.

Now, we can get to the research article from O’Keefe and Jensen.  So, what happens when you Nudge with a frame?

According to the meta-analysis, not much.

They scoured the literature and located 53 research reports that provided tests of gain versus loss frames on disease detection.  These studies involved 9,145 participants on a wide variety of topics.  The key comparison was that difference between loss framed messages versus gain framed messages.  Theory predicts that loss frames should be a lot better at motivating people to do the “right” behavior.

And, technically, this is exactly what they found.  Loss frames did produce more behavior change than gain frames and this average change was beyond the .05 level of statistical significance.  Now, the bad news:  the effect size, expressed as the correlation r, was .039.

For those of you with no stats background, a “small” effect r is considered to be anything over .100.  Thus, the obtained r of .039 only one third of a small effect.  Expressed in the Windowpane format this is a 49.39% versus 50.61% difference.  Which is a quantitative way of saying, “Nothing happened” (although that is not what we say when we look at a computer screen and realize our next grant application just died).

Now, quickly to the tut-tuts and “Sir, may I interject the observation that . . .”  There is much more to the analysis.  Lots of numbers.  Lots of nuance.  Lots of potential exceptions.  I agree.  More research is needed!

Yet and still, focus on the Main Point:  All the smart money has been on a very particular bet for over 20 years and that bet is proven to be baseless with this meta-analysis.  Let me provide as unadorned a quote from the paper as I’ve ever read in a scientific research report:

This must be counted a rather disappointing conclusion.

Can I get a big “Hell, yeah!” from all the redneck girls and boys like me on this?

Let’s divide the remaining comment into two streams:  Science and Policy.

The science here is surprised.  Dan O’Keefe does good work and this report is yet another example of his skill, patience, and prudence.  If he has any skin in the game, I have no idea where it is.  The report is objective, balanced, and driven by science.  It provides strong evidence of a theory failure.  I would argue that the first place to pick up the pieces on the science side is through a thorough dual process analysis.  Many framing studies are done without carefully controlling or measuring the processing state of the participant (and I’d make that assertion about a lot of Prospect Theory research).  Framing can function either as a cue or an elaboration moderator and it makes a huge difference in outcomes.  Thus, frames, prospects, and Nudges still may work as claimed, but they need a large rewrite.

The policy, I fear, is probably going to be largely unaffected.  Nudgers, in particular, will probably not read this report and if they do, will find a way to isolate it from their beliefs.  Really smart Nudgers will most likely read this article, discern the master stroke missing from the meta-analysis, and adjust their thinking without testing it.  My largest worry is that people will try to Nudge within the Federal government even in the face of this empirical failure.  These results should not have occurred.  Unless you can demonstrate empirically why and how framing should still work here, you should seriously reconsider Nudging anything.

Otherwise, you are Nudging for Nothing.