Thank you for participating in my psychology experiment on decision-making. Please read the instructions below:

Most modern theories of decision-making recognize the fact that decisions do not take place in a vacuum. Individual preferences and knowledge, along with situational variables can greatly impact the decision process. In order to facilitate our research on decision-making we are interested in knowing certain factors about you, the decision maker. Specifically, we are interested in whether you actually take the time to read the directions; if not, then some of our manipulations that rely on changes in the instructions will be ineffective. So, in order to demonstrate that you have read the instructions, please ignore the sports items below. Instead, simply continue reading after the options. Thank you very much.

Which of these activities do you engage in regularly? (write down all that apply)

1) Basketball

2) Soccer

3) Running

4) Hockey

5) Football

6) Swimming

7) Tennis

Did you answer the question? Then you failed the test.

The procedure above is known as an Instructional Manipulation Check, or IMC. It was first outlined in a 2009 paper, published in The Journal of Experimental Psychology, by the psychologists Daniel Oppenheimer, Tom Meyvis and Nicolas Davidenko. While scientists have always excluded those people who blatantly violate procedure – these are the outliers whose responses are incoherent, or fall many standard deviations from the mean – it’s been much harder to identify subjects whose negligence is less overt. The IMC is designed to filter these people out.

The first thing to note about the IMC is that a lot of subjects fail. In a variety of different contexts, Oppenheimer et al. found that anywhere from 14 to 46 percent of survey participants taking a survey on a computer did not read the instructions carefully, if at all.

Think, for a moment, about what this means. These subjects are almost certainly getting compensated for their participation, paid in money or course credit. And yet, nearly half of them are skimming the instructions, skipping straight ahead to the question they’re not supposed to answer.

This lack of diligence can be a serious scientific problem, introducing a large amount of noise to surveys conducted on screens. Consider what happened when the psychologists tried to replicate a classic experiment in the decision-making literature. The study, done by Richard Thaler in 1985, goes like this:

You are on the beach on a hot day. For the last hour you have been thinking about how much you would enjoy an ice-cold can of soda. Your companion needs to make a phone call and offers to bring back a soda from the only nearby place where drinks are sold, which happens to be a [run-down grocery store] [fancy resort]. Your companion asks how much you are willing to pay for the soda and will only buy it if it is below the price you state. How much are you willing to pay?

The results of Thaler’s original experiment showed that people were willing to pay substantially more for a drink from a fancy resort ($2.65) than from a shabby grocery store ($1.50), even though their experience of the drink on the beach would be identical. It’s not a rational response, but then we’re not rational creatures. (Thaler explained this result in terms of “transaction utility,” or the tendency of people to make consumption decisions based on “perceived merits of the deal,” and not some absolute measure of value.)

As Oppenheimer et. al point out, the IMC is particularly relevant in experiments like this, since the manipulation involves a small change in the text-heavy instructions (i.e., getting a drink from a resort or a grocery store.) When Oppenheimer et. al first attempted to replicate the survey, they couldn’t do it; there was no price difference between the two groups. However, after the scientists restricted the data set so that only those participants who passed the IMC were included, they were able to detect a large shift in preferences: people really were willing to pay significantly more for a drink from a resort. The replication of Thaler’s classic paper isn’t newsworthy, of course; it’s already been cited more than 3,800 times. What is interesting, however, is that the online replication of an offline experiment required weeding out less attentive subjects.

The results of the IMC give us a glimpse into the struggles of modern social science: it’s not easy finding subjects who care, or who can stifle their boredom while completing a survey. If nothing else, it’s a reminder of our natural inattentiveness, how the mind is tilted towards distraction. As such, it’s a cautionary tale for all those scientists, pollsters and market researchers who assume people are paying careful attention to their questions. As Jon Krosnick first pointed out in 1991, most surveys require cognitive effort. And since we’re cognitive misers – always searching for the easy way out – we tend to engage in satisficing, or going with the first acceptable alternative, even when it’s incorrect.

This is a methodological limitation that’s becoming more relevant. In recent years, scientists have increasingly turned to online subjects to increase their n, recruiting participants on Amazon’s Mechanical Turk and related sites. This approach comes with real upside: for one thing, it can get psychology beyond its reliance on Western Educated subjects from Industrialized, Rich and Democratic countries. (This is known as the W.E.I.R.D. problem. It’s a problem because the vast majority of psychological research is conducted on a small, and highly unusual, segment of the human population.)

The failure rates of the IMC, however, are a reminder that this online approach comes with a potential downside. In a recent paper, a team of psychologists led by Joseph Goodman at Washington University tested the IMC on 207 online subjects recruited on Mechanical Turk. The scientists then compared their performance to 131 university students, who had been given the IMC on a computer or on paper. While only 66.2 percent of Mechanical Turk subjects passed the IMC, more than 90 percent of students taking the test on paper did. (That's slightly higher than the percentage of students who passed on computers.) Such results lead Goodman, et. al to recommend that “researchers use screening procedures to measure participants’ attention levels,” especially when conducting lengthy or complicated online surveys.

We are a distractible species. It’s possible we are even more distracted on screens, and thus less likely to carefully read the instructions. And that’s why the IMC is so necessary: unless we filter out the least attentive among us, then we’ll end up collecting data limited by their noise. Such carelessness, of course, is a fundamental part of human nature. We just don’t want it to be the subject of every study.

Oppenheimer, Daniel M., Tom Meyvis, and Nicolas Davidenko. "Instructional manipulation checks: Detecting satisficing to increase statistical power."Journal of Experimental Social Psychology 45.4 (2009): 867-872.

Goodman, Joseph K., Cynthia E. Cryder, and Amar Cheema. "Data collection in a flat world: The strengths and weaknesses of Mechanical Turk samples."Journal of Behavioral Decision Making 26.3 (2013): 213-224.