Skipping dessert to stay healthy, saving money for a rainy day, not shouting at your boss even when there might be good reason—psychologists group these and other human behaviors under the large umbrella of “self-regulation.” That is, despite the near-term temptations, longer-term goals prevail. But a recent study in Nature Communications suggests that the concept of self-regulation isn’t a cohesive phenomenon, and that a person’s skills in this arena aren’t clearly predicted by many of the tasks psychologists use to measure this concept. “The phrase ‘self-regulation’ isn’t doing us that much good,” says Ian Eisenberg, a graduate student in psychology at Stanford University in California and the first author on the study.

He and his advisor, Russell Poldrack, wanted to create a broad theory, or ontology, of all the aspects of thought involved in so-called self-regulation. They recruited 522 subjects online (see “News Feature: How online studies are transforming psychology research”) to complete 10 hours’ worth of surveys and psychological tasks thought to be relevant to self-regulation. For example, in surveys, participants rated how closely they agreed with statements such as “I am able to resist temptation.” The tasks tested abilities thought to be relevant to self-regulation, such as one’s ability to stop doing something. In one case, subjects responded to a series of shapes by pushing a key unless they received a signal not to. Participants also answered questions related to real-world self-regulation, regarding characteristics such as body weight, smoking habits, and mental health.

The researchers then created a graphical “psychological space” based on how well each survey or task result correlated with the other measures. If two tests measured the same psychological concept, they should overlap or sit close together in this space; if they measured different characteristics, they should be spaced further apart. The researchers found that survey results grouped together, as did the assessments from tasks, but that the two categories were clearly separated. “Surveys and tasks do not relate to each other at all,” says Eisenberg.

Within just the surveys, it was possible to discern categories of self-regulation that tended to go along together, such as eating control or financial risk-taking. Similarly, certain tasks grouped together, such as those that measured caution. The data are freely available online for other scientists to plumb.

Then, the authors asked how well the psychological surveys and tasks predicted a person’s real-world achievements, such as healthy habits. Surveys moderately predicted real-world outcomes; tasks made few solid predictions. “If you think self-regulation is a thing, then the tasks are certainly not probing it very well,” says Eisenberg.

That may be because the tasks the researchers chose, such as ones that monitored intelligence or risk-taking, don’t closely map to self-regulation, says Kathleen Vohs, a behavioral scientist at the University of Minnesota in Minneapolis. “In my view, they didn’t really give the notion of self-regulation tasks a fair shake,” says Vohs, who did not participate in the study. “Self-regulation is more appropriately studied using real behavioral measures, such as overcoming urges, working instead of goofing off, eating something healthy instead of candies and chocolate, and exercising when what one really wants to do is lie on the couch and watch TV.”

Eisenberg says that the group intentionally probed a broad range of psychological tasks, going beyond those traditionally studied in the self-regulation field, in an attempt to define the “borders” of self-regulation–related tasks in their psychological space. Nonetheless, he says that several tasks were relevant for what psychologists consider to be self-regulation. For example, several of the tests measured executive function, which allows people to plan and carry out tasks. “Risk-taking and intelligence may also be thought of as important for self-regulation,” he says.

The results offer an important reminder to the psychology field that laboratory tasks don’t necessarily measure real-world traits, says Brent Roberts, a psychologist at the University of Illinois at Urbana–Champaign. But the findings aren’t all that surprising, he adds. “Self-regulation was never a cohesive thing to begin with,” says Roberts, who wasn’t involved in the study. “This is a multidimensional thing.”

Hence, psych-lab tasks, while popular and easy for researchers to perform, probably aren’t the best way to study self-regulation. That doesn’t mean such studies are impossible, adds Roberts, pointing to a recent paper in the European Journal of Personality. That study’s authors pinged people via cell phone several times daily to ask if they had recently done something they considered unpleasant, and what strategies they’d used to complete the annoying task. In principle, it was a more direct, real-world assessment of the ability to persist in unpleasant activities to meet goals.

Still, Roberts suggests that the same methods that Eisenberg and Poldrack used to create their ontology could be applied to a variety of psychological concepts, other than self-regulation, to check their validity. “It’s the same as quality control when you’re building an airplane,” he says. “We should be doing this all the time.”