Sentience Institute (SI)1 is a relatively new think tank within the effective altruism movement. According to SI’s website, the organization was founded in June 2017 with the stated mission of “expanding humanity’s moral circle” through researching the most effective strategies to achieve this goal, with an initial focus on farmed animals. They have released a few publications on this topic—one of which is their “Survey of US Attitudes Towards Animal Farming and Animal-Free Food October 2017.” Here, we review this survey in order to better understand the extent to which we can draw conclusions from the survey’s results. This is important, as the survey produced several surprising results. Assessing the validity of those results will be helpful in determining to what extent our beliefs should change. SI plans to use this initial survey to provide baseline data for the same survey in future years, in order to monitor how attitudes toward animal farming and animal-free food shift as the animal advocacy movement grows. By adequately monitoring changes in these attitudes, this survey could be a valuable resource for annually assessing the extent to which the animal advocacy movement is accomplishing its goals.

Overview

In October 2017 Sentience Institute—in collaboration with the survey company Ipsos—sent a survey of 17 questions to 1,0942 U.S. adults to assess their attitudes toward “animal farming” and “animal-free foods.”3 As this was an exploratory survey, there were no statistical hypotheses SI was looking to confirm. They did however have three main goals:

to provide baseline data for future research using the same survey to test predictions of how adults will respond to conduct exploratory analyses on the associations between responses and demographics

SI found that, overall, U.S. adults showed a stronger opposition toward animal farming and more significant support for animal-free food than expected—with 33% supporting a ban on animal farming and 54% trying to consume fewer animal-based foods and more plant-based foods. It is worth noting that the numbers highlighted in SI’s report exclude those who chose “no opinion.”4 After the survey, SI conducted an exploratory analysis to better understand the association of respondent demographics with responses. Although this analysis did suggest some associations as likely,5 SI noted that these results “should be taken with large grains of salt” because the analysis was not pre-registered, and multiple comparisons weren’t accounted for.6 They do, however, make the raw data and code publicly accessible (and they encourage its use in future research). Since this section of SI’s report focuses more on potential avenues for future research, this blog post will primarily discuss the strengths and limitations of the goals, methods, and conclusions of the survey itself.

Strengths

This survey is not entirely novel—previous Researchers have conducted similar surveys. This includes Faunalytics, who has been collecting information on opinions and behavior relating to animals since 2009. A quick look through the Faunalytics archive of publishings on attitudes toward animals (in addition to a search through Google Scholar7 ) yielded many publications on the topic of attitudes toward animal welfare. However, most of these studies8 seem to focus strictly on animal welfare rather than on the overall concept of raising and slaughtering animals for food (and the potential banning of such practices). SI’s survey seems to be one of the first surveys to directly ask questions regarding these concepts, making it a very important and relevant addition to the field. It is worth noting that this survey was repeated almost identically by the Department of Agriculture at Oklahoma State University in January 2018. The substantially similar results attained by OSU provide significant validation of SI’s results.

From a methodological perspective, SI took steps to ensure the external validity9 of the results by using a census-balanced audience, and took steps to avoid publication bias10 by committing to publish their results before running the survey with the Open Science Framework. They also provided the raw data and code for their exploratory analysis, allowing other Researchers to repeat the analysis. In order to increase the validity of the survey results, SI attempted to avoid social desirability bias11 by not mentioning their name in the survey’s introduction and by using a Likert scale to minimize possible indications of a desired response. Additionally, according to the first footnote within the report, a disqualifying question was used to ensure that all responses used in the analysis were from individuals who had carefully read each question.12

The remainder of this post mainly focuses on a few apparent limitations of SI’s survey and their interpretation of its results. Despite these limitations, however, we feel that this research is an important addition to the field of effective animal advocacy.

Limitations

Survey Questions

SI does not offer an explanation of the rationale that led them to choose the items used in the survey. We believe that doing this would have been quite helpful. Only three of the 17 items include references to previous research,13 so we expect that readers may wonder about the origin of the remaining 14 questions. If the other questions are original to SI and/or if there were important reasons for their selection, that would be useful information to know—especially for those who may like to build on this research in future.

Although SI took steps to reduce the risk of bias and measurement errors, there is one question where these risks seem to have been overlooked. The “personal choice” question seems to be a double-barreled question, which may significantly diminish the value of its responses. The first barrel states, “Whether to eat animals or be vegetarian is a personal choice,” and the second states, “nobody has the right to tell me which one they think I should do.” These are two completely different concepts. The first addresses personal opinions, whereas the second addresses whether people have the right to voice those opinions. If most of the respondents agree with second part of the statement, it somewhat contradicts other responses to the survey.14 For example, if something like a third of adults in the U.S. support a ban on animal farming,15 then some of the same adults who indicated that they believe vegetarianism is a personal choice favor a ban that to a significant extent seems to impose upon that choice. In the next section, we will discuss the tension between the responses to different items in the survey in more detail.

Survey Results and Conclusion

The results of this survey showed some surprising and interesting statistics, especially with regard to slaughterhouses, animal farming, and perceptions of animal welfare. At the beginning of the report the author lists several results that seem most compelling to SI, accompanied by commentary on some of the implications they believe such statistics may have. Some of the results could have benefited from more discussion, such as the following two:

97% of US adults agree “Whether to eat animals or be vegetarian is a personal choice, and nobody has the right to tell me which one they think I should do.”

49% of US adults support a ban on factory farming, 47% support a ban on slaughterhouses, and 33% support a ban on animal farming. The support for the latter two exceeded even Researcher expectations. This suggests that animal-free food advocates might be able to succeed with stronger proposals than they currently use, though readers should take into consideration social desirability bias in this type of survey.

These results seem contradictory—if there were no animal farming or slaughterhouses, people would not be able to make their own decisions about how to eat. Though the author does attribute these surprising results to potential social desirability bias, the lack of definitions for terms used in the survey16 might mean that people are responding to the question(s) using their own concepts for those terms, which may be substantially different from the meaning intended by the survey authors. For instance, perhaps most respondents thought that they could still easily eat animals if there was a ban on slaughterhouses. That said, the 2018 survey replication asked respondents who agreed with a ban on slaughterhouses a follow-up question with a brief description of slaughterhouses17 and approximately 73% of those who agreed with the ban indicated that they knew that a ban would mean “you would not be able to consume meat.”18 This leads us to believe that the discrepancy in the results previously highlighted should be attributed to other factors besides respondents’ lack of knowledge of the relevant concepts. The discrepancy may be largely attributable to social desirability bias or to the fact that people’s beliefs do not always match their individual behaviors.19

The conclusions associated with the following results also seem to merit closer examination:

58% of US adults think “most farmed animals are treated well,”20 despite over a decade of undercover investigations in the US and USDA data suggesting over 99% of farmed animals live on factory farms.21 This suggests either we have insufficient awareness of factory farming, or people have refused to accept the evidence, perhaps due to the emphasis on individual diet change and subsequent cognitive dissonance with their own eating habits.

75% of US adults say they usually buy animal products “from animals that are treated humanely,” despite estimates suggesting fewer than 1% of US farmed animals live on non-factory farms.22 This suggests a psychological refuge effect where people justify their animal product consumption by incorrectly assuming they are eating ethically-produced food.

When reading these two results separately, they offer very interesting conclusions—especially given the USDA data SI provides. However, when reading them together, they seem to offer very similar results. It would make sense that if 58% of U.S. adults agree that most farmed animals are treated well, then those same adults would believe they are purchasing humanely-raised animal products. Also note that for the reported 75%, SI did not include those who responded “no opinion” when summarizing these results. Since 20% of U.S. adults responded “no opinion” to this statement, the overall percentage of those who agreed with the statement is 60%.23 SI offers two distinct conclusions for these statistics, and, given that these results have similar implications, the conclusions could apply to either. For example, if U.S. adults think most farmed animals are treated well simply because they have a lack of knowledge on the topic, then they may believe the meat they are buying is humanely-raised because they lack knowledge of the fact that it is likely not humanely-raised. Similarly, if U.S. adults think this way due to cognitive dissonance, this could be the same reason they believe that their meat comes from a humane source. SI suggests the latter statistic may be due to a “psychological refuge effect.” While they do describe this concept in a couple of words, there is no reference provided for the reader to further investigate.24 There seems to be room for SI to expand on the conclusions they draw here, provide references for the psychological concepts used, and better explain their reasoning for suggesting different conclusions.

Conclusion

The most surprising result from this survey is the large proportion of respondents who seemed to be in favor of banning factory farming, slaughterhouses, or animal farming. Though the 2018 replication certainly helps to validate SI’s results, the apparent suboptimal wording of the personal choice question still remains a limitation. If SI is to repeat this survey without any changes, it may be helpful to further explain their original design choices and analysis in order for those following this research to better understand its implications. If they do address some of the limitations, it will be important to explain any divergences from their original design—so as to not detract from the usefulness of a repeated survey as a way to track trends over time.

SI’s survey had several important methodological strengths, including census-balancing, pre-registration, attempts to avoid social desirability bias, and a disqualifying question to ensure that responses used in the analysis were from individuals who had carefully read each question. Ultimately, given the methodological strengths of the survey—even in light of the limitations mentioned—we feel that this research has the potential to be an important asset in the assessment of the extent to which the animal advocacy movement as a whole is accomplishing its goals.