Introduction

In this blog post we respond to a persistent critic of ACE’s work. Mr. Harrison Nathan has expressed concerns about our research and communications—in a recent online post he criticized our focus on effectiveness, our intervention evaluations, and our analysis of corporate outreach. We’ll focus on those three topics here.

We are a small staff of dedicated advocates and Researchers who strive for maximal transparency and open communication with our followers. For this reason, our email addresses are readily available on our website. Anyone who wishes to engage with us directly about our work is welcome to email any member of our staff; we will typically respond within 24 hours. We track online mentions of our name, but it is not our policy to respond directly to every mention—especially when we feel that most of the information we would provide in response is already available on our website (for instance, in our descriptions of our evaluation criteria, our cost-effectiveness estimates, and our policies). Still, we appreciate Mr. Nathan’s thoughtful and critical engagement with our work. We have decided that it may be useful to provide his recent criticism with a more detailed response, in order to allay certain misconceptions.

Table of Contents:

Our Focus on Effectiveness and Other Ideological Concerns

ACE’s goal is to find and promote the most effective ways to help animals, whatever those may be. We strive for a world in which nonhuman animals live the best lives they can and suffer as little harm as possible, and we work to promote interventions and charities that we think will bring us closer to that ideal.

Given the limited resources of the animal advocacy movement—as well as the many ways we

might try to help nonhuman animals—we feel it is important to focus attention on areas where our efforts can be especially impactful. In particular, we believe that the vast scale and horror of industrial agriculture of farmed animals, its comparative tractability, and the relatively few resources brought to bear against it, make it an especially important arena in which to advocate for animals. For the same reasons, we support recent efforts to advocate for farmed fish—whether on the part of specialized charities, charities that work against many forms of factory farming like the Albert Schweitzer Foundation, or charities that promote alternatives to fish products like The Good Food Institute.1

Mr. Nathan suggests that there are certain charities which we would “probably never recommend” because of their stance “in support of animal rights as opposed to utilitarianism.” On the contrary, we consider any animal charity that we think is plausibly an impactful giving opportunity. It is true that many of our currently recommended charities include a commitment to reduce animal suffering as much as possible in their organizational philosophies; we find that organizations who actively try to maximize their effectiveness often adopt promising strategies for helping animals. We do not think, however, that only those organizations can be among the most impactful giving opportunities. We evaluate charities based on the outcomes we expect they can achieve, not whether they toe a particular ideological line.

Some charities can be highly effective in expectation by working for longer-term goals instead of more immediate improvements. Such charities can help bring abhorrent practices to light, galvanize the animal advocacy movement, and, at their best, achieve systemic change. Some such charities focus on securing animals’ rights. We recommend the Nonhuman Rights Project (NhRP) because we believe their work is a promising long-term route to a better world for nonhuman animals. NhRP’s campaigns (which currently focus on extending legal personhood to captive chimpanzees) seem unlikely to cause significant improvements in many animals’ lives in the short term. However, we feel that achieving legal rights for animals is sufficiently important that even slightly improving the likelihood of that legal reform has a high expected value. ACE also currently recommends The Good Food Institute and New Harvest, both of which are working towards long-term systemic change—although their efforts are also likely to reduce farmed animal suffering in the shorter term.

When deciding whether to recommend charities to donors, one of our major considerations is the existence of evidence indicating that they would be able to make good use of new donations. Historically, our recommendations direct several hundred thousand dollars per year to each of our Top Charities.2 As such, we generally restrict ourselves to recommending charities that have indicated a capacity for absorbing funding on this scale.3 The size of the charity can be an important indicator of this capacity, since smaller organizations may struggle to expand their operations in response to a major increase in donations. We also consider other evidence for charities’ room for more funding: in particular, whether they have concrete plans for expansion, and to what extent their track record indicates that those plans are achievable. In some cases, as Mr. Nathan notes, these factors may give us confidence that a small charity is capable of handling more donations than it is accustomed to.4

Our Leafleting Report and Use of Data

Mr. Nathan has been publicly critical of our 2014 leafleting report. Some of his concerns are addressed in our December 2016 blog post about common critiques, as well as our more recent general explanation of our use of data in making cost-effectiveness estimates.5 We will not detail much of our early thinking on leafleting here, as that thinking is documented in our 2014 report. We have stated clearly that the initial report does not represent our current views and publicly acknowledged that it was a mistake to wait until 2017 to update the report. We are currently working on an update that we intend to publish later this year.

We expect that the updated leafleting report will improve on our original leafleting report in a variety of ways by including (amongst other things):

An in-depth discussion of the limitations of the available animal advocacy studies relevant to leafleting and the quality of the evidence that they provide

A meta-analysis of the key results from what seem to be the most pertinent animal advocacy studies relevant to leafleting

A detailed assessment of the various risks of bias in those most relevant animal advocacy studies

A greater discussion of relevant literature from the social sciences

Information from multiple case studies, as well as interviews with advocates who have a considerable amount of experience with leafleting

Because the field of animal advocacy research is quickly developing, we expect that our intervention reports might often go out of date within a few years of their initial publication, and plan to update them as needed.6 We now publish the date that each report was last updated at the top of each page.

The cost-effectiveness estimate in our 2014 leafleting report was derived from our assessment of the results of the 2013 Farm Sanctuary/Humane League Labs study (FS/HLL 2013). Mr. Nathan has stated that he thinks the outcome measure that we used to assess the results of that study lacked validity.7 One possible point of disagreement we have with Mr. Nathan is on the appropriate treatment of validity: we treat it as a continuous rather than a binary construct and he may treat it as a binary rather than a continuous construct. So while we agree with Mr. Nathan that the outcome measure in question wasn’t ideal, we believe that it possessed sufficient validity for it to truthfully describe important features of the results from the study in question. The outcome measure in question was able to importantly describe of-interest features of the reported data by accounting for the equivalent number of new lacto-ovo-vegetarians for farmed animals to some extent.8 Another favorable aspect of this measure in our opinion at the time was that it didn’t rely on the accuracy of self-reported consumption data as much as some other possible measures; it only relied on respondents to accurately report whether they completely avoided a product—rather than to precisely describe how much they had changed their consumption. Still, we didn’t pre-commit to the use of this outcome measure, and for that reason we don’t entirely rule out the possibility that our decision to use it was importantly influenced by some subconscious bias or biases.9

Mr. Nathan has also been critical of our use of the results from FS/HLL 2013 instead of the results from our own study to estimate the cost effectiveness of leafleting in our 2014 leafleting intervention report. The staff involved with the report who currently remain at ACE have limited recall regarding the rationale for this decision, and we are hesitant to speculate too heavily. Still, in the interest of transparency we list below what we believe were the likely reasons for why we preferred to base the cost-effectiveness estimate on the results from FS/HLL 2013:

As we note in the original leafleting report, our 2013 leafleting study had a smaller sample size than that of the FS/HLL 2013. All else equal, smaller sample sizes mean one should have wider confidence intervals about the effects of a given intervention.

At the time that we analyzed the results from our leafleting study, we had already completed a cost-effectiveness estimate for leafleting based on FS/HLL 2013. We preferred not to combine the results of the two studies because doing so would have involved a more complicated, time-consuming analysis that would have been more difficult to communicate.

At the time of the analysis, we didn’t think that the leafleting cost-effectiveness estimate derived from FS/HLL 2013 was “absurdly high.” 10 At that time, we were aware of several estimates of the effects of similar interventions that were similarly high. 11

At that time, we were aware of several estimates of the effects of similar interventions that were similarly high. At the time of the analysis, we were more familiar with the limitations of our own leafleting study than with the limitations of FS/HLL 2013.12

In retrospect, it may well have been an error on our part not to base our estimate of the cost-effectiveness of leaflets on our own leafleting study to a greater degree. As we have previously noted in detail, at the time of this analysis our team was very small and our research products were not subject to the same level of internal and external scrutiny that they are now. We agree with Mr. Nathan that there are a number of significant limitations to the results from FS/HLL 2013. Yet even when a study has limitations, it can still meaningfully reduce our uncertainty about the effects of an intervention. We think it is a mistake to assume that flawed research is worthless research. While poorly-conducted studies must be carefully analyzed to avoid misunderstanding their implications, ignoring their results entirely will often leave us less informed.

It is worth noting that we disagree with Mr. Nathan regarding what our 2013 study results showed about the effectiveness of leafleting. Mr. Nathan has claimed that our study “found no effect.”13 In fact, we did identify an effect in our analysis: statistically significant effects on the proportion of those who stopped eating red meat and poultry.14 These effects were supported by a later reanalysis.15 We think it is misleading for Mr. Nathan to repeatedly state that our leafleting study found no effect without acknowledging that our analysis did report the previously mentioned significant differences.

While it’s important to consider all available evidence, we believe it’s equally important to continue improving the quantity and quality of the evidence. Research is crucial to improving animal advocacy, and we are seeking to improve the movement’s research base in a variety of ways, including by administering the Animal Advocacy Research Fund.16 ACE also works continuously to improve our own research, establishing standards such as publicly pre-registering our analysis plans on the Open Science Framework to avoid ad-hoc decisions motivated by the data.17 We have also updated our research review process, assigning “primary critics” to all research content and calling on external reviewers to critique each major project before publication. We recently hired a research scientist with experience in animal advocacy research to pioneer our new experimental research division. We aim to continue improving the quantity and quality of advocacy research so that we can continue to improve our recommendations.

As Mr. Nathan points out, the fact that any given intervention can be implemented in many different ways presents a challenge for our intervention research, but it is a mistake to conclude that evaluating interventions as a class “doesn’t make much sense.” There are many ways to approach evaluating an intervention despite variation in its implementation. Some studies use multiple randomly assigned treatment groups with different versions of the relevant intervention. Some studies test which commonly used implementation of an intervention is most effective, so that other studies can test for the upper limit of the intervention’s effectiveness. All of these studies can provide evidence about the effectiveness of particular interventions across different modes of implementation.

Concerns About Corporate Outreach

The Number and Impact of Campaigns

Mr. Nathan’s assertion that “ACE doesn’t differentiate between policies according to their content” is simply not true. We do not just “tally” the number of corporate commitments each charity achieves.18 We have always considered the details of each policy to evaluate the impact it has for animals. For example, when we evaluated the impact of cage-free policies, we estimated that they improved conditions for laying hens by -25% to 46%.19, 20 In other words, our models are entirely consistent with the possibility Mr. Nathan suggests, that cage-free policies might actually make conditions worse for laying hens.

Since charities are generally consistent in the policies they encourage corporations to adopt, we can assume fairly uniform effects for each animal affected by a given campaign of a given organization. This is not the same as assuming the effects of all corporate concessions are equal. For a given organization’s campaigns, our primary considerations are the effects of the specific changes negotiated for, the number of animals they will affect, and the organization’s responsibility for these changes being adopted. As Mr. Nathan says, “a serious analysis [requires] examining the substance of policies.” Since policy demands are often fairly standardized across organizations and they tend to draw on relevant research, we believe it is well within ACE’s capacity to conduct such analyses. We also think it is possible to estimate the degree to which a given campaign leads to changes that would have happened regardless—though it may add significant uncertainty to our analysis.

Counterfactual Considerations

Mr. Nathan suggests that we overestimate the responsibility that charities bear for convincing corporations to commit to animal welfare reforms. In particular, he is skeptical that The Humane League (THL) was 90% responsible for the decision by United Egg Producers (UEP) to stop culling baby chicks. The new technology is more efficient, he argues, so groups like UEP would have eventually adopted it anyway.21

When we model the cost-effectiveness of a corporate campaign, we always consider the possibility that the corresponding policy change might, at least in part, be due to outside factors. We account for this possibility when we estimate each charity’s “responsibility for changes.” We estimated that THL was between 70% and 99% responsible for UEP’s recent pledge, since it was the only charity negotiating with UEP and in ovo sexing commitments were almost unprecedented at the time.22, 23

The number of corporate commitments that occur soon after specific-issue campaigns seems too great to be merely coincidental. Gestation crates were acknowledged as an issue by industry in the early 2000s, but it was not until the Humane Society of the United States ramped up their campaigns against gestation crates in 2012 that we saw a wave of corporate commitments to ban them. Similarly, there were a few early commitments to improve conditions for laying hens, mostly from companies with a strong welfare component to their branding. Then there was a wave of cage-free policies drafted by dining services providers (first at individual locations and then throughout their operations) which coincided closely with campaigns by THL. Following this, there was a wave of policy commitments through other sectors, again coinciding closely with campaigns by advocacy groups.

To our knowledge, corporations have only rarely committed to policies that improve welfare unprompted, even when such changes appear to benefit their bottom line.24 For example, the PETA page Mr. Nathan quotes regarding the profitability of controlled-atmosphere killing (CAK) also claims that “despite the clear benefits of CAK over electric immobilization and its minimal ROI [return-on-investment] time, North American poultry companies have been slow to adopt it.” We are not aware of any major effort on the part of the animal agriculture industry to implement any of the reforms that animal advocates might lobby for. As a result, we find it unlikely that companies which adopt a policy just after becoming the target of a campaign would have adopted the same policy at the same time without those efforts.

Still, we assume that each commitment only impacts animals for a limited amount of time (usually about one to 15 years, with an estimated mean duration of about 5 years). This is consistent with the possibility that outreach campaigns simply cause corporations to make policy changes sooner than they otherwise would have.

Uncertainty Regarding Adherence

The fact that corporate commitments are nonbinding is a concern for us. The history of corporations making these pledges on a large scale in the U.S. is quite short. We hope to have a better understanding of the long-term impact of these commitments in a few years. Groups like Compassion in World Farming are already working to track and follow up with corporations, and we’re watching to see what happens.25 Part of our motivation for assuming that most commitments will lead to between one and 15 years of impact for animals includes our countenancing the possibility that corporations temporarily adopt new practices in response to campaigns, but eventually return to their previous practices.

Mr. Nathan is particularly concerned about the wording of the UEP’s pledge to use in ovo sexing technology to eliminate the culling of male chicks, which states they will do so “by 2020 or as soon as it is commercially available and economically feasible.” He claims that this technology “was developed by industry because it is more efficient than the current method,” and seems to believe that the poultry industry would likely adopt it regardless of THL’s work. In reality, the technology is still very much in development, and it is unclear if or when the poultry industry would be incentivized to adopt it on their own. Furthermore, though they may end up being beneficial to the poultry industry in the long run, most egg sexing projects are not being developed by said industry.26 While they have seen some success in lab settings, none of the in ovo methods we are aware of have been implemented on an industrial scale. Considering the speed, accuracy, and cost effectiveness required of techniques used at a large scale, we think it quite possible that the technology will not be viable by 2020. (This would have been an even more plausible possibility when the pledge was made in June 2016, prior to the publication of several recent advances.)

Given the situation, we believe THL’s approach was reasonable. Once in ovo sexing becomes available on an industrial scale, the pledge gives THL grounds to campaign against UEP should they not adopt the technology—especially after 2020. Since galvanizing public outrage about the harms of chick culling seems particularly tractable, we think the threat of a campaign gives THL significant leverage to demand that UEP replace culling with in ovo sexing, even if it proves to be more expensive.27

The Impact of Broiler Reforms

Animal charities have begun to push for companies to use breeds of broiler chicken that experience higher welfare; requirements to do so are now part of many charities’ broiler campaigns. However, due to the selection practices historically implemented by breeders, most breeds that have higher welfare are also slower to grow than the fast-growing breeds currently being raised.28

We do not believe that corporations will slaughter these slower-growing chickens at a lower weight than faster-growing chickens. Rather, they will likely allow the slower-growing chickens to live longer so that they can be slaughtered at the same weight as their faster-growing counterparts. In other words, we believe these reforms might cause chickens to live slightly better but slightly longer lives.

We understand the concern that these broiler reforms will still reduce the net welfare of broiler chickens, due to their longer lives being spent in poor conditions. In fact, we accounted for this possibility in our 2016 cost-effectiveness estimate for Mercy For Animals (MFA). We estimated that the commitment MFA secured from Perdue would improve conditions for broiler chickens by -25% to 40%, noting that a “major risk” of the commitment is that “the additional days of suffering due to slower growth” might outweigh the welfare improvements.29 In other words, we think it is likely that adopting these broiler reforms will cause an overall improvement in the welfare of chickens, but we do not assume this to be the case.

While the evidence on the matter is not conclusive, we believe it is unlikely that broiler reforms will lead to a greater number of chickens being raised by the poultry industry. As Mr. Nathan notes, a recent industry document claims that 15% more slow-growing chickens would have to be raised to yield the same amount of ‘prime’ meat, even if they were slaughtered at the same weight as faster-growing chickens.30 However, an older Poultry World study indicates that slow- and fast-growing broilers slaughtered at near the same weight yielded very similar proportions of leg, breast, and wing meat. The evidence and reasoning behind the industry report is unclear, so it is difficult to assess its claims—though there could well be differences between the study’s conditions and prospective industry procedures.

There are also reasons to think that poultry reforms might reduce the number of broiler chickens raised. The Poultry World study suggests that higher-welfare chicken breeds will have lower mortality rates than the broiler chickens currently being farmed. The aforementioned industry document also indicates that the new breeds could lead to cost increases as high as 49% due to their longer lives and increased metabolism.31 This could lead companies to raise the price of chicken meat, which has the potential to significantly lower demand; price elasticity research from 2010 indicates a poultry price increase of 49% would reduce demand by between 22% and 45%.32

Overall, we feel that our weighting of the potential risks and benefits of broiler reforms has been reasonable and supported by the available evidence. We think the likelihood that reforms will lead to more chickens being raised is far lower than Mr. Nathan seems to believe, but our models still account for that possibility. Our estimates of the effect of broiler reforms involve a significant amount of uncertainty, and will benefit from further evidence; at present, we believe these programs are promising on balance.

Closing Thoughts

As an effective altruist organization, one of our fundamental values is a willingness to revise our approach based on new information. ACE is not wedded to the notion of any particular organization or intervention being effective. What matters to us is finding and supporting organizations that help animals through evidence-backed, impactful work. We are committed to updating our evaluations regularly because the landscape of charities constantly evolves, as does our own thinking. We believe that our current recommendations are exceptionally strong giving opportunities, but we are always looking for new charities that are even better at helping animals.

As we’ve written about at length, our conclusions are often uncertain. In particular, our cost-effectiveness models sometimes must rely on estimates and subjective judgment calls. They provide ranges of expected value, but should not be assumed to literally calculate the amount of good that charities or interventions do. We have sometimes struggled to communicate the proper level of caution to take in interpreting our estimates. To better indicate the uncertainty of our cost-effectiveness estimates, we now express them as subjective confidence intervals33 (SCIs) rather than single values. Expressing estimates as ranges helps communicate not only our degree of uncertainty, but the simple presence of uncertainty—making them less prone to misinterpretation.

We continue to make and publish quantitative cost-effectiveness models because they support our mission, they are useful for making direct comparisons, and they increase our transparency. Our models help us assess the implications of current research for particular interventions and determine what further research would most influence our future recommendations. They also allow our readers to identify specific points of contention. If our evaluations were entirely qualitative in nature, it might be harder for people who disagree with us about the effectiveness of a program to pinpoint the source of their disagreement, since our qualitative statements are more open to interpretation than structured, quantitative ones.

There is room for reasonable disagreement about some of the estimates we use in our CEEs; that’s precisely why we publish our full models and provide all of our reasoning. Any reader who disagrees with one of our estimates is free to take that into account when they consider where to donate. We invite donors to trust us, but we also provide all of our reasoning so that donors need not simply trust us; they can read our models and make their own decisions.

We are grateful to anyone in the animal advocacy community who draws our attention to potential problems with our thinking or our recommendations. Since our entire staff is highly passionate about animal advocacy, we are happy to have the opportunity to discuss it with any interested parties. We strive to give timely responses to each individual, but need to direct our limited staff time towards research projects as well. So although we may not always be able to compose formal detailed public responses, anyone who wishes to engage us in conversation is always welcome to comment on our blog posts, respond on our Facebook page, or directly email any member of our staff.