How scientific research can become weaponized (despite intentions otherwise)

Jacob Z. Hess, Ph.D.

Note: My writing focuses on the cultivation of greater space for open, productive disagreement – something essential to both scientific progress and healthy public discourse. I was invited by Kendall Wilcox earlier this year to participate in a panel asking questions to John Dehlin about his research, but was unable to make it. Since that time, my concerns about Dehlin’s work have multiplied as I’ve watched more of his public commentary. Due to the serious nature of my critique, I have worked hard to be thorough in my review of relevant papers and presentations – this, in order to provide adequate illustration for how and why I have reached these conclusions. While increasing the length substantially, I decided these footnote references were important to engage Dehlin et al.’s claims openly and transparently. Given the vast imbalance in political diversity documented within psychology right now, I had little confidence that a robust critique of Dehlin’s work would be given space in an academic journal – and opted, instead, to prepare this for a public audience. While Dehlin and his colleagues will undoubtedly disagree with my conclusions (and see them as unfounded, inaccurate and unfair), this is the nature of public deliberation about contested truth claims. Even so, I reached out to Dehlin’s team prior to publication for clarification of some lingering questions and a commitment to correct anything they see as technically inaccurate (in terms of methodological descriptions or details).

John & I subsequently met for a breakfast discussion allowing a chance to ask questions face to face. I found the discussion quite helpful and came away better understanding some additional context I have incorporated below (new edits marked in blue). Three takeaways for me:

(1) John insists (and I believe him) that he does not want his research findings to be used to pressure people who identify as gay or same-sex attracted. Rather than conceiving this weaponization described below as conscious and deliberate, then, I reiterate here the important point already underscored in the essay that much of this pressure involves inadvertent, unintended (but arguably still real) implications of certain ways of approaching research and its presentation

(2) The context out of which Dehlin’s study emerged was a time in which decades of therapeutic approaches involving (what many came to experience as) some level of subtle aggression remained widely accepted. Critically examining some of these trends was part of his motivating intention — something that seems both understandable and valuable. John and I share a concern with any sort of “fixing, forcing” aggression used in therapy — subtle or otherwise — and the crucial place of mindful, authentic space to allow exploration and choice.

(3) As a dissertation study, Dehlin’s work had the resources available to a graduate student (aka limited). Especially considering that context, I agree with John that he made substantial and earnest (and yes, impressive) efforts to generate a diverse sample — a point I had not adequately acknowledged in my original article. That being said, sample diversity was never my primary concern, as illustrated in the article itself; despite all the attention sampling receives, it’s only one of many nuanced questions in the structure, design and presentation of a research project. On that question — as with other nuanced issues — I believe John and his team did their best in making a decision that felt right to them. That there remains dispute on various judgment calls should not be surprising to anyone who does research or knows the history of science.

As a final acknowledgement, it also seems important to point out that the issue of homogeneity/heterogeneity within research teams is a widespread pattern — and certainly not limited to this instance. As the world becomes more fragmented and tribalized, greater attention is necessarily being paid to this aspect of sensitive research questions by scholars across the political spectrum.

Following our initial conversation, I’ve made adjustments throughout – including to the title (which John’s feedback confirmed to me could too easily be construed as an attack on his character). I will continue to update this as I learn more from John. With a number of mutual friends between us, I could imagine John one day becoming a “trustworthy rival.” In the meanwhile, I acknowledge this as my own attempt to hold Dehlin and his team accountable for, primarily, the nature of their public commentary and secondarily, the particular approach to research that generated the distinctive data on which their conclusions draw.

Executive Summary:

Over recent years, conclusions from survey research led by John Dehlin have been promoted as authoritative indicators regarding the experience of LGBT-identifying Mormons and former Mormons. In the absence of any substantial public scrutiny and critical examination of their approach, these conclusions have exerted an oversized impact on public discourse and been received by many as timely, crucial guidance in personal and family choices.

Closer examination of the process by which Dehlin et al. gathered and analyzed data, however, shows clear evidence of a confirmation bias that I argue tangibly influenced the generation of distinctive results (and subsequent generalizations about those results). Rather than intentional manipulation or deception, I make the case for how and why this influence most likely arises (as it does in other research areas) from the ideological uniformity of the research team relative to the questions at hand.

Despite intentions otherwise, both Dehlin’s conclusions and the way he has framed them publicly have heightened an already constrained, pressurized public conversation about otherwise sensitive, important questions. In particular, I provide illustrations for the argument that certain aspects of Dehlin’s presentation potentially increase pressure on gay/SSA-identifying individuals to step decisively away from faith communities, marriages and personal commitments they had once cherished.

As a way to foster more collective, critical inquiry relative to all of these issues, key concerns are summarized, with a focus on details most likely to have influenced Dehlin’s own findings. Rather than “de-bunking” these results, the aim here is to invite greater attention to the complex process by which researchers arrive at whatever conclusions they reach—making, as well, space for a public conversation about contrasting interpretations of the same study or data, which never speaks for itself (not without a human interpreter!)

Background. Last month, the LDS Church released an updated, dedicated website offering support and encouragement to Mormons who experience same-sex attraction or identify as gay. Central to the Church’s attempt at improved ministry were a series of interviews highlighting various ways people had come to relate to unique sexual attractions without, at the same time, divorcing themselves from their faith community.

One takeaway message from the Church’s effort was the possibility of peace and happiness available for those who opted to remain actively involved in seeking to honor Latter-day Saint faith commitments.

Sounds simple enough…right?

Not so fast. In a high profile report from the Washington Post the day the website was released, John Dehlin expressed some forceful warnings. These personal accounts featured on the website were, by his estimation, “dramatic exceptions” standing in sharp contrast to the “vast majority” of people who found such a reconciliation of faith and sexuality to be “untenable” and even “toxic.”

Wow! On what basis could one individual speak with such boldness and authority? And how could Dehlin be so confidently dismissive of the likelihood of happiness for gay, active Mormons?

Well, because the science said so …. his own science, that is.

Two years prior, when Mormon men with same-sex attraction living happily in traditional marriages were featured in a television special, Dehlin underscored analyses from this same study as evidence that others with similar experiences could not realistically expect to find happiness in pursuing an orthodox Christian marriage. Backed by a dizzying array of seemingly decisive statistics, Dehlin urgently cautioned the public to not be deceived into believing that a happy life in orthodox Christian marriages would be possible for many.

His own authoritative pronouncements have subsequently been leveraged by many others eager to add their voice to the ongoing public conversation. For instance, after one man (who experiences same-sex attraction) shared a story of his happy marriage to his wife after a Radio West broadcast, one commentator responded as follows: “referring to the isolated incident of your marriage is useless. I’m sure there are mixed orientation marriages that work. Good for them.” While conceding that adults are “entitled to their choices,” this person alluded to Dehlin’s work in cautioning this man against allowing his story to be an example to anyone else, “ESPECIALLY when that is PROVEN to be damaging to most people who cram themselves into it” and “causes incredible havoc and pain in most people’s lives.”

When another individual pushed back at this characterization as overly aggressive and de-legitimizing, another commentator insisted that “No one is attacking your own mixed-orientation marriage, or asserting that your happiness isn’t real.” Instead of attacking him, they were, in his words, simply “reacting against the notion that this [mixed orientation marriage] is a healthy alternative.” He added, “Research overwhelmingly suggests that for the majority, it is not. It’s a dangerous pathway for those who don’t understand its ramifications. In fact… it rarely works.”

As reflected in these comments above (and thousands of others like it), Dehlin’s study has left its indelible mark and imprint.

For some, of course, these empirical claims are important revelations that merit ready praise, extensive dissemination and ongoing application in the service of expanding social justice. After all, isn’t one of the purposes of science to challenge cherished views by subjecting them to scientific testing?

For others, the acceptance and eager dissemination of Dehlin’s claims have been just a bit quick—with interest in, at a minimum, space for more questions, especially this one: How exactly were the researchers’ own cherished views subjected to testing as part of their process?

On that point, I cannot help but be direct: Is it surprising that Dehlin’s research study reached the conclusions it did?

No…it is not.

On multiple levels, it’s hard to overlook the subtle, cumulative influence of various formative decisions Dehlin’s team made in shaping the ultimate conclusions reached. As I argue below, that’s precisely the danger when an ideologically unified research team[1] generates lots (and lots) of data about a highly contested and sensitive topic—choosing to navigate a host of tricky design and analysis issues without any publicly verifiable check-and-balance in their team against their own cohesive convictions about the question at hand.

Emphasis on data-set size. The size of Dehlin’s data set is almost always his first talking point and often how he introduces his “study of 1,612 LGBT/Same-Sex Attracted Mormons.” That sample size is brought up over and over as part of the rationale for why his statements merit special attention and authority[2]—aka, the bigger a research sample, the more reliable, valid and objective it must be, right?

As all researchers know, trustworthy results are quite a bit more complicated than that. Rather than establishing unquestioned validity, large data sets have been long appreciated among scientists as introducing even greater risk for manipulation, confirmation bias and sheer error—both on a statistical and qualitative level.[3]

Sample size aside, literally hundreds of small decisions are involved in the creation and execution of any research study—decisions that have a tremendous, cumulative impact on the results. In a randomized controlled trial of a mindfulness-based class I recently completed (manuscript in preparation), our team was astonished by the seemingly never-ending stream of choice-points needing resolution. Depending on the underlying questions, the measures used, the characteristics of people answering the questions, the analytic decisions and the framing of results, dramatically different conclusions can be reached. I have learned this over and over in my reviews of other research literatures, especially the studies emphasized as “clearly establishing” the “safety and efficacy” of various psychiatric medications. [4]

The intricacy of extensive research projects is boggling enough that even well-intentioned researchers can (quite unwittingly) come to “see” what they have always believed to be the case. Results from survey research are especially malleable to competing interpretation, since by design[5], this approach invokes fairly brief, one-dimensional snapshots of whatever group has agreed to answer the particular set of questions provided. Especially in survey research, small leanings in the question framings, answer options and sample demographics can lead to large cumulative effects on results.[6]

This complexity only multiplies once analysis starts. Out of hundreds of pages of survey responses,[7] for instance, what patterns are highlighted and shared (and which are overlooked), which statistical analyses are performed (and which are not) and how are the hundreds of possible results ultimately packaged and presented (or not)?

Scientific choice-points. Even after responses are aggregated and averaged, different decisions can be made on what constitutes an “especially interesting” pattern or significant result, and whether specific findings can be further accentuated or punctuated by other techniques or devices. For instance, rather than reporting the straight-up “mixed-orientation marriage” divorce statistic documented among those in his sample (51%), Dehlin decided to work up “projected” divorce rates that he speculated on different occasions would likely rise to “75%” or “~70%” or “ at least 69%” in future years (bolding and underlining his own).

It is this projected statistic, rather than the statistic they documented, that has been widely shared by Dehlin’s team and highlighted by media outlets. [8] The other number that gets referenced frequently is the “0% of people” who reported “an alteration in their core erotic attractions” (Bradshaw, Dehlin et al., 2014, pp. 4, 18). As the authors themselves acknowledge in auxiliary supplemental material, the view of change they chose to operationalize excludes other therapeutic outcomes and possible ways of thinking about change, which were (in their words) “not considered to be ‘change’ using this narrow definition” (Bradshaw, Dehlin et al., 2014, Online appendix). In other words, despite the fact that “change” is defined in many different ways by those with same-sex attraction seeking to remain in the LDS church, Dehlin’s team chose to emphasize an especially restrictive definition of change centered on “reversing a nonheterosexual orientation.” [9]

Having made that decision, they went on to discover that only 1 in 1,000 people (“0%”) had reported that kind of change (see Dehlin’s comment below the video). Typically, when a researcher discovers NO variation on what is supposedly a key defining variable (especially one you yourself have operationalized for people), it’s time to go back to the drawing board: “hmmm…looks like we’re not really capturing meaningful difference in the experiences of our sample….maybe it’s time to re-think how we’re framing this variable.”

Instead of doing this, however, Dehlin went on to broadcast far and wide their shocker-finding of “0%” of people reporting change in sexual orientation as a particularly damning empirical reality— aka “Zero percent of participants were able to eliminate same-sex attraction.” In his TEDX talk he accentuated this point with particular emphasis: “Perhaps most amazing of all, our data revealed that zero percent of our sample reported being able to eliminate their same sex attraction… Zero percent !” (emphasis his own).

Ignored in this figure, once again, are the many ways change might be experienced that simply haven’t been acknowledged (or measured) in this study. This is indicative of what Duarte et al. (2015) caution about in a politically homogeneous scientific process; that is, where “liberal values and assumptions can become embedded” into theory, method and research questions to such a degree that the study “make(s) some constructs unobservable and unmeasurable.”

Alongside the mixed-orientation “projected” divorce figure, the resulting “0%” statistic has been a constant presence in virtually all of Dehlin’s subsequent reports, presentations and interviews. Over time, these particular framings of these particular results yields some remarkable persuasive power. As one viewer remarked almost gleefully after watching his presentation, “Just watched this twice. 0% of 1,600 Mormons who are gay found that treatments to change orientation worked. Wow.”

If it’s true that specific methodological and analytic decisions can have a profound impact on ensuing results, this would certainly not be an isolated example. As noted earlier, those who have investigated similar kinds of details about how research on the outcomes of psychiatric medications are set-up, you discover remarkably similar methodological patterns as those outlined above. In particular, definitions of therapeutic “change,” “effectiveness” or “successful outcomes” are often framed in interestingly narrow ways (aka short-term evaluations of symptom/behavior ratings, with hyper-attention to any positive effects). Associated studies measuring those same delimited meanings then regularly (and predictably) find the evidence necessary to “prove effectiveness” (see tangible example with the drug Strattera prescribed for ADHD).[10]

Is something similar at play here?

The dangers of ideological unity. Rather than merely “influencing the findings,” I would argue that these self-confirming study decisions can play a profound and formative role in data generation, interpretation and the eventual creation of the same eventual findings.

And this is also precisely what many have come to believe makes socio-political diversity essential for any research team investigating contested issues. Think about it: How would you feel about a research study on same-sex parenting conducted (exclusively) by Focus on the Family or one on abortion supervised (entirely) by Planned Parenthood or a large pharmaceutical trial bankrolled by Eli Lilly?[11]

Well then, how about a study on the most intimate experiences of members of the Church of Jesus Christ of Latter-day Saints originated, directed, analyzed and promoted by a man well-known, even by his supporters, to have a serious beef against this very same church?

To be clear, this is not to suggest that Dehlin has intentionally manipulated anything. The problem I’m proposing is more subtle—and less sinister. It’s also backed up by emerging evidence for how our deepest convictions inevitably influence what we see, think and believe—even beneath our awareness (see a video summary here). If that’s true, then even the best, most well-intentioned researchers (and research team) can easily fall prey to confirmation bias, simply for not having adequate internal accountability checks against its own shared convictions (especially those relevant to the topic under study).[12]

At a minimum, any of this might give pause to any research team ideologically unified on the subject matter under investigation—especially if those questions are highly sensitive and publicly contested and especially if any member of the team is particularly invested personally.[13]

For instance, one might say, “Hmmm, I’ve got some pretty strong feelings about this … how will I make sure to check this in a publicly verifiable way that helps people trust our results?”

That’s how it’s supposed to work. And perhaps that’s why Dehlin took great pains to point out the inclusion of Bill Bradshaw on their research team as a “former BYU professor [of biology], an active, believing member of the Church, former mission president, stake president.”

On the face of it, Bradshaw’s presence ought to provide the very check needed. However, Bradshaw is elsewhere acknowledged as a “strong ally and advocate for LGBT issues” (aka politically progressive). Thus it’s difficult to see how his participation on a research team of progressive, self-identified LGBTQ allies provides any sort of functional check on shared convictions regarding the questions at hand.[14]

As Duarte et al. (2015) write: “Having common values makes a group cohesive, which can be quite useful, but it’s the last thing that should happen to a scientific field.” Reflecting on this broader tendency in academia, they note that when “left unchecked” a group “can become a cohesive moral community, creating a shared reality that subsequently blinds its members to morally or ideologically undesirable hypotheses.”

Similar to these published concerns about the broader psychological discipline, it’s worth asking: “might a shared moral-historical narrative in a politically homogeneous [research team] undermine the self-correction processes on which good science depends?”

Rather than an isolated voice raising this kind of a question, I’m one of a growing number of psychologists and researchers drawing attention to the consequences of political homogeneity in science. Indeed, Duarte et al. (2015) propose this moment in American society as a “golden opportunity…to take seriously the threats caused by political homogeneity” and concluding that “the case for action is strong” to explore ways that the discipline of social science “can increase its political diversity and minimize the effects of political bias on its science.”

Given all this, I would argue that the socio-political ideological diversity crucial to ensuring key checks and balances in exploring this kind of a contested question simply did not exist in Dehlin’s study. In that vacuum, it’s almost impossible (even for well-intentioned researchers) to avoid the kind of groupthink that has been a fatal flaw for many otherwise promising research efforts.”[15]

Welcoming socio-political diversity. Rather than an outside observer of Dehlin’s work, these concerns arise for me as a researcher and writer myself—focused in many of the same kinds of questions that motivate their team. As an active Mormon with a (largely) conservative worldview, I would never imagine trying to study these kinds of questions without progressive research partners—especially those who are openly gay. Over the last decade of my own work and writing, I have published peer-reviewed articles with 20 collaborators—most of whom are politically progressive and see the world in fundamentally different ways than I. Among my writing and dialogue colleagues are a number of insightful and talented colleagues who identify as gay, lesbian, queer and transgender—and who I consider some of my dearest friends.[16]

Far more than an intense engine of learning, these collaborations have provided a crucial and powerful atmosphere of checks and balances where we can press and challenge each other’s contrasting interpretations: “hey, this doesn’t sound quite right”…”are you realizing how that might sound to conservatives?”…”that wording definitely won’t accurately capture the experience of my own community!”

Unfortunately, the absence of these kinds of subtle safeguards was apparent in Dehlin’s study from the very beginning, when many active Latter-day Saints who experience same-sex attraction refused to complete the survey given various overt indicators of its subtle and implicit leaning.[17] What would these missing voices and insights have added to the results?

At the very least: (More) Nuance. Richness. And balance. In my own analysis of an extensive qualitative data-set focused on a similar set of questions, I have found thoughtful, smart people disagreeing profoundly about virtually every facet of the gay-Mormon experience—from diverging views of identity, the body and sexuality to how they think of faith, suffering, acceptance, love, God and their current and future happiness.

By comparison, it is hard not to see running throughout Dehlin et al.’s publications a remarkably consistent pattern. While some interesting differences are acknowledged in the details, the take-away conclusions from virtually all these analyses hew with striking consistency to his own worldview, values and chosen life path.

Staying in the LDS church? Counseling with leaders? Seeking guidance about sexuality from prayer and scripture study? Prioritizing personal righteousness? Waiting for a covenant marriage? All turn out to be remarkably bad for gay people. Walking away from the same faith community and the same leaders? Letting go of covenant marriages (or future hopes for one) – and becoming sexually active? Why that turns out to be remarkably good for gay people![18]

Any of this may have been an obvious red flag in other areas of study. And public outcry would have been substantial if his team had been ideologically unified around conservative views of identity and faith.[19] Given the nature of his conclusions, however, the general public has been downright “excited” to embrace what Dehlin has announced, with academic journals clearly welcoming of his results as well.[20] Those who have raised thoughtful concerns with Dehlin’s methodology, by contrast, have found those same journals unwilling to publish their critique.[21]

Lest any of this sound conspiratorial and baseless, realize that there is compelling evidence to suggest that “one key type of viewpoint diversity is lacking in academic psychology – political diversity.” So conclude Jonathan Haidt and colleagues who note that in the last 50 years, psychology has lost most of its previous diversity; for instance, between 80 and 95% of social psychologists (depending on survey methods), identify as politically liberal. Haidt et al. go on to underscore how the implications of viewpoint diversity span multiple levels, from the creation of research to its publication: “this lack of political diversity can undermine the validity of social psychological science via mechanisms such as the embedding of liberal values into research questions and methods…and producing conclusions that mischaracterize liberals and conservatives alike.”[22]

This kind of a group-think atmosphere in academia more broadly might explain why it took so long to expose fraudulent data on gay marriage at UCLA in 2015. Listen to the dynamics that surrounded its perpetuation at the time: “LaCour’s impossible-seeming results were treated as truth, in part because of the weight [the co-author]’s name carried, and in part, frankly, because people — researchers, journalists, activists — wanted to believe them. There was a snowball effect here: The more the study’s impact and influence grew, the greater the incentive to buy into the excitement” (see The Case of the Amazing Gay-Marriage Data: How a Graduate Student Reluctantly Uncovered a Huge Scientific Fraud).

To be clear, no one is accusing anyone on Dehlin’s team of fraud here. I am, however, arguing a similar dynamic has taken place in terms of the quick and widespread acceptance Dehlin’s study has received popularly and academically.

The question is this: Is it really deserved?

When you start to add it up—from a study design oriented towards one particular view, to analytic decisions showcasing one particular view, to publication outlets philosophically inclined to validate these same views (while refusing to provide space to critics) —the multi-leveled influence of confirmation bias is hard to not overlook.[23]

If this sounds like a public challenge to Dehlin’s work as a trustworthy guide for our public conversation, that’s because it is. When sufficient checks and accountability are not present within a research project, accountability is supposed to come from the outside—whether from other researchers or the broader public. That kind of open disagreement is healthy for both scientific community and our larger public discourse.

Scientific evangelizing. In the absence of any such push-back or accountability, however, Dehlin’s conclusions have been cited over and over in public discussion over the last year as authoritative pronouncements of the truth of the matter – aka, “the best data out there.” Indeed, Dehlin’s favorite highlights of the study have often been used to show people how naive it really is (“scientifically speaking”) for those who experience same-sex attraction to anticipate growing happiness in their faith (or their orthodox marriage). For example, when the North Star community (of same-sex attracted Latter-day Saints) advertised their 2014 conference entitled, “At Home in the Gospel of Christ,” Dehlin’s statistics were used to create a “Warning Label” meme by critics devoid of any context (see below).

Dehlin himself could have mitigated some of this by speaking more modestly of his study’s conclusions and limitations and being careful to not overstate what was found.[24] The reverse has unfortunately been true—at times, in dramatic fashion.[25] Aside from some modest every-study-has-its-limitations acknowledgments about a non-random sample, Dehlin and Bradshaw, in particular, have been remarkably willing to offer sweeping statements that imply far-reaching generalization and applicability of results.[26] At every opportunity, these findings have been presented in a scientifically straightforward way as if they reflected devastatingly clear conclusions in relation to these remarkably complex and personal questions.

Amplifying the overstatements. As a result, media outlets have unsurprisingly followed suit—passing along the same interpretations with the same generalizing and conclusive tone, accompanied with titles such as: “Study Reveals What Really Happens When Gay Mormon Men Marry Straight Women”/“Study: Most Mixed-Orientation Mormon Marriages Fail” and take-away lines such as “The results? Married gay Mormons are three times as likely to get divorced.”

In one article widely distributed through other national news outlets, a journalist (despite noting the non-random study sample which would not allow broad generalization), stated matter-of-factly that that study found that “between 51 percent and 69 percent of mixed-orientation Mormon marriages end in divorce, well above the roughly 25 percent of LDS couples who split up” and “more than 70 percent of LGBT/same-sex attracted Mormons leave the LDS Church.”[27]

Period. End of story. Without any assistance, the public doesn’t (and won’t and hasn’t) known any better.

Questions for Dehlin’s team. Are you willing to start assisting them, John? Bill? ( really assisting them). Have you seen how others are interpreting and using your own interpretations of findings? Are you at all concerned by that?

I have to wonder whether a part of you isn’t perhaps happy to see many people accepting your own conclusions cart blanche as authoritative, with remarkably little critical analysis of results – especially in relation to the LDS Church itself? One observer stated, “It seems the results of this study are a really good empirical test of the fruits of the church’s doctrine and position on how LGBT people should live their lives.”

As reflected here, the results are received as largely speaking for themselves – conveying indisputably damning indictments of Dehlin’s former faith. After hearing one of Dehlin’s talks, one individual said, “I’m Mormon and this deeply upset me but I’m GLAD it did. Those statistics woke me up to the fact that a church that I love can produce such harmful effects.”

Another commentator added, “I appreciate data and I also know the price one pays by denying data. I look at your data – and I look at church policy – and the phrase that comes to mind is climate change denial. You have overwhelming data moving in one direction, but I don’t really see that there’s been a millimeter’s worth of movement on the ecclesiastical side. I don’t think this has made any difference for the men setting the policies!”

Given your current posture towards the church, John, I can’t help but imagine these kinds of statements delighting you to some degree. Am I wrong?

If so, are you and your team willing to stand up to these kinds of fundamentalist interpretations – and provide more of a public check on this rhetoric yourselves? Are you willing to nuance your own analyses further to make greater space? Are you willing to involve ideologically diverse collaborators on your future efforts?

Here’s another example of how you’ve shaped public discourse over the last couple of years. Rather than hear out and acknowledge the experience of those not aligning with the progressive ideology of sexuality (e.g., using words and narratives those persons themselves would understand), you have often framed these minority experiences in a language and manner reflecting your own ideology. For instance, you, Bill, stated in one presentation, “Although there are some of our participants who self-identify as heterosexual, we believe they are, in fact, homosexual because of other data that they provided about their life experience.” In that particular paper, virtually every instance of an individual sharing their experience of moving forward happily in an orthodox LDS pathway was minimized and explained away in a variety of ways – especially as reflecting an underlying nature that you insisted was (really) bisexual.[28]

I would suggest this as a striking example of what Duarte et al. (2015) characterize as a significant “risk point” of scientific inquiry involving political homogeneity, namely that: “Negative attitudes regarding conservatives can produce a psychological science that mischaracterizes their traits and attributes.”

Why not acknowledge (really acknowledge) alternative explanations and competing narratives? Is it really that hard to make (real) space to understand other nuanced ways to work with same-sex attraction?

Scientific blitzkrieg. As it stands, the corpus of your papers effectively strikes at the heart of the very things most precious to Latter-day Saints: (1) the seeking of a fundamentally changed nature through Christ (2) becoming united in a covenant marriage and (3) being a part of the body of Christ. With laser-like precision, you claim as a team to have empirically documented why pursuing any of these three aims for a gay Mormon is “statistically” not a good idea and “scientifically speaking” more likely to lead a life-satisfaction-worse-than-lupus (definitively! with the best data out there! with 6 journals vindicating everything we’ve done!)

Behold, the weaponization of scientific research.

I acknowledge these are serious claims I’m making about Dehlin’s work. That being said, Dehlin’s team has made equally serious claims about virtually all of the most important aspects of life: faith, health and intimate relationships. To my mind, anyone making this kind of serious claim merits equal amounts of scrutiny. As illustrated above, the concerns I raise are not mere academic quibbles and extend far beyond simple “limitations” of the study. If survey questions (with a certain leaning) are provided to a convenience sample of participants (with a certain leaning), with analysis conducted by a research team (with a certain leaning)…well, it’s not hard to see how it can all add up![29]

Speaking again about the risk of politically homogeneous scientific efforts, Duarte et al. (2015) ultimately caution about “a higher risk of reaching unjustified conclusions” that exists “when most people…share the same confirmation bias.” They later argue “that the collective efforts of researchers in politically charged areas may fail to converge upon the truth when there are few or no non-liberal researchers to raise questions and frame hypotheses in alternative ways.”

In the absence of such political heterogeneity, research teams focused on contested issues will understandably do the best they can. And in sharing results publicly, Dehlin and Bradshaw naturally present themselves as simply objective researchers—insisting on “not advancing a cause” and instead merely “helping inform.”[30] Like researchers are wont to do, they also talk of their data as reflecting a reality that speaks for itself.

But it does not. And it cannot. Data is generated by humans, and requires a human interpreter…always.

In this case, that person generating it, analyzing and interpreting it is… you . So essentially what you are asking your audience to do is trust your judgments about the data, comprising your design, your analysis and your conclusions about this all. Right?

Another way. The reality is that with a few simple modifications of his approach, Dehlin and his team could have prompted a much richer conversation and avoided many of these problems entirely. Indeed, the presence of passion, conviction (and even indignation) may not be a problem on a research team as long as disagreement is present on the same team (aka, people passionate or convicted in the reverse direction). As Rosik and colleagues proposed in 2012: “We believe that the scientific investigation [of these kinds of questions] will have the best opportunity to be advanced when studies are jointly conducted by an ideologically diverse group of researchers”—adding that “such a collaborative approach can serve to provide some degree of counterbalance to the current gravitational pull [from various sides].”[31]

Another research team conducting a similar survey study has decided to do just that. The project leader, Lee Beckstead, is a respected, openly gay psychologist, and has invested hundreds of hours in ongoing, pointed conversation with others of different life experiences and contrasting spiritual, ideological and political views. Even though a solo study would be much more efficient, Beckstead has insisted on inviting many of these same dialogue partners to become central partners in the entire study—including in the design of questions, construction of research methodology and interpretation of results.

Needless to say, Beckstead and his colleagues are doing the heavy-lifting necessary to generate data the public can trust (no matter what their sample size). I will be among those ready to hear (and learn from) what this team discovers—and believe leaders and other active members of the church will feel the same way. Whatever is learned, however, (and this is equally important), I also have no doubt that Beckstead and his team will articulate results in a way that supports a more thoughtful public conversation and that stimulate further inquiry, rather than polarizing or constraining it. (If they confirm everything Dehlin’s team has found, of course, then I owe John some Aggie ice cream!)

My motivation. In the end, this is why I felt a need to speak out. I am among those who believe that those studying highly contested scientific questions have an ethical responsibility to conduct research in a way that ensures appropriate checks and balances against their own strong biases, not to mention sharing findings in a way that promotes more thoughtful public conversation. As documented above, it’s hard not to conclude that Dehlin’s team has fallen short on both counts. To many observers, Dehlin has pursued an approach that (wittingly or not) turned his research conclusions into effective weapons—increasing pressure on individuals (and institutions) who dare to still believe and teach the possibility of finding happiness within orthodox Judeo-Christian norms as a gay/same-sex attracted man or woman.

To his credit, Dehlin has publicly admitted to some of this: “It’s easy when you see these data to want to try and use it like a billy club. And I’ve fallen into that trap of doing that. I admit it, I own that…and I can tell you it doesn’t work and it probably creates psychological resistance on the part of those you’re trying to assault.”

Update: To reiterate, John also told me in our face to face meeting that he is not happy with his research being used to pressure people. I believe him – and see that perhaps we hold a disagreement about what it means to pressure and what kinds of activities or rhetoric might translate into pressure. For instance, for me, the way questionable statistical analyses have been shared in this study as an objective reflection of reality – aka “just to inform people” – represents an especially effective source of pressure; I’m sure John would see that differently.

Real-life consequences. It’s important to point out that the nature of the kind of rhetoric summarized here has real-life implications that go far beyond mere academic disputes. For how many people have the results of this study (contrary to Dehlin’s stated intentions) been used to press people to walk away their faith community or their marriage? How many have been prompted to feel a growing despair about a life, a love or a faith that they might otherwise loved and nourished? How many have relinquished hopes of a future covenant marriage working out after hearing Dehlin’s pointed recommendations—convinced that their hoped-for happiness is simply not possible within the expectations of the church? [32]

I know many who praise Dehlin for helping their family find greater health and well-being. I myself am personally convinced that the consequences of Dehlin’s voice have often been profound in the reverse direction—including for the most vulnerable members of this conversation. Separately, I have written about why I believe the talking points coming from John Dehlin and others have been especially dangerous for teenagers navigating their own sexuality.[33] As one columnist put it, “The numbers behind [Dehlin’s] study tell a depressing story for gay Mormons.” Laurie Campbell noted that despite Dehlin’s better intentions, “treating a limited study as if it applies to a representative cross section and declaring percentages based on such is definitely wrong, and discouraging for many.” Kyle Merkley another panelist invited to comment on Dehlin’s research added, “I just wonder how [your interpretations] comes across – for example, gay youth who want to remain members of the Church. And they want that – they’re seeking that. But because of the way you present your study, they feel totally hopeless being both gay and Mormon.”

Dehlin himself has not been shy to spell out in his own presentations all the reasons that an orthodox Mormon pathway may, indeed, come to feel hopeless for many. In his TEDX talk, for instance, he spoke of LDS individuals with same-sex attraction through the following summary: “So if they didn’t choose it, if it doesn’t go away, if attempting to change it causes harm, if mixed orientation marriages have high failure rates, and if celibacy has incredibly low quality of life ratings, of course they would turn to suicide as what they feel is their only option.”

As I’ve written previously, “this kind of ‘damned if you do/damned if you don’t’ rhetoric may well be contributing to some of the very pain that these teenagers we all care about are facing.” Referring to Dehlin’s own work, I posed the following: “Want to make a teen in the LDS Church despair? Convince them that who they are demands either acting in violation of their most sacred convictions or living a life not quite as enjoyable as ‘those who have lupus.’” Recollecting his own suicidal period, one LDS man recollected, “I figured if I couldn’t have the life I wanted, and the life I could have from what I read wasn’t conducive to the gospel…then my only option was to kill myself” (VH-DEC).

Looking forward. Moving forward, I would challenge Dehlin and his team to consider several possible adjustments in their ongoing analysis and continued interaction with the public:

In future analyses and projects, insist doggedly upon sociopolitical diversity inside your own team. Diversified research collaborations are simply not a luxury when it comes to exploring contested issues, especially in the face of accelerating political homogeneity in the social sciences. As Duarte et al. (2015) notes, “The most obvious cure for this problem is to increase the viewpoint diversity of the field. Nobody has found a way to eradicate confirmation bias in individuals, but we can diversify the field to the point where individual viewpoint biases begin to cancel out each other.” Instead of projections or elaborations or generalizations of your findings, underscore primarily what you found. If opting to continue using the “70-75%” projected divorce rate, emphasize the original statistic too. If choosing to continue citing the “0%” change rate, at a minimum mention the rationale for the narrow[34] way “change” was defined in your analysis, and note that there are other ways of defining or describing change that participants may have experienced that were not captured in the survey. Rather than minimizing the high quality of life finding for those who integrate faith and sexuality, find ways to emphasize it as a way to balance your presentation of results [see footnote 37]. Consider future analyses of your own data that may shed additional light on some of the nuance and complexity around this issue. For instance, rather than only examining whether “change” happens or not, it would be powerful to see a qualitative analysis of the kinds of changes people describe experiencing (I volunteer to join you on that one!)[35] And instead of only examining whether a mixed-orientation marriage fails or not (and presuming that this arises exclusively from the role of same-sex attraction), deepen the inquiry to examine various contributors to the success or failure of these marriages.[36] Related to, but distinct from all the above, you might also look into further ways to expand the transparency of your own research process. Erik Little, a qualitative researcher investigating similar questions in this area, mentioned, “I think it would be great if they put their data up on an academic data source (like the ARDA) so that other researchers could replicate their findings and see their survey/sampling.”

In the end, speaking to Dehlin’s team directly, my purpose here has not been to “de-bunk” everything you’ve done in this project. The battle of de-bunkers is a war that leads nowhere and I’ve only highlighted methodological concerns here that pertain to adequate sociopolitical balance in collaborations.

And to clearly reiterate, I am also not accusing your team of intentionally manipulating findings, doing “pseudoscience” or “junk science” or any of that name-calling that we devolve into when disagreeing about research. I am raising some questions about how you’ve conducted the study, and brought forth results – drawing attention to some implications that many might find concerning. That isn’t to say there aren’t some interesting points arising from the study definitely worth exploring; in fact, I would say the seeds of something that could spark a more productive public discussion are evident in some of the findings you’ve chosen not to emphasize as much.[37]

Even so, because I believe the larger, cumulative impact of your representation of the science has been to constrain an already tortured public conversation, I feel a responsibility to speak out and share my own concern in hopes of encouraging a more thoughtful public conversation that makes (non-coerced) space for different life paths.

To you and others involved, I close with a few questions: Could it be true that (many) who experience same-sex attraction can find happiness within the LDS church and its teachings, while (many) others can find happiness outside it? Help me understand why is seems (emotionally) difficult to acknowledge that some people who experience same-sex attraction enjoy a rich life within orthodox Christian marriage—while others do not?

Rather than trying to silence any particular community, or impose a monoculture where there is diversity, what if we used research findings to open up more thoughtful conversation—including a “greater appreciation,” as you, John, once articulated, “of the range of options likely to produce happiness.” What if we stopped trying to pressure people with same-sex attraction toward (or away from) one particular life or label (even just with “information” or “the best data out there”)—instead agreeing to make authentic space for a diversity of choices in that regard? And what if we could focus our collective energies on helping people find a life authentic to their values and goals, rather than insisting that our own values and goals be pursued?

Yes…what then?

Jacob Hess is the author of 14 peer reviewed articles exploring contrasting narratives of mental health and sociopolitical issues. Jacob has (co)authored three books: You’re Not as Crazy as I Thought, But You’re Still Wrong, Once Upon a Time…He Wasn’t Feeling It Anymore and A Third Space: Proposing Another Way Forward in the LGBT/Religious Conservative Impasse (Disagreement Practice, Treasonous Friendship & Trustworthy Rivalry in the Face of Irreconcilable Difference). His work with Phil Neisser at State University of New York has been featured on This American Life and was recently honored by Public Conversations Project. His many wonderful writing collaborators and dialogue partners disagree in all sorts of ways with Jacob’s religious conservative views, but he loves them anyway! As a proud partner of Living Room Conversations, the Village Square and a long-term member of the National Coalition of Dialogue & Deliberation, Jacob’s life work is dedicated to making space for thoughtful, good-hearted people to find understanding (and affection) while exploring together the deepest of disagreements.

Notes:

[1] Whatever other demographic diversities existed on the team, the evident reality is that when it came to the central questions of the study, Dehlin’s team was ideologically and philosophically unified.

[2] One of the early slides in his main presentation features the sample size, “N = 1,612,” in large bold letters—followed by a second slide highlighting several other previous studies with notably smaller sample sizes: “Our sample is eight times larger than the next biggest sample.” Clearly, this sample size is supposed to matter a lot! Indeed, Dehlin acknowledges that, “Our primary design goal was to have a large, diverse sample.” This size of the sample has subsequently been leveraged in strong imperatives about whose voice should be guiding people’s decisions, such as: “When helping young people make decisions that will have such huge consequences, we should give them the best information available. The experiences of 1600+ people is far more valid than the anecdotal experiences of a few acquaintances or a few high-profile individuals.”

[3] Although little acknowledged in public discourse, these perils specific to large data sets are basic cautions in graduate statistics courses and well-known to any academic. One scientific author explains the challenge this way: “all real data has variation in it, and when you have a very large data set, you can usually subset it enough that eventually you find a subset that, just by chance, fits your preconceived view.” This “presents a very serious problem because to the untrained eye (and sometimes even to the trained eye), they seem to show scientific evidence for [specious] positions, and an enormous number of the studies and ‘facts’ that [people] cite are actually the result of this illegitimate sub-setting of large data sets.”

Related to this, several scholars recently highlighted the unique risks that come with “flexibility in data collection, analysis, and reporting” which “dramatically increases actual false-positive rates.” In many of these instances, they note “a researcher is more likely to falsely find evidence that an effect exists than to correctly find evidence that it does not.” Given Dehlin’s use of numerous data collection measures (while reporting just a few for each publication), there has been related concern about the dangers of this flexibility in coming to find a false significance in certain correlations.

[4] There are many examples of a particular assortment of seemingly small decisions leading to a certain result. For instance, in 1997, Emslie and colleagues published a study that was received as the “first evidence that antidepressants work for children”—making a dramatic and almost immediate impact in justifying expanding these prescriptions for children. After parents began to report adverse effects such as suicides to the FDA, however, the data were reanalyzed. This lead to the discovery that although five different measures were made in the study to compare medicine and placebo (three self-report—parent and child—scales and two clinician ratings), Emslie‘s original conclusions were largely drawn from clinician ratings, which were notably different from parent/child ratings, but judged to be more reliable. A reanalysis giving equal weight across measures reached more guarded conclusions [see Safer, D. J. (2006). Should selective serotonin reuptake inhibitors be prescribed for children with major depressive and anxiety disorders? Pediatrics, 118, 3, 1248-1251].

[5] Compared to in-depth interviews or mixed method studies involving observation, survey research is uniquely positioned to generate data adaptable to various socio-political views. Why? Because human beings are (unsurprisingly) adaptable to these same views. Simply put: if you ask a bunch of people who think a certain way questions about their life, they will provide answers (by and large) that justify and confirm their own current trajectory. Thus, if you survey people using pornography about whether pornography is harming their life, they’re going to tell you NO. If you survey energy drinkers about whether their preferred beverage is hurting them, they’re going to tell you NO WAY. And if you survey religious people about whether religion is a great idea, they’re going to say YOU BET.

In other words, each of these studies might simply be measuring the well-established human tendency towards confirmation bias. That tendency would only be heightened on a topic about which a high degree of public controversy exists – especially given the way that research participants have been known to seek validation by answering in a way that contributes to a positive impression of their lives (even without meaning to). This kind of slight bias can be enough to lead someone to choose a 6 rather than a 5 on a scale, for instance. Given the highly charged public discussion and extremely sensitive nature of sexuality questions, it’s hard to know the influence of social desirability bias on Dehlin’s results since their team did not seem to control for any measure of that bias.

[6] Subtle evident leanings in Dehlin’s study were evident at multiple levels. For instance:

(1) Dehlin and his team talk at great length in their presentations about their efforts to reach out to a balanced sample. Despite this, in part perhaps due to some of the issues below, Rosik and colleagues note that “the sample consisted overwhelmingly of participants who were moderately to highly disaffected from the LDS Church, which again raises concerns about the representativeness of the sample and the response bias this disaffection may have introduced” [e.g., when adding up “inactive” + “disfellowshipped” + “excommunicated” + “resigned” groups = 71.2% of the current sample being disaffected in some way from the LDS church, with 28.8% active].

[In speaking to John about this point, I came away convinced that his interest and efforts to generate a balanced sample were sincere, real and worth seriously acknowledging. It’s not as if (as someone could mistakenly perceive) John somehow started off trying to gather a sample that would generate the findings he was hoping for….I certainly don’t believe that. Rather, he and his team really did go great lengths to try and gather a diverse sample. Whether or not these efforts were successful (and other factors that may have remained inadvertent barriers to that happening), can and will likely remain points of disagreement. But especially for something that was just a dissertation research project, I want to better acknowledge here the sincerity and extensiveness of efforts to reach out].

Another way to think about the imbalance is that individuals with same-sex attraction who are happily married and doing well in the church were less likely to participate. This imbalance is not surprising since, as one commentator pointed out, “happily married couples with one partner experiencing some degree of same sex attraction generally don’t advertise their situation.” Another panelist invited to comment on Dehlin’s work, Laurie Campbell, likewise argued that “people doing well in a mixed-orientation marriage are those not paying attention to [these efforts]. The ones doing the best are the ones you’re not getting reports from.” She elaborated, “we know there are people in mixed-orientation marriages who cannot be reached—those most likely doing well—because they are not in touch with a support group (not in need of support) and word of mouth won’t reach them, either, because those people in the support groups don’t know them either.” Given that, it might be concerning that only 22% of the study participants were reached through more random methods (news sources and web searches). Kendall Wilcox acknowledged the same point, “The couples who are doing just fine and have nothing to prove often remain silent…therefore, we don’t know their numbers and don’t know how their numbers compare to the ‘failed’ mixed orientation marriages.”

While this point is sometimes acknowledged by Dehlin’s team [“it is probable that a significant number of both highly devout and highly disaffected current and former LDS church members did not become aware of or feel comfortable participating in this study” (2015, p. 10), as highlighted later in the paper, this limitation is frequently minimized in the face of detailed descriptions of the effort at balance and glowing summaries of the quality of the data set.

(2) The design of certain questions were also suggestive. For instance, participants were asked to rate their sexual orientation therapy experiences on a 5-point scale, from 1 = highly effective, 2 = moderately effective, 3 = not effective, 4 = moderately harmful, and 5 = severely harmful.” In addition to combining effectiveness and harm in the same scale, Rosik raised concern that “the midpoint of the scale is not effective“—which is problematic given “midpoint response bias, wherein respondents often tend to choose a middle response when they are rushing or when they are uncertain or have no opinion.” The upshot? In a subtle way, this decision about the midpoint almost certainly contributed to a particular portrayal of religious conservative experience. As Rosik summarized, “Certainly outcomes would have been more favorable [to religious conservatives] had Dehlin et al. defined the midpoint as not harmful rather than not effective.”

(3) Rather than selecting scales generally accepted as measures with established fairness across socio-political difference, Dehlin chose to use the Rosenberg Self-esteem Scale. Speaking of this scale, Rosik raises concern that the authors of the Rosenberg scale “define their construct in a manner that is inherently biased against religious values.” As a result, he argues, scores specific to that scale “may reflect differences between humanistic values and theistic beliefs (e.g., elevation of the self versus the virtues of humility and self-negation) more than the construct purportedly assessed by the instrument, which in the present case was self-esteem.” He points out that “research has suggested that when antireligious humanistic dimensions of the Rosenberg scale were statistically controlled, the self-esteem ratings of conservatively religious persons were significantly improved.”

What, again, is the upshot? “The implication for the Dehlin et al. study is the distinct possibility that self-esteem levels were suppressed and might actually have been higher than indicated for participants who remained conservatively religious and therefore were more likely to report positive sexual orientation change experiences.”

Bottom line: No matter how hard you’ve worked to reach a diverse sample, the study still must be set up fairly enough in order to openly inquire about people’s diverse experiences. In the absence of that sort of careful balance, distinct conclusions can and will be reached. In the case of therapy for those with unwanted sexual attractions, for instance, Rosik et al. conclude, “These serious limitations [in Dehlin’s study] virtually guarantee reports of SOCE [therapy] harm are likely to be inflated and accounts of success suppressed.”

[7] Commenting once on the enormous amount of data generated, Dehlin once remarked that it was enough for “30 other dissertations” given the number of other questions and analyses that could be run.

[8] For instance:

His published study shares “projections suggesting an eventual divorce rate of 69%.”

The slides used in presentations state a “~70% projected divorce rate for mixed-orientation marriages.”

In his 2013 TEDX talk, Dehlin cited an even higher number, claiming that “the divorce rate for mixed orientation marriages approached 75%”

On a radio interview, Dehlin went even further, suggesting divorce rates could likely be even higher than projected rates because as people see more and more happy people in same-sex marriages, they’ll be even more likely to divorce. [Sometimes this figure has been misreported as being as high as 80-90% by those who would seek to “use” Dehlin’s study to discount or “warn” against other approaches or responses to same-sex sexuality].

In fact, I’ve never personally heard Dehlin present to the general public anything other than his projected statistic. As if this was not a dramatic enough statement, Dehlin’s team also chose as a comparison statistic one of the lowest published divorce rates on record (linked to high education/high income groups), rather than comparing that number to an analogous figure for mixed-racial or mixed-religious marriages. By comparing his own projected statistics to another statistic widely believed to understate average divorce rates, Dehlin commonly claimed that “those who enter into mixed-orientation marriages are up to 300% more likely to get divorced than the U.S. average ” (bolding and underlining again his own).

This high projection of a statistic has subsequently been used to deride orthodox conservative possibilities – e.g. this comment parodying what Dehlin sees as the harmfully delusional interest to invite others to seek orthodox marriages: “There’s a 1 in 4 chance of a mixed orientation marriage being successful– but you should go for it anyway because some people can make it work.”

One could argue, as an anonymous contributor did, that this kind of a “use of estimates of divorce rates given their findings has no statistical grounding and is speculation.” Speaking to Dehlin’s theoretical explanation for the projection (that since the marriages are an average of 16.7 years, those divorce rates will increase once the kids are gone), Laurie Campbell adds: “For all we know, if they’ve made it that far given the opposition, then they’re likely to stay intact.”

I would call this projection a salient example in Dehlin’s study of what some authors call a “questionable research practice,” which are remarkably common across all studies and can have an especially large impact in areas of study that are highly contested [See 2015 article, “Navigating treacherous waters—one researcher’s 40 years of experience with controversial scientific research”].

[9] Noting how “the study concentrates on whether or not sexual orientation has changed,” one woman with same-sex attraction living happily in a marriage with her husband said that while she knew many who experienced positive changes in therapy, she didn’t know anyone who used language saying their “sexual orientation has changed” (which she considered a strangely outlier way of discussing change in therapy).

Reflecting on the current public discussion, Laurie Campbell adds, “Please, can we all just move beyond making an argument about whether or not ‘sexual orientation’ can change and agree that it’s a bad idea to frame ‘change’ that way?” After sharing her own experience of finding with her husband profound levels of personal, emotional and spiritual intimacy (without her “sexual orientation changing”), she raises concern about one moment in Dehlin’s presentation to PFLAG: “At one point, he says—while mockingly waving his arms about— that one or two people stand up and say, “Hey, I’ve changed. And they’ll be the ones to say, ‘hey, you can change, too.’” She remarks, “Josh Weed, Ty Mansfield and I have probably been the most public over the years, and none of us claim to have ‘changed sexual orientation.’ None of us say, ‘Hey, you can change, too.’ Sure, we’ve experienced changes. Most people change, especially faithful Mormons striving to live any gospel principle they find challenging. Liz Diamond talks about that in her book, ‘Sexual Fluidity.’ She quotes a woman who had identified as lesbian all her life, and been involved with women, but then falls in love with a man. The woman ‘continued to consider herself unlabeled, and she emphasized that for her, love revolved around the person as opposed to his or her biological sex.’ That’s how it happened with my husband. Also, the spiritual attraction was more intense than anything I’d ever experienced with a woman. It’s tough to explain, but the spiritual attraction, becoming ‘one heart’ with my husband, is the strongest connection that leads to our physical intimacy, becoming ‘one flesh.’ I was solely attracted to women at one point in my life, and now I am attracted to women, and my husband. Gratefully, my attractions to women no longer create distress nor are they a source of conflict for me. That is change, for sure.”

The point in sharing all this is that Laurie’s kind of experience and perspective was much less likely to be heard and acknowledged in Dehlin’s study – simply by virtue of how it was designed and pursued.

[10] Similar to the child Prozac study in footnote #4, here’s another example from my 2011 review with Jeff Lacasse at Florida State University: Under the Freedom of Information Act, Cohen et al. (2009) obtained all 11 studies originally submitted to the FDA for approval of the ADHD drug Strattera. In dissecting the details of these studies, these researchers found that the average number of measures for positive drug efficacy exceeded measures of possible adverse effects by a significant margin: While there were between three and seven measures of positive short-term drug effects across studies, most studies had few, if any, rigorous measures of negative drug effects: In nine published reports, measurement of adverse effects was elicited only via “self report” (one study), “spontaneous reports from parent or child” (one study), “unsolicited adverse event reports” (two studies), and “open-ended questions” (four studies). One study did not report on how measurements of adverse effects were collected. (p. 325) In light of such evidence, Cohen et al. (2009) decried a troubling pattern of “selective presentations and publication of adverse effect data,” wherein “published results . . . tend to distort or conceal negative findings and emphasize positive findings.” Based on this review, they went on to share an overall impression that “ascertaining harm from treatment takes a distinctly subordinate position to the goal of establishing superiority of a tested drug to placebo” (cited in this book, pp. 316, 320, 324).

[11] This doesn’t mean, necessarily, that research should be dismissed simply because it was sponsored or supported by an institution or industry that deeply cares about the results. When scientific standards are met, even research closely tied to a passionate agenda may provide valid and legitimate results. As I have reviewed in the context of mental health research, however, there is abundant evidence these standards are too often not met. As a result, Lexchin and colleagues (2003) found that industry-funded studies were more likely to reach outcome conclusions favoring the sponsor’s product, when compared with research independent of corporate interests (OR 4.05, 95% CI 2.98 – 5.51).

[12] Jonathan Haidt and collaborators have issued clear warnings about this trend. In his book “The righteous mind” in 2012, Haidt underscored the crucial necessity for genuine diversity of perspectives: “In the same way, each individual reasoner is really good at one thing: finding evidence to support the position he or she already holds, usually for intuitive reasons…This is why it’s so important to have intellectual and ideological diversity within any group or institution whose goal is to find truth (such as an intelligence agency or a community of scientists) or to produce good public policy (such as a legislature or advisor board)” (p. 90). In a subsequent collaboration with José Duarte at Arizona State and several others, Haidt cautioned in 2014 (in a quote worth repeating) that, “If left unchecked, an academic field can become a cohesive moral community, creating a shared reality that subsequently blinds its members to morally or ideologically undesirable hypotheses and unanswered but important scientific questions” (p. 8). Likewise Chambers and colleagues warned in a Psychological Science article around this same time: “To the extent that social scientists operate under one set of assumptions and values, and fail to recognize important alternatives, their scientific conclusions and social-policy recommendations are likely to be tainted” (p. 148).

[13] It’s worth reiterating here an important point: to be passionate or convicted about something does not in itself disqualify associated research efforts. It could be argued that central to any research endeavor is inherent interest and conviction on the part of researchers originating the project. The issue is not the presence of personal conviction, as much as what is done with that conviction: how transparent individuals are about it, how willing they are to “check” their own biases both internally and publicly.

[14] John clarified in our face-to-face meeting that by no means are they all a team of “anti-Mormon” scholars, as sometimes insinuated by critics. Not only was John himself an active member of the church “teaching Elders Quorum” when they started, but Bill Bradshaw has been actively involved in his face community his whole life. And Renee Galliher has never been a member of the church. This seems to be a helpful qualification, reinforcing the fact that there is some legitimate and relevant diversity among them (and an intention on John’s part, as he attested, to involve actual diversity on the team).

All that being said, diversity on the research team relative to the key questions being asked remains absent. As Rosik 2016 summarizes, “to their credit, the study’s authors make clear that they are all ‘LGBTQ allies’ ….and ‘have been active in supporting the LGBTQ community, online, and national/international engagement.” Elsewhere, Dehlin describes Bradshaw himself as an “avid supporter” of the gay community (aka, gay rights). Given this, public insistence on crucial team diversity seems just a bit disingenuous.

[15] It’s hard to imagine a scenario where this lack of internal checks and balances would not yield some kind of self-confirming results. In their Behavioral and Brain Sciences piece, social psychologist Jonathan Haidt and his colleagues made a case for how this same lack of ideological diversity is harming the integrity of social science more broadly—and for precisely some of the reasons outlined here (e.g., multiple levels and “ways in which political homogeneity can threaten the validity” of scientific studies Duarte et al., 2015).

[16] I compliment John’s new effort to experiment with more of this co-equal disagreement practice in his exchanges with Patrick Mason. Had his original podcasts emerged from more of the same, I believe they could have become a profoundly helpful guide to public discourse—rather than what they ultimately have become: interviews that any active member of the church would recognize has heavily slanted tools in ramping up pressure on the LDS church.

[17] Potentially hundreds of people fall into this camp—as estimated by many people I personally know who tried to participate. In this case, their non-participation wasn’t because they weren’t invited, but because when they tried to participate, they were forced to conclude that the design of questions simply didn’t make adequate space to hear their own experience (in a way they felt to be accurate). Right from the outset, there were indicators of this skew to religious conservative participants. For individuals with same-sex attraction who preferred to not identify as “being LGBTQ”, for instance, there were many instances in the survey where there was no option to do so.

One commentator raised concerns how the Kinsey scale was used, prompted by the following question: “What do you call a man who is attracted to only men, and then meets a woman and develops a sexual attraction to her, but still feels no sexual attraction to any other woman at all, and still feels sexual attraction to many men? Is being attracted to one woman enough to make someone bisexual…or is attraction a little bit more complicated than that?” This author continues, “The Kinsey scale deals in generalities, and may not be capable of adequately capturing the distinctions that exist in the lived reality of many in mixed-orientation marriages. An instrument more specifically designed to deal with this group and the situations that frequently arise may be more appropriate. For example, in such an individual does the fact that they are attracted to only a single woman mean that they are only incidentally heterosexual, or does the fact that many of them are sexually monogamous with an opposite sex partner mean they are exclusively heterosexual, and if their partner died would that make them exclusively homosexual or still incidentally heterosexual since there at one time existed a woman that they were attracted to? Thus depending on exactly the questions that were asked in this study, and how subjects interpreted them, the Kinsey scores may be somewhat more difficult to interpret than advertized.”

Citing Dehlin’s report of “starker findings” about Kinsey scores in relation to the likelihood of Mixed-orientation marriage divorce, Laurie Campbell similarly added, “In truth, we do NOT know what it suggests because they did not ask what people still in mixed-orientation marriages would have ranked themselves before they married someone of the opposite sex. Perhaps they, too, ranked closer to a 5 until they found someone they were attracted to enough to marry. And, the men whose mixed orientation marriages broke up could be even more likely to rank themselves with more exclusively homosexual attractions and rejecting of the idea of opposite-sex attractions.”

[18] What are the decisions appearing to be “Correlates to Well-Being for LGBT Mormons and Former Mormons” according to Dehlin? To begin, Dehlin states emphatically that “praying, and fasting, and drawing closer to Christ and being extra righteous” in the context of seeking for help and guidance in relation to sexuality are “negative correlate for health and well-being,” alongside “remaining active in the Church” and counseling with church leaders.

All of these are portrayed as the “most ineffective/harmful methods” of seeking answers and growth in relation to sexuality. Bradshaw states soberly, “Most people who remain affiliated with the church report continual negative mental health consequences” with “active LDS have higher depression scores than those who have disengaged from the Church.” In addition, Dehlin reports that “Interestingly enough [those not sexually active] had the lowest quality of life scores by far.” He adds, “That was interesting….”

By contrast, “leaving the LDS church, either through resignation, disfellowshipping, or excommunication” is reported as a positive correlate for well-being. And “being sexually active” is underscored as positive for health” – more specifically, “becoming sexually active, preferably in a long term, committed, same-sex relationship,” emphasized as follows: “What we found is, by far, those reporting the best quality of life were sexually active and in committed relationships; but even not being in committed relationships you showed a significantly higher quality of life reported.”

These points are underscored again and again. In one single presentation, four separate comments were made in different parts of the discussion to underscore these points. Among them, Bill Bradshaw stated, “The important point, as highlighted in blue, is that the personal efforts turned out to have the highest negative rankings and the last effective ones…those efforts that normally Latter-day Saints would think would be the most effective in doing what’s right and getting God’s blessings, those were those ones least effective.”

In one presentation, Dehlin went so far as to suggest that each and every tangible step away from the Church represents a predictably elevated step in terms of well-being: “We found that, in fact, that…we got a slight bump…Participants got a slight bump if they went inactive, another slight bump up if they were disfellowshipped, and then interestingly enough, a very large bump happened a very large bump happened if they reported resigning from the church or if they were excommunicated.” He concludes, “and excommunicated members actually had the highest quality of life of those who participated. So that was an interesting finding!” [yes, indeed…!]

Slide from Dehlin et al.’s main PowerPoint presentation depicting the progressive “bump” in well-being each step gay-identifying people take moving further away from the Church.

In short, Dehlin pretty much concludes that most everything lining up with an active LDS life turns out to be BAD for gay people – including choices regarding “LDS church disaffiliation (vs. activity), sexual activity (vs. celibacy), and legal same-sex marriage (vs. remaining single or mixed-orientation marriage)” which were all associated by different measures chosen by this research team to be linked to “significantly lower levels of internalized homophobia, sexual identity distress, depression, and higher levels of self-esteem and quality of life.” Each and every such association can now be claimed as having the imprimatur of science.

So what do we make of this? Is there suffering and pain happening among active members of the church with same-sex attraction? How could there not be with the “lines of the culture war running through their own soul” as my friend Arthur would say. Explanations for where this pain are coming from differ sharply, as I’ve written about before (see here and here and here). Certainly exploring, understanding and addressing this pain could be a collective task we willingly participate in together. That being said, if you over-sample people disaffected to the Church, ask them all how helpful and effective prayer, scripture study, bishop support and personal righteousness was, and then average all those ratings to determine a so-called overall “helpfulness” – don’t be surprised if the average ratings you generate “reveal” the “unhelpfulness” of the very thing most of your sample currently sees in retrospect (through a filter of current disdain). That’s precisely the unique weakness of survey research referenced in footnote #5.

[19] To be more clear: If I had assembled a research team unified in a conservative perspective, with a series of research articles raising challenging questions for the progressive gay community from a religious conservative perspectives…would any of these research details been approved and my results allowed to be published? Of course not – and of course people would have seized upon the ideological mono-narrative on this imaginary research team as a fatal flaw.

This is an example of what Dr. Phil Tetlock, now at the University of Pennsylvania calls a “turnabout test,” which is “a simple tool for identifying and avoiding embedded-values bias.” For instance, this might involve someone “imagin[ing] a counterfactual psychology field in which conservative political views were treated as scientific facts and disagreements with conservative views treated as denial or error” (see also: Duarte et al., 2015).

[20] His reception by the broader scientific community is another point Dehlin and Bradshaw underscore frequently—highlighting it a robust vindication of their work and “an important validation of our methodology.” For instance, Dehlin states in one presentation “7 or 8 peer reviewed scientific journals wouldn’t have accepted it if it wasn’t a credible sample – if it didn’t have a solid methodology.” In a written summary, after listing all the names of the journals to publish their work so far, Dehlin adds, “We feel like this broad endorsement speaks very well to the quality of our sample.” In another presentation Dehlin responds to existing critique of his study by saying: “we’ve had people comment about our methodology and our sample and we just want to make sure people understand that while no sample is perfect…that certainly the sample was good enough for 7 established journals to accept the findings from our studies.” Bradshaw adds, “our studies meet the criteria for publication by professionals in the field.”

As one commentator notes, “Dehlin attempts to preempt criticisms concerning sampling issues…by stating that his group has a number of articles in reputable publications.” Nowhere, however, does Dehlin acknowledge the degree to which evident bias among social scientists themselves strongly favors his own views and conclusions. As Rosik et al. note, “Especially given the limited ideological diversity within most professional mental health organizations and academic institutions, the way different research is evaluated has been proven to dramatically differ depending on its conclusions.” Redding is then cited as saying, “how findings are interpreted and received, and the degree of critical scrutiny such studies receive frequently is dependent upon scientists’ sociopolitical views” (p. 440). More than mere opinion, Tierney summarizes how this effect has been established empirically: “In a classic study of peer review, 75 psychologists were asked to referee a paper about the mental health of left-wing student activists. Some referees saw a version of the paper showing that the student activists’ mental health was above normal; others saw different data, showing it to be below normal. Sure enough, the more liberal referees were more likely to recommend publishing the paper favorable to the left-wing activists. When the conclusion went the other way, they quickly found problems with its methodology.”

Elaborating on the scope of this problem connected to political bias, in particular, Duarte et al. (2015) adds, “The peer-review process likely offers much less protection against error when the community of peers is politically homogeneous. Ideally, reviewers should scrutinize and criticize the methods of a paper equally closely regardless of whether or not they approve of the findings. Yet, confirmation biases would lead reviewers to work extra hard to find flaws with papers whose conclusions they dislike, and to be more permissive about methodological issues when they endorse the conclusions. This is exactly what has been found in experimental studies.” They continue: “Findings that support liberal values are at risk of being waived through without sufficiently critical review. Therefore, whenever researchers review a manuscript or grant proposal that touches on ideologically charged topics, they should try a turnabout thought experiment in which one asks oneself and one’s colleagues how they would react to researchers using the same standards of evidence and proof to argue for the mirror-image ideological conclusion.”

[21] This article, for instance, was submitted by Rosik, Mansfield & Cox to the Journal of Counseling Psychology in response to the paper by Dehlin et al. entitled, “Sexual orientation change efforts among current or former LDS church members.” While it’s both common and expected for journals to publish thoughtful critiques of articles from other researchers, their paper was refused. Since then, Rosik has spoken publicly about what he sees as the larger problem—see “Scientific Integrity and the Issue of Harm: The Triumph of Advocacy over Science.”

[22] They go on to argue that: “Increased political diversity would improve social psychological science by reducing the impact of bias mechanisms such as confirmation bias, and by empowering dissenting minorities to improve the quality of the majority’s thinking.”

[23] This same pattern shows up in some of Dehlin’s podcasts as well. In so many of the sensitive, nuanced topics Dehlin has turned his attention to—topics begging for nuanced, complex, balanced, comprehensive, treatment—he has moved in another direction, imperceptibly at first and more explicitly leading up to (and since) his excommunication. Instead of the desperately-needed-care these conversations could have benefited from to ensure delicate topics weren’t unduly shaped by a single voice, Dehlin remained that single voice framing the conversations and guiding them in ways he wanted them to go. By contrast, Kadlec and Friedman (2007) underscore the crucial need for “careful control and design” of a trusted conversation setting—ensuring, for instance, that “no single entity with a stake in the substantive outcome of the deliberation should be the main designer or guarantor of the process” (p. 7).

Absent that kind of careful design, any conversation – whether featured in a study or a podcast – can have predetermined, built-in features that move its “take away” points in a certain direction. Over time, the cumulative effect of confirmation bias can ripple out across many studies and shape perceptions of consensus. As I have previously argued with Jeff Lacasse: “In the absence of attention to [their own bias] we submit that researchers may, even unaware, set-up studies that are predisposed towards [certain] results—e.g., short-term investigations using surface-level variables that minimize both participant voices and negative outcomes. Ensuing results may then be presented to potential clients and the broader public as ‘evidence that X therapy or Y medication works.’ As similar dynamics are replicated across many studies, claims of consensus may then eventually appear, with general statements on the safety and efficacy of a particular treatment appearing in evidence-based treatment guidelines, academic textbooks and even forming the basis for institutional standards of care” (p. 305). As Duarte et al. (2015) add, “In this way, certain assumptions, theories, and findings can become the entrenched wisdom in a field, not because they are correct but because they have consistently undergone less critical scrutiny.”

[24] When he began publishing these articles, Rosik and colleagues raised this caution: “Given the passionate professional and political environment surrounding [sexual orientation change efforts], we hope that Dehlin et al. will continue to be forthcoming about the limitations of their work… exercising restraint in their public pronouncements and diligently guarding against the uncritical use of their work in anti-[sexual orientation change efforts] advocacy. It will undoubtedly take considerable fortitude for them to withstand these pressures.” Rosik et al further cautioned about making “definitive statements from limited or ambiguous data,” when compared to “a longitudinal design measuring outcomes of therapist-assisted sexually oriented change efforts” that might permit more confidence. Here Rosik highlights a well-known challenge of retrospective research: since a significant period of time may have elapsed between the ending of people’s sexual change efforts and the time of Dehlin’s survey, there are many other factors that can play a role in shaping their current view.

Rather than qualify these results, however, they are often presented by Dehlin and colleagues as definitive evidence of damage or harm experienced by same-sex attracted individuals at the hands of the church (books, rituals, counseling, worship – you name it!) See footnotes #18 & #25 for illustration. So what would a more accurate of his data look like according to these critics? As Rosik et al. suggest, “At most, Dehlin et al. have provided evidence that some prior participants of [sexual orientation change efforts] who are now likely to be opposed to the goals of [these efforts] may look back upon their experience as harmful or not effective.” They go on to offer the following analogy: “Imagine a project wherein researchers surveyed former marital therapy patients who identified as being conservatively religious at the time of their therapy and who had subsequently divorced. Imagine further that the researchers used these results to determine the treatment’s effectiveness and harm and then made sweeping conclusions about this therapeutic modality. Would this be a fair and scientifically justified use of the findings?”

[25] Since publication, Dehlin has spoken with remarkably absolute statements, starting with how he consistently summarizes his work as reflecting an evident superiority to any other available evidence (see footnote #2). In a single written summary, he describes his data three different times as reflecting “the best available statistics” and “To my knowledge, there has been no better sample ever assembled…” and “To our knowledge, this study utilizes the best dataset (as in largest, and most representative) of LGBT Mormons (and possibly LGBT individuals of any religious group) ever assembled to address topics such as mixed-orientation marriages, celibacy, reparative therapy, religiosity, etc.

As illustrated elsewhere, some of the many claims made are packaged in a particularly shocking way. Footnote #8 details the example of “projected divorce statistics” and #9 details the example with “0% change” In underscoring the shocker statistic in regards to changing “core erotic attraction,” Bill Bradshaw stated, “overwhelmingly there is no evidence there of significant change through these herculean efforts.”

As a third example, instead of using other types of marriage and single statuses as an analogous figure in terms of quality of life measures (with plenty of available figures to cite), Dehlin also chose to specifically and publicly (and repeatedly) cite the comparison with Lupus for those who opted to not pursue same-sex relationships– e.g., “Perhaps more alarming, we discovered…quality of life scores lower than the scores of than people who have a debilitating illness called lupus, which symptoms include difficulty breathing, chest pains, bleeding, infections, skin rashes, nose sores, hair loss and seizures.” While explaining that as a mere reflection of the medical patients that measure was developed to track, that decision has clearly been used rhetorically to invoke additional shock value in his audience. Indeed, for a comparison figure that Dehlin once described as “unfortunate,” he sure comes back to comparisons with lupus and “debilitating disease” over and over. I’ve virtually never heard Dehlin present about his results without drawing on that comparison or allusion – e.g., “measures of quality of life for individuals in these groups is lower on average than for persons suffering from some debilitating diseases.”

[26] For instance, Dehlin’s co-author Bill Bradshaw once stated “Empirically, the data show that the rate of divorce in these marriages is very high.” Footnote #25 includes other samplings of the kinds of generalization Dehlin and his collaborators have made. As Laurie Campbell summarizes, “Despite the fact that they point out their study is not random, they still present percentages as fact and as if they can be applied to the SSA/LGB LDS population as a whole.” Citing another example, Laurie adds, “Although John admits it is not possible to determine causality regarding bisexuality and successful Mixed Orientation Marriages (MOM), he report it as if such is the case” – quoting one of Dehlin’s early write-ups as stating, “Bisexuality is Essential in Preserving a Mixed-Orientation Marriage.”

Although Dehlin and Bradshaw both acknowledge the limited generalizability of findings, they also often use the “limitation” section of published papers and presentations to make especially complimentary statements about their approach and sample – even reflecting the “best data out there” (which shows up in virtually every presentation as an oft-repeated refrain). For instance, in one paper’s section describing “limitations” of the study, they speak glowingly about their methodology and the data it generated as reflecting “powerful and consistent findings with regard to sexual orientation change efforts outcomes.” They also insist that: “The demographic characteristics of the participants suggest that our survey successfully reached the broad target population” (Bradshaw, Dehlin et al., 2014, pp. 18-19).

[27] These are statements that can never accurately—or ethically—be made from a non-random sample (which is not generalizable to the broader population even of Latter-day Saints), regardless of how hard he insists trying to get a broad sample of experience. In that case, data should only be reported as representing Dehlin’s sample only—not all mixed-orientation marriages or all members of the Church. As Rosik and colleagues note following their own review, “we believe the findings of Dehlin et al.’s study cannot be definitively or legitimately generalized beyond the sample population examined. It is a sample purported to be more representative but which in fact is overwhelmingly represented by currently LGB identified persons who are disaffected from their LDS Church.”

In fairness, Dehlin et al. do acknowledge this point more openly on occasion: “Our reliance on the convenience sample limits our ability to generalize our findings to the entire population of same-sex-attracted current and former LDS church members” (p. 10). Despite this kind of a statement, however, so many statements that Dehlin and Bradshaw both make (and the public following suit) do just that! There seems to be a kind of discrepancy between some statements (like this above) and others (like the generalized statements quoted throughout). As a result, a serious problem remains of using non-representative data in a representative way and generalizing data that cannot be generalized. Not only do ethical research standards require that Dehlin be extra careful in his own reporting of the data, but these standards would enjoin him to challenge public misinterpretation and misreporting whenever possible.

Although Dehlin qualifies on occasion, in the very same moment he typically makes the same kinds of generalized statement. More than a matter of “imperfect technique,” this pertains to Dehlin’s professional ethics in reporting research results. As a result of limitations in that regard, readers of the study speak in similarly black and white language. For example, Daniel Parkinson translates Dehlin’s results as follows, “This study doesn’t say that LGBT people can’t be happy in the church. It just says that LGBT people who are in the church are statistically less happy than LGBT people who have left the church.”

To repeat using the divorce statistics, one can never accurately or ethically say that “51% of mixed-orientation marriages end in divorce” (to say nothing of the projected “at least” 69%). What can only be said is that “51% of survey respondents who had been married to someone of the opposite sex were divorced.

[28] This has become a regular refrain in many of their presentations. For instance:

“Those who identify as bisexual (or are near the heterosexual end of the scale)…find a greater range of options for accommodation, including heterosexual marriage” (Dehlin, et al., 2015, p. 20)

“Participants who remained in [mixed orientation marriages] reported significantly lower Kinsey attraction scores than those who reported being divorced, possibly suggesting that bisexuality is a significant factor in keeping a [mixed-orientation marriage] together” (Dehlin, et al., 2014, p. 299).

“For those individuals claiming some positive effects from counseling to help with unwanted same-sex attraction, they hinted these outcomes largely reflected a bisexual nature: “The average Kinsey Scale score for attraction for these 22 participants was at the midpoint of the scale” (Bradshaw, Dehlin et al., 2014, p. 10).

“Our data showed that an accommodation is most probable for those who identify as bisexual” (Bradshaw, Dehlin et al., 2014, p. 18)

Bradshaw states in one presentation “Since homosexuality is determined by biology those who are programmed so as to have capacity to have erotic feelings for both men and women – those people have options that are not available to those on the high end.”

Dehlin states in the same presentation: “Those who were able to remain in a mixed orientation marriage were by far self-reported as being bisexual. So bisexual attractions was a very important marriage in having your marriage succeed.”

The overall message is that those managing to stay in church are really just bisexual, those pursuing heterosexual marriages are really just bisexual and those reporting internal change are really just bisexual. Rather than acknowledge people’s own self-understanding, Dehlin et al. insisted on strongly refuting the language in which people understood themselves. This dismissive attitude is observed in a few other occasions. As one commentator noted, Dehlin puts “lifestyles” in quotes when talking about mixed-orientation marriages: “It is generally a good idea not to use typography that appears to question that legitimacy of your study subjects’ life decisions when engaged in purportedly serious research.”

This was evident more substantially in the published article Bradshaw, Dehlin et al., 2014 where the authors attempted to account for each person who chose to follow an orthodox pathway and didn’t report simply affirming their orientation. Rather than exploring the different kinds of changes people reported experiencing, they attempted to dissect the characteristics of these anomalies – sharing short, pointed excerpts of each person’s survey – e.g., “same sex attraction