The National Center for Transgender Equality (NCTE) has launched a campaign to gather information about trans identified people called the U.S. Trans Survey. The intent may be noble, but the data collection methods leave much to be desired. Here is some background on the project from its organizers (emphasis added):

[The U.S. Trans Survey] is the follow up to the groundbreaking National Transgender Discrimination Survey, which was conducted by the National Center for Transgender Equality and the National LGBTQ Task Force in 2009, and examined the lives of over 6,400 trans people in the U.S. The results were released in a 2011 report called Injustice At Every Turn. As the single-most cited study about trans people, it has changed how the public understands the challenges facing our community.

That last sentence is the crux of what bothers me. An internet based questionnaire is the single-most cited “study” about trans people? Are you serious? In what world is an anonymous survey with no sample validity considered the primary or best evidence by which “the public understands the challenges” faced by any group of people? Look, I’m no statistician but I did take Stats 101 in college. Even I can tell this “study” has holes the size of the Grand Canyon.

Before we review some of the most glaring flaws, I want to make perfectly clear that trans people are not the sole stakeholders of this survey’s results. The entire U.S. population is potentially impacted by legal and policy changes that cater to special interest groups. Revising bathroom and locker room access–from urogenital configuration (physical sex) to personal preference (gender identity)–is probably the best example of how policies designed to protect one group of people can inadvertently expose others to situations they find objectionable or even dangerous. As such, every single person in the U.S. has a stake in the conversation surrounding this project. It is every person’s business.

PLEASE VALIDATE YOUR IDENTITY, IF YOU CAN

The concept of identity is subjective by definition. I-dentity, as I like to spell it, is a self reported statement about the subject by the subject. It has no measurable criteria; it has no external reference by which we can judge whether the person claiming the I-dentity is lucid or delusional or somewhere in between. “Because I said so” is the entire standard of proof. This is not enough to accurately demarcate the boundaries of a discrete class of persons about whom conclusions can be accurately drawn and understood by others. I-dentity alone is a weak, even illusory, basis by which conduct a study that is intended to influence public policy.

More specifically, when we speak of gender I-dentity, internal feelings must be prioritized over material reality. The idea that your subjective insides don’t match your objective outsides is precisely what makes someone “trans” or “not trans.” See cisgender theory for more. Personally, I have no clue what inside/outside matching means because it reeks of essentialist and/or spiritual nonsense, but my point is that self-proclaimed gender I-dentity is wholly characterized by internal feelings. It’s all in your head, by definition.

Further, the meaning of what’s in your head is frequently contested. Being “trans” may include just about anyone or no one, depending on who you talk to. The DSM 5 requires dysphoria. But this recent article tells me that not all “trans” people experience body dysphoria. So, what’s The Truth? The problem is that there is no Truth About Trans People’s Internal Feelings. And that compounds the problems discussed above about the incoherent nature of I-dentity as a criteria for study.

Prevalence rates for transgenderism are hard to pin down, because the definitions are obscure, and the decision to identify as transgender is a subjective one. [source]

Indeed, the survey is not just for “trans” people. If you identify as “genderqueer” or “non-binary,” you are also encouraged to complete the survey. One of the emails I received from Sandy James, Survey Project Manager for the NCTE, explained that “getting the most diverse sample yet” was their goal. Hmmm, ok. Well, I might be genderqueer. Look at me, sitting here in my pants with no make-up and short hair! That’s, like, everyday baby. My sister asked me this past weekend if I had stopped shaving my legs. And shoot, I’m a lesbian too! Ok, I’m definitely genderqueer. How about you? Are you feeling “non-binary” in your mind? If so, this survey might be for you, wink wink.

VOTE EARLY AND VOTE OFTEN!

The U.S. Trans Survey has no automated participant controls or sample validity whatsoever. As you can see in this screenshot, it actually asks if you’ve taken it before. Um, no? Well, maybe. Who knows! Not you! Once again, we’re working entirely on the honor system: because I said so, dammit!

I’m really good at using The Google, so I did a little research into online polls. I quickly learned that ballot box stuffing is a common problem that serious researchers attempt to avoid.

Traditionally, ballot box stuffing applies to elections. It means to illegally fill a ballot box with more votes than the number of real voters. It also applies to surveys when a participant takes the survey more than once, or even multiple times. This immediately diminishes the quality and integrity of your survey results. That’s why it’s so important to limit survey responses to one unique response per participant. [source]

I also found that there are tools you can employ to prevent repeater trolls from messing up your game! More simplistic polls use cookies to prevent people from taking the survey multiple times. Some track IP address. There are more sophisticated methods on the market as well. None of these have been used by the NCTE’s survey.

GLOBAL ACCESS

The U.S. Trans Survey is intended to evaluate trans people’s experiences in the United States. Yet my (highly reliable) sources tell me that people outside the U.S. can not only access the survey, but depending on your “because I said so” answers, you can continue with the rest of the questions. Hey, you know how sometimes you can’t watch a video from the BBC in America? Well, the survey makers could have employed some country restrictions. But they didn’t.

NCTE isn’t using any of this technology to protect the integrity of their data. With over a million dollars pledged to the organization in 2013 (2014 report not available), I see no excuse for this failure except a blatant disregard for the accuracy of the data being collected.

BIAS IN, BIAS OUT

As the intended successor to the National Transgender Discrimination Survey, also known as Injustice At Every Turn, the survey is designed and targeted to prove the conclusion it has already come to: that “trans” people are in great danger and therefore in urgent need of political attention, sympathy, and access to public resources.

The name of the predecessor project alone is highly suggestive. Discrimination! Injustice! Its baldly stated purpose is to reach people who believe that they are or have been persecuted on the basis of being trans. This is not a random sample of the general population, or even of the trans population. The survey is specifically geared towards those who feel aggrieved or victimized because they self-I-dentify as as trans. These people’s experiences will necessarily be over-represented in the survey. This bias could be significantly mitigated by simply not telling people what the survey is about in the first place (and including more questions about everyday life).

The survey has also been marketed to people with a vested interest in the outcome of the survey. This is another form of bias inherent to the project: the targeted respondents are uniquely positioned to benefit from public policy changes and increased resource allocation that may flow from the survey’s findings. There is potential motivation, therefore, to answer questions in a particular way. Namely, the most persecutory way possible that maximizes the respondents’ perceived victimization.

The NCTE has a corresponding political agenda to prove its relevance, sustain current operations, and to solicit additional funding for expansion. From another email sent by Sandy James (emphasis in original):

Almost everything we know about trans people came from the National Transgender Discrimination Survey (NTDS). That ground-breaking study has provided the data the LGBT movement needed to help paint a clear picture of trans lives and show the world how deep our struggles really are. Because of that study, we’ve been able to transform our advocacy and fight for life-saving laws and policies.

The motives and goals of this project are crystal clear: to prove that trans people are the ultimate social victims. Both the creators and the participants have an interest in generating the most sensational results possible. And I have no doubt they will succeed at doing so.

GAUGING SUICIDALITY

On a positive note, I was pleased to see an improvement in the quality of the questions. There are more questions and more detailed questions this time. The section about suicidality is of particular interest given the irresponsible and hyperbolic reporting (<must read link) about “data” from Injustice at Every Turn. In that survey, there was only one question about suicide. In an independent analysis, the Williams Institute explains:

[T]he NTDS questionnaire included only a single item about suicidal behavior that asked, “Have you ever attempted suicide?” with dichotomized responses of Yes/No. Researchers have found that using this question alone in surveys can inflate the percentage of affirmative responses, since some respondents may use it to communicate self-harm behavior that is not a “suicide attempt,” such as seriously considering suicide, planning for suicide, or engaging in self-harm behavior without the intent to die (Bongiovi-Garcia et al., 2009). The National Comorbity Survey, a nationally representative survey, found that probing for intent to die through in-person interviews reduced the prevalence of lifetime suicide attempts from 4.6 percent to 2.7 percent of the adult sample (Kessler et al., 1999; Nock & Kessler, 2006).

By contrast, the U.S. Trans Survey features approximately 7 questions about suicidal thoughts and attempts, including whether professional after-care was sought (questions 16.1 through 16.10). The possible responses are still strictly limited to pre-populated choices, but it’s some kind of progress. There is also a section on emotional health (questions 12.2 and 12.3), which I hope will be analyzed in conjunction with information about self-harm.

Significantly, however, the survey does not ask if you been diagnosed with or self-identify as having a mental disorder. It is well known that people who who suffer from concurrent mental disorders, depression for example, are at much higher risk for fantasizing about or engaging in self-harm regardless of gender I-dentity. This is a very basic oversight that compromises the accuracy of the data and the inferences that can be legitimately drawn therefrom.

If you would like to see a(n almost) full copy of the questions, please download this pdf. USTransSurvey_questionnaire_2015

LEADING AND CONFUSING QUESTIONS

It is well established that a study’s target concepts should be clearly defined. As discussed above, even the words “trans” and “non-binary” are not defined by the U.S. Trans Survey. Additionally, questions about psychologists and counselors (section 13) are separated out from questions about health care providers (section 12). First, psychologists are medical practitioners. Full stop. Second, this doesn’t make sense to those of us who receive medical treatment, paid for by health insurance companies, from certified mental health therapists or counselors. In fact, trans people who want to medically transition require a mental disorder diagnosis (see “gender dysphoria” in the DSM-5) in order to access hormonal treatments from endocrinologists. So, should we leave answers about our experiences with psychologists and mental health counselors in section 12 or section 13? We don’t know! This confusion negatively impacts the accuracy of the data. It could be resolved by defining these concepts such that respondents will answer consistently.

Survey questions should not lead the respondent to answer in a particular way. Sandy James has made public statements about the complex design of the study:

James also pointed out that the 2015 survey “is the most dynamic and individualized of its kind. For example, if you report that you didn’t serve in the military, you won’t get questions pertaining to military service.”

This may be true about the military questions, but it is not true overall. The more politically controversial sections are extremely leading. The name-change section (questions numbered 10.1 through 10.18) asks the same series of questions multiple different ways, apparently expecting or seeking a pre-conceived response. This feels very much like leading the respondent to express a desire to change her name, even if she doesn’t want to. Same problem in the section about sex work:

Repeated questions can be found in many sections of the survey, including those about mistreatment by health care providers and what forms of medical intervention you are seeking for your trans identity (even if you made clear in the beginning of the survey that you identify as non-binary, not trans). The leading format of these questions is another way in which the survey uses flawed methodology to manipulates the responses it will receive.

GOOD DATA

Good public health data is based on verifiable health outcomes and can be repeated with newly generated samples. For example, the CDC uses in-person interviews and medical record reviews. Good discrimination data is based on actual complaints filed, as the EEOC has done here.

I fully understand that political groups have political agendas. I appreciate that the NCTE is sincerely interested in representing trans I-dentified people and advancing “social justice” on their behalf. That’s all well and good, but in light of the problems discussed above, the U.S. Trans Survey will artificially inflate the challenges– and therefore misrepresent the experiences of– people who identify as trans and their gender non-conforming brethren. I don’t know if this is a result of methodological negligence/ignorance or if it’s an intentional attempt at self-serving data collection. Either way, there are major weaknesses in the survey. I think the NCTE can do better by all of us– trans people and the general public alike.

We need long term studies about trans people. We need accurate and reliable data about trans people. We need to define what we mean by “trans” people in the first place. We need to validate that sample subjects are actually trans-itioning and in what ways. We need to prevent ballot box stuffing, international tampering, leading questions, confusing questions, and un/conscious bias. These problems are not strictly inevitable or unavoidable. There are well-known methodological controls that go a long way towards mitigating each of them.

The NCTE can “clean” the U.S. Trans Survey data all they want. Unfortunately, there is no way to correct a survey’s design flaws after the survey has been released. Garbage in, garbage out.