In the windows of some small cafes and churches around Central Brooklyn, there are little white stickers with rainbow-colored writing. These signs put up by the Audre Lorde Project say “Safe Space,” designating those buildings as places of sanctuary for LGBT people who are experiencing harassment or violence on the street.

Despite New York City’s inclusive policies for LGBT residents, the borough of Brooklyn still saw four “hate violence” incidents against them in the space of just two weeks this summer, according to the National Coalition of Anti-Violence Programs (NCAVP). People were threatened with death, they report, punched in the face, slurred at, socked with rocks.

For one particular community, these instances of violence happening around the country have higher chance of becoming fatal. On Aug. 14 the number of transgender people murdered in America this year hit a historic high of 15, according to advocacy organizations like the National Center for Transgender Equality. This somber milestone was hit when the remains of Elisha Walker, 20, were discovered in a North Carolina field several months after she went missing. Like the majority of the other victims, Elisha was not just transgender but a young transgender woman of color.

“These are all characteristics of people in the United States who are more susceptible to violence,” says the Center’s Mara Keisling, “of people who are more marginalized economically and educationally, people who end up having a bullseye on their back.”

The legal victories and increased media coverage of LGBT people in recent months has been largely positive for the community, experts like Keisling say. More people feel comfortable coming out, giving others the chance to meet and befriend someone who is transgender or gay, building the personal relationships that activists say are often the foundation for acceptance.

But the heightened visibility has also put more people at risk of being harassed or hurt. While images of Caitlyn Jenner receiving a standing ovation accepting an award in a Versace dress might seem to herald a sunny time for transgender Americans, most of them are still greatly disadvantaged socially and economically.

“Right now we’re experiencing a Dickensian time, where it’s the best of times and it’s the worst of times at once,” says transgender rights advocate Masen Davis, who formerly ran the Transgender Law Center. “We’re seeing a marked increase in the public awareness about transgender people and really incredible progress for trans rights, especially from a legal perspective. At the same time, we still represent and are part of a community that experiences incredibly high rates of unemployment, poverty and violence.”

Transgender people are four times more likely than the general population to report living in extreme poverty, making less than $10,000 per year, a standing that sometimes pushes them to enter the dangerous trade of sex work. Nearly 80% of transgender people report experiencing harassment at school when they were young. As adults, some report being physically assaulted trains and buses, in retail stores and restaurants. Greater awareness has not yet translated into broad acceptance, says Kris Hayashi, executive director of the Transgender Law Center: “The majority of society does not understand who transgender people are in ways that lead to the violence and the murder and the harassment that we’re seeing.”

The risk is even greater for transgender women of color, who often grapple with both transphobia and racism. Sixteen of the at least 20 LGBT people murdered in 2014 were people of color, according to the NCAVP; 11 were transgender women, and 10 were transgender women of color. “People who are marginalized both because of their race and being transgender, it’s like a double whammy,” says Shannon Minter, legal director for the National Center for Lesbian Rights.

These figures likely don’t give a full picture of violence against the transgender community. Though a federal hate crimes law requires the collection of some statistics related to violence against transgender people, experts are dubious about the numbers they’re getting. “A lot of jurisdictions report zeroes, even in places where we know there are hate crimes,” Keisling says. Most state laws don’t require the collection of such statistics, according to Minter.

He says numbers are often misreported too. Incidents may not be determined to be hate crimes because there was no investigation, for instance. Crimes against transgender men like Brandon Teena—who was raped and murdered in Nebraska before his story was told in the film Boys Don’t Cry—may be recorded as crimes against women because many don’t have the money (or desire) for medical intervention.

The NCAVP, which collects the most complete figures on hate crimes against LGBT people, notes that the Bureau of Justice Statistics estimated there may be 40 times more hate crimes occurring nationally than the FBI reports. Overall, the NCAVP has been receiving far fewer reports of hate violence toward the LGBT community in the last two years, down to 1,359 incidents in 2014. But they say this is a result of the collection process, not “an actual decrease of bias-based violence,” and they say their statistics do not represent exhaustive national numbers.

Minter says that the murder rate, as well as the chronic harassment many transgender people face, is best tackled through better education and more community-based programs, like those white stickers in Brooklyn windows that create networks of support among people who walk the same streets each day. Hate crime legislation is helpful in sending a message about the value of lives, he says, but it’s not going to solve the problem.

“We all have a responsibility to stop this violence,” he says, “and that means if you see a transgender person being harassed, we all have an obligation to speak up, to do something.”

Get The Brief. Sign up to receive the top stories you need to know right now. Please enter a valid email address. Sign Up Now Check the box if you do not wish to receive promotional offers via email from TIME. You can unsubscribe at any time. By signing up you are agreeing to our Terms of Use and Privacy Policy . This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply. Thank you! For your security, we've sent a confirmation email to the address you entered. Click the link to confirm your subscription and begin receiving our newsletters. If you don't get the confirmation within 10 minutes, please check your spam folder.

Write to Katy Steinmetz at katy.steinmetz@time.com.