JAKARTA (Reuters) - About 50 members of a hardline Muslim group barged into an apartment in the Indonesian capital to break up what they said was a gay sex party, angering rights groups, and police said on Monday no one in the flat had done anything wrong.

The lesbian, gay, bisexual and transgender community is largely accepted in urban areas of Indonesia, the world's largest Muslim-majority country, and being gay is not illegal, but there is growing concern among rights activists about intolerance.

South Jakarta police chief Iwan Setiawan said the 13 men in the apartment were questioned and released on Monday because there was no evidence they had broken any law at the weekend gathering.

Our Voice, a group supporting gay rights, said police should have detained the members of the Islamic Defenders Front (FPI) for breach of privacy.

"For a government agency to allow such thing to happen, it is terrifying. It is a warning for this nation," said Hartoyo, a director of Our Voice.

The FPI, which is also known for raiding bars and nightclubs, as well as helping prevent pop star Lady Gaga perform in the country, has vowed to continue such actions.

"The FPI since its birth has been and in the future will continue to fight to uphold what is good and to ban what is bad for society at all costs," Novel Bamukmin, the group's Jakarta secretary general, said.

The FPI backed a large rally on Nov. 4 against Jakarta's Christian governor, who is accused of insulting the Koran.

One person was killed and more than 100 were wounded when the protest, the biggest in the city in recent years, briefly turned violent.

(Reporting by Gayatri Suroyo and Agustinus Beo Da Costa; Editing by Nick Macfie)