(CNN) -- The Beverly Hills Hotel has been a haven for Hollywood's elite since 1912 -- two years before the City of Beverly Hills was even created.

But now some stars are rushing for the exits.

On Monday, a number of celebrities took part in a demonstration across from the famed Sunset Boulevard lodge, protesting not the hotel but its ownership -- an investment group controlled by the Sultan of Brunei.

In late April, the small Muslim nation on the northern coast of the island of Borneo implemented Sharia law, marked by strong punishments for those found guilty of homosexuality or adultery -- including stoning the convicted to death. The law was praised by the sultan, who isn't just the country's ruler but also a major political and cultural figure.

"We're just making people aware," Jay Leno, who was among the demonstrators, told CNN.

"It's not a political issue. This is not something that's debatable. ... It's people being stoned to death," he said.

A number of organizations have moved -- or threatened to move -- events scheduled for the hotel.

The Los Angeles Times reported that the Feminist Majority Foundation moved its Global Women's Rights Awards from the Beverly Hills Hotel to the Hammer Museum in nearby Westwood. The awards, which were held Monday night, are co-chaired by Leno and his wife, Mavis.

The International Women's Media Foundation similarly pulled its Courage in Journalism Awards ceremony from the venue.

"We, the IWMF, can no longer hold an event at a venue whose owner stands for the violation of the very human rights we fight for," Elisa Lees Munoz, executive director of the IWMF, said in a statement.

Also saying they're staying away, according to the Los Angeles Times: The Hollywood Reporter's annual Women in Entertainment breakfast, the Motion Picture & Television Fund's Night Before the Oscars and a fundraising luncheon hosted by Teen Line, a peer helpline.

Moreover, such notables as Ellen DeGeneres and mogul Richard Branson have announced they won't patronize the hotel or other properties in the Dorchester Collection of luxury hotels, which owns and manages the Beverly Hills Hotel and the Hotel Bel-Air.

"I won't be visiting the Hotel Bel-Air or the Beverly Hills Hotel until this is resolved," DeGeneres tweeted to her 28.8 million followers.

Leno said he was well aware of the concerns.

"I know a number of people are canceling events at the hotel, and it's all economics," he told CNN.

But Christopher Cowdray, the Dorchester Collection's CEO, said the celebrities and groups protesting are taking out their anger on the wrong place.

"The actions they're taking are unfounded," he said.

"It's going to hurt our employees and this has nothing to do with them whatsoever," he said. "It's not their fight."

The sultan has owned the Beverly Hills Hotel since 1987. The Dorchester Collection was established in 1996 to oversee that hotel and others in its luxury portfolio, including the Hotel Bel-Air, Paris' Plaza Athenee and London's Dorchester.

Cowdray said that celebrities wanting to influence politics in Brunei would be better off lobbying the U.S. State Department to take action.

At a briefing Monday, State Department spokeswoman Marie Harf said, "We have very serious concerns."

The City of Beverly Hills is also making its opinion known. Mayor Lili Bosse is introducing a resolution before City Council that would urge the sultan to divest his interest in the hotel that bears the city's name.

"We have a history of standing strong against injustice," she said.

Like Leno, she emphasized that the issue wasn't the hotel, but what was going on in Brunei.

"The resolution is not a call for a boycott of the Beverly Hills Hotel," she said. "This is really a focus on the government and their laws and not a statement about the hotel."

CNN's Todd Leopold contributed to this story.