HONG KONG — Police arrested more than 500 people who refused to leave a street in the city’s financial district Wednesday, a day after tens of thousands of people joined a massive march to demand democracy that’s free from China’s interference.

The march has become an annual affair held on the anniversary of the day China took over Hong Kong from Britain on June 1, 1997, with the promise to give the city a high degree of autonomy for 50 years. But there is growing unease among its residents — especially the youth — that the Western-style civil liberties they’ve know all their lives are being eroded as Beijing has increasingly tried to impose its authority over the freewheeling capitalist enclave.

The fears are only going to be heightened following the pre-dawn crackdown by the Hong Kong police, who normally do not have an antagonistic relationship with the people, unlike the security forces in mainland China.

Police said 511 people were arrested for unlawful assembly in the central business district and preventing police from carrying out their duties. After warnings failed to dislodge them, the police moved in to remove the protesters, who lay down on the street with arms locked, taking them away one by one. The protesters had vowed to stay until 8 a.m., just before the height of rush hour begins, but the police started moving in to evict them at about 3 a.m.

Some left willingly but others were forcibly removed, taken away by officers on both arms or carried away off the ground. One officer stood behind a protester and put an arm around the man’s neck as he and other officers tried to pull his hands free, knocking off the eyeglasses of the demonstrator he was clinging to.

Those arrested were mostly students who had decided to occupy Chater Road after taking part in Tuesday’s rally, which police said attracted 98,000 people at its peak. Organizers said 510,000 people turned out, the highest estimates in a decade. Hong Kong researchers put the number at between 154,000 and 172,000.

Whatever the numbers, the march and the vehemence of opposition is certain to raise alarm in Beijing, which tried to keep the news away from people in mainland China.

The Chinese media predictably did not report it, and comments about the protests were deleted from microblogs and other social media. Some users posted comments saying friends’ accounts were blocked after they discussed the protests.

A Hong Kong protester, Kennie Chan, lamented that Beijing was less restrained now in exerting its influence over Hong Kong.

“In the past, it seemed like they were doing it step by step, but now, it’s obvious that they cannot stand Hong Kong people. We are not obedient anymore, and are resisting more and more,” said the 30-year-old, who works as a stage manager.

China’s Communist leaders have pledged to allow Hong Kong residents to vote for the leader by 2017. However, they’ve rejected calls to allow the public to name candidates, insisting instead that they be vetted by a Beijing-friendly committee like the one that has hand-picked all leaders since the handover.

Also, three weeks ago, the Chinese government released a so-called white paper that said Hong Kong’s high degree of autonomy is not inherent but is authorized by the central government in Beijing.

Anger against those moves was on display during the march Tuesday when peaceful crowds carried banners and posters urging democracy. The steady throng of people walked in sweltering heat and occasional rain from Victoria Park, through a broad boulevard lined with skyscrapers to the financial district. Thousands of police kept watch and ordered the city’s iconic trolleys shut down along the boulevard to reduce overcrowding.

“After seeing the white paper’s content, we should be worried,” said Jeff Kwok, 28, an export firm employee at Victoria Park, where six soccer fields and surrounding areas were jammed with people. Beijing is “trying to tell the Hong Kong people that … Hong Kong is just one of their regions. They’re trying to tell us they have absolute power to rule us.”

Kwok complained that Beijing doesn’t respect the principle of “one country, two systems” under which Hong Kong is allowed to retain control over much of its own affairs.

Ahead of the rally, one group of protesters burned a copy of the white paper outside a ceremony attended by officials to mark the handover.

The protest comes days after nearly 800,000 residents voted in an informal referendum aimed at bolstering support for full democracy. Beijing denounced the referendum as a political farce.

Leung Chun-ying, Hong Kong’s Beijing-backed leader, tried to soothe tensions, saying in a speech that he’ll do his “utmost to forge a consensus” on implementing universal suffrage on schedule. But the government later released a statement saying it is unlikely that public nominations will be allowed because it’s legally “highly controversial.”