HONG KONG -- On the 20th anniversary of Britain's handover of Hong Kong to China, thousands of people filled parts of downtown Hong Kong Saturday demanding that the Chinese government respect the former British colony's special status and the rights that come with it.

This city is the only place in China where you can openly protest against the Chinese government, reports CBS News correspondent Ben Tracy. That's because when Hong Kong was given back to China, the agreement stated it would retain most of the freedoms it had under British rule. But now many here feel those freedoms are slipping away.

Much of the anger is directed at Chinese President Xi Jinping who swore in Hong Kong's new chief executive today. Carrie Lam is widely seen as someone who will defer to Beijing.

Hong Kong Chief Executive Carrie Lam and Chinese President Xi Jinping Bobby Yip

It's Xi's first trip to Hong Kong since taking office five years ago, and the Chinese government is clearly trying to show its strength while celebrating the handover anniversary.

"We don't celebrate. We mourn," lawyer and author Jason Ng said.

Ng said Hong Kongers worry China is undermining the agreement it made when Britain returned the city in 1997. Known as "one country, two systems," Hong Kong is part of China but retains a high level of autonomy and rights such as freedom of speech and an independent judiciary. The hope at the time was that this city of seven million people would help make the rest of China more open and free.

"Do you think China is becoming more like Hong Kong? Or is Hong Kong becoming more like China?" Tracy asked.

"The general sense that China would slowly democratize and Hong Kong would benefit from that seems to be a pipe dream… And in the past five or six years, we have seen really an acceleration of China's political agenda on Hong Kong," Ng said.

Beijing is tightening its grip. It recently blocked pro-independence lawmakers from taking office and abducted five publishers who sold books critical of the Chinese government. Lam Wing-kee was detained for eight months and said he was forced to sign a confession. He said if the Chinese government does not follow Hong Kong's constitution, the city will have no choice but to push for independence.

Most people here don't believe China will ever allow Hong Kong to be independent, but in the past decade, the number of protests has soared more than 200 percent as citizens fear their way of life is under attack.

Activist Joshua Wong, 20, was arrested this week at a protest. He led the opposition to so-called patriotic education in Hong Kong schools and was part of the massive protests that shut down large parts of the city in 2014. Wong said despite the 20th anniversary, "it's not a time for celebration."

"It's time for demonstration," Wong said. "They hope to return to 'one country, one system.' We hope to maintain democracy and ask for autonomy."

But in his speech before he left Hong Kong, Xi issued a stern warning, saying any challenge to China's sovereignty or central government's authority crosses a red line and won't be tolerated.