Hong Kong (CNN) China's President Xi Jinping has warned that any efforts in Hong Kong "to challenge the power" of Beijing are "absolutely impermissible," as the former British colony marked 20 years of Chinese rule Saturday.

"Any attempt to endanger China's sovereignty and security ... or use Hong Kong to carry out infiltration and sabotage activities against the mainland is an act that crosses a red line," Xi said, in a nationalistic speech which emphasized Beijing's control over the city, now a special administrative region of China.

It was Xi's first visit to the city since he took power in Beijing in 2013, and came amid fears by some Hong Kongers of increasing Chinese encroachment on the city's autonomy -- guaranteed under a framework known as "one country, two systems."

A protester raises a poster with Chinese President Xi Jinping's portrait as thousands march during the annual pro-democracy protest in Hong Kong, Saturday, July 1, 2017.

Xi left Hong Kong in the early afternoon Saturday before the kick-off of an annual July 1 pro-democracy march, which saw streets packed with protesters despite stifling heat and heavy rain.

Organizers said 60,000 people turned out, far less than the 110,000 they said took part last year . Police -- who use a different method to count participants -- said 14,500 joined the march, compared to 19,300 last year.

Ron Wong, 17, who was marching with his parents, said Xi's visit had been a "show of power of who's in charge."

"China has barricaded itself off (from criticism)," he said.

The plight of Liu Xiaobo , the Nobel Peace Prize-winning Chinese dissident recently granted parole for cancer treatment, was also a rallying cry. Many protesters carried banners calling for his release, or pictures of empty chairs -- echoing the award of his 2010 Nobel prize that took place while he was in prison.

Giant banner of Liu Xiaobo near the front of the march, which is leaving Victoria Park now: pic.twitter.com/EHUOjXd0fs — James Griffiths (@jgriffiths) July 1, 2017

Sheltering from the rain near the end point of the march -- Hong Kong's Legislative Council building -- Au Yeung-dong said it was a "successful protest."

"I think we managed to get Xi's attention," said the 39-year-old, who was wearing a chair with Liu Xiaobo's name on it.

Protesters carry a large image of jailed Chinese Nobel Peace laureate Liu Xiaobo as they march during the annual pro-democracy protest in Hong Kong, Saturday, July 1, 2017.

Lam sworn in

Earlier, Xi swore in Carrie Lam as Hong Kong's first female Chief Executive. Lam called for unity to heal the city's divided society -- just hours after police scuffled with pro-democracy protesters attempting to reach the harbor-side venue where the ceremony took place.

Lam said she recognized the political and economic challenges facing Hong Kong but added that the city's problems cannot be solved overnight.

Chinese President Xi Jinping, right, shakes hands with Hong Kong's new Chief Executive Carrie Lam after Xi administered her oath-taking ceremony.

Protesters had hoped to march on an early morning flag-raising ceremony, but were prevented from moving more than a few hundred meters by flag-waving counter-demonstrators and a huge police cordon.

Leaders including high-profile campaigner Joshua Wong were detained and bundled into police vans as their supporters shouted "shame." He was later released.

Lawmaker "Longhair" Leung Kwok-hung, a veteran pro-democracy activist, was also briefly taken into custody. "Patriotism is the last resort of thugs," he said after his release.

Hong Kong lawmaker Leung Kwok-hung (right) at a pro-democracy rally Saturday. Protesters were prevented from leaving the venue by pro-China counter demonstrators.

The confrontation, and police response, exposes the tensions in the city between those who seek greater freedom and those loyal to leadership on the mainland.

At a gala event Friday night, Xi urged Hong Kongers to unite and build on the progress made by previous generations.

Photos: With Hong Kong's leaders, China's President Xi Jinping sings a song entitled "My Country" at a gala event in Hong Kong to mark 20 years since the handover of the city from British to Chinese rule. Hide Caption 1 of 6 Photos: The event came on the second day of a three-day trip, Xi's first since he became Chinese President in 2013. Hide Caption 2 of 6 Photos: Dancers put on a show for the Chinese President. Hide Caption 3 of 6 Photos: Singer Leon Lai performs at the event on June 30, 2017, surrounded by dancers. Hide Caption 4 of 6 Photos: Singer Karen Mok was among a star lineup on stage, while the audience was filled with Chinese and Hong Kong guests and officials. Hide Caption 5 of 6 Photos: China's President Xi Jinping makes a toast during a banquet, urging Hong Kongers to unite and build on the progress of their predecessors. Hide Caption 6 of 6

"The road ahead is by no means smooth, but our faith in implementing 'one country, two systems' will not change, our determination won't be swayed," he said, referring to system that affords Hong Kong a degree of separation from the mainland.

Show of force

Xi's visit, which has seen unprecedented levels of security with roadblocks, huge barriers and a massive police presence, was largely peaceful.

The Chinese President oversaw a military parade Friday, the largest since the city's handover from the UK to China in 1997 , with around 3,000 members of the People's Liberation Army taking part.

Photos: China's Xi parades at Hong Kong PLA garrison Chinese President Xi Jinping inspects the Hong Kong garrison of the People's Liberation Army (PLA) at their Shek Kong base in Hong Kong, on Friday, June 30, 2017. Hide Caption 1 of 9 Photos: China's Xi parades at Hong Kong PLA garrison The parade was the largest since Hong Kong's handover from British to Chinese rule in 1997. Hide Caption 2 of 9 Photos: China's Xi parades at Hong Kong PLA garrison A total of 20 squadrons paraded in front of Xi, including land, sea, and air forces. Hide Caption 3 of 9 Photos: China's Xi parades at Hong Kong PLA garrison The PLA normally keeps a low profile in Hong Kong, where its presence is still a sensitive subject. Hide Caption 4 of 9 Photos: China's Xi parades at Hong Kong PLA garrison Around 6,000 troops are based in the city, with their headquarters in Admiralty, next to the Hong Kong government offices. Hide Caption 5 of 9 Photos: China's Xi parades at Hong Kong PLA garrison According to a recent article in a highly respected Communist Party journal, the Hong Kong garrison has transformed from a "symbolic presence to a show of force." Hide Caption 6 of 9 Photos: China's Xi parades at Hong Kong PLA garrison A small crowd of onlookers were specially invited to the parade alongside Xi. Hide Caption 7 of 9 Photos: China's Xi parades at Hong Kong PLA garrison Security at the event was tight, with reporters told not to bring pens, phone batteries, or umbrellas. Hide Caption 8 of 9 Photos: China's Xi parades at Hong Kong PLA garrison On July 1, 2017, Hong Kong marks 20 years of Chinese rule at a ceremony overseen by Xi. Hide Caption 9 of 9

Multiple attempts by protesters to get anywhere near Xi during his time in the city ended in failure, while a pro-Hong Kong independence rally scheduled for Friday night was canceled after police refused organizers permission to use a public square.

Two dozen activists were arrested Wednesday after they stormed and occupied the public square where the flag-raising took place. The protesters, including "Umbrella Movement" leaders Joshua Wong and Nathan Law, were not released until early Friday morning.

JUST WATCHED Hong Kong residents fear new law Replay More Videos ... MUST WATCH Hong Kong residents fear new law 02:45

Protests and challenges

The mood among many pro-democracy campaigners was somewhat muted during Xi's visit, nothing like the anger and frustration that saw hundreds of thousands occupy the streets of central Hong Kong during 2014's "Umbrella Movement."

But deep rifts in society remain, a problem that Chief Executive Lam has vowed to tackle

Lam has indicated she may revive efforts to pass a controversial anti-sedition law known as Article 23.

Article 23 requires the Hong Kong government to "enact laws on its own to prohibit any act of treason, secession, sedition, subversion against the Central People's Government."

Some warn the law could be used as an excuse to crack down on criticism of the government or the discussion of controversial topics like Hong Kong independence.

Many locals fear the law could be the final nail in the coffin for the city's autonomy from China. A previous attempt to introduce the legislation drew half a million protesters onto the streets and resulted in the resignation of a key minister in 2003.

On Friday, Chinese Foreign Ministry spokesman Lu Kang said the Sino-British Joint Declaration, the agreement by which Hong Kong was handed over from the UK to China, was now just an "historical document" with no practical significance.

Chris Patten, the city's last British colonial governor who presided over the handover in 1997, said that China had been pressuring Hong Kong "in all sort of ways."

"You see this in attacks on the judiciary and the rule of law, interference in court cases, you see it in the atmosphere of hostility toward the independence of universities and the media," he told CNN before the anniversary.

A moving copy of Picasso's Guernica with an empty chair for Liu Xiaobo: pic.twitter.com/7DE3hVlgFz — James Griffiths (@jgriffiths) July 1, 2017

However, for one Chinese student visiting the city, Hong Kong, with its right to protest still intact, still feels distinct.

"It's very different from the mainland environment," said Liu, 18, who only gave her surname, from the sidelines of the protest march.

"(In China) we'd never have any opportunity to speak out like this."