Young activists have formed new groups promoting independence or a local Hong Kong identity separate from the mainland, alarming Beijing.

In a speech marking 20 years since the city became a semi-autonomous Chinese region after its handover from Britain, China President Xi Jinping pledged Beijing’s support for the “one country, two systems” blueprint, under which Hong Kong controls many of its own affairs and retains civil liberties including free speech.

And he appeared to put on notice a new wave of activists pushing for more autonomy or even independence, saying challenges to the power of China’s central government and Hong Kong’s leaders wouldn’t be tolerated.

Any attempt to challenge China’s sovereignty, security and government authority or use Hong Kong to “carry out infiltration and sabotage activities against the mainland is an act that crosses the red line, and is absolutely impermissible,” Mr. Xi said, moments after presiding over the inauguration of Hong Kong’s new leader, Carrie Lam.

“Making everything political or deliberately creating differences and provoking confrontations will not resolve the problems,” Mr. Xi said. Hong Kong “cannot afford to be torn apart by reckless moves or internal rifts.”

Young activists have formed new groups promoting independence or a local Hong Kong identity separate from the mainland, alarming Beijing.

Meanwhile, incidents such as the secret detentions of five Hong Kong booksellers on the mainland have stirred fears that Beijing is undermining the “one country, two systems” blueprint.

Mr. Xi’s speech “was a mixture of reassurance and warning,” as he signaled that the system in place since 1997 won’t change, said Jean Pierre Cabestan, an expert on Chinese politics at Hong Kong Baptist University. “At the same time, there was a strong warning to the localists and the pro-independence people.”

While former colonial master Britain and other Western democracies have expressed concerns about Beijing’s actions in Hong Kong, China has increasingly made clear it brooks no outside criticism or attempts at intervention.

Mr. Xi said China had made it “categorically clear” in talks with Britain in the 1980s that “sovereignty is not for negotiation.”

“Now that Hong Kong has returned to China, it is all the more important for us to firmly uphold China’s sovereignty, security and development interests,” he said.

Chinese foreign ministry spokesman Lu Kang sent a similar message in Beijing on Friday, saying Hong Kong was strictly China’s domestic affair.

The 1984 Sino-British Joint Declaration laying out terms for Hong Kong’s return to Chinese rule is “no longer relevant today, and has no binding force on the Chinese central government’s governance over Hong Kong,” Lu said.

“The U.K. has no sovereignty, governance right or the right of supervision over today’s Hong Kong,” Lu said.

Mr. Xi oversaw a swearing-in ceremony for Ms. Lam, Hong Kong’s fifth chief executive since 1997 and first female in the job. The career civil servant and her Cabinet swore to serve China and Hong Kong and to uphold the Basic Law, the territory’s mini-constitution.

In a speech that ran a fraction of Mr. Xi’s 32-minute address, Ms. Lam reviewed the dynamic financial center’s achievements and challenges, pledged to support central government initiatives and declared “the future is bright.”

There was other symbolism hinting at the balance of power.

Ms. Lam took her oath of office and delivered her address in Mandarin, China’s official language, save for a few lines at the end in Hong Kong’s Cantonese dialect. The official transcript of Mr. Xi’s speech was printed in the mainland’s simplified characters instead of Hong Kong’s traditional complex characters.

Even the Chinese flag displayed behind Mr. Xi as he spoke was noticeably larger than Hong Kong’s beside it.

“It speaks volumes to me who is the boss, who is calling the shots,” said Cabestan.

Ms. Lam prevailed over a much more popular rival in selection process decried by many as fundamentally undemocratic, with only 777 votes from a 1,200-seat panel of mostly pro-Beijing elites. Hong Kong has more than 3 million registered voters.

Mr. Xi later departed for Beijing, ending a three-day visit under heavy security aimed at stirring Chinese patriotism.

Ahead of a flag raising ceremony Sunday, a small group of activists linked to the pro-democracy opposition sought to march on the inauguration venue carrying a replica coffin symbolizing the death of the territory’s civil liberties. They were swiftly stopped by police and pro-China flag-waving counter protesters in an hour-long standoff.

Other protests were planned later Saturday, including an annual march that often draws tens of thousands.