Storm clouds loomed over Hong Kong as tens of thousands of pro-democracy protesters took to the streets, giving way to a downpour as the day wore on that deterred many from joining the throngs of dissent.

With last year’s protest at 110,000, as estimated by organisers Civil Rights Fronts, this year saw a reported 60,000 take to the streets, on a day that coincided with the end of the contentious three-day visit of China’s state leader Xi Jinping. Police calculate the number as being significantly lower, at a paltry 14,500.

The protest, which saw participants of all ages march through the bustling streets of a city campaigning for universal suffrage and against the degradation of civil liberties, comes as Hong Kong marks its 20th anniversary since it was returned to China from British colonial rule.

It has also coincided with the inauguration of new chief executive Carrie Lam, who is seen by detractors as a puppet to Beijing. The news that Nobel prize winner and Chinese dissident Liu Xiaobo is terminally ill and has only recently been granted medical parole has been another source of discontent.

The bad weather, and – according to organisers – anxiety around safety in an increasingly fraught political climate, meant turnout here on Saturday was much lower than the hundreds of thousands in previous years.

Small congregations of pro-China groups were present, but were far outnumbered by the pro-democracy contingent.

Reports of clashes between protesters and local gangsters, alongside allegations of attacks from police, marked an otherwise subdued and non-violent affair in which placards borrowing the symbols of 2014’s Umbrella Movement were wielded throughout.

The now iconic image of China’s contentious strongman leader Xi Jinping brandishing a yellow umbrella were seen among the crowds, where high profile activists, politicians, artists, and pop stars were making their presence felt.

“The handover is not a time for celebration, but a time for demonstration,” activist and founder of political party Demosisto Joshua Wong said from a booth where he and his party campaigned and solicited donations from the public.

Alleging that members of the police had manhandled members of Demosisto and another political group, the League of Social Democrats, when taking part in a rally on Saturday morning, Wong decried this treatment as an insult to Hong Kong’s enshrined right to peaceful protest.

A fellow Demosisto campaigner and Hongkonger Helena Wu, who flew in from overseas to attend the protest, said that fears of violence had impacted crowd numbers.

The third and final day of President Xi's first state visit to Hong Kong coincided with Saturday's landmark anniversary, and in a speech earlier on the Chinese leader issued a stark warning that Beijing will not tolerate any challenge to its authority in the port city.

Police blocked roads, preventing pro-democracy protesters from getting near to the harbour-front venue where Mr Xi swore in the new chief executive, not far from where the last colonial governor Chris Patten tearfully handed back Hong Kong – again in the pouring rain – in 1997.

Mr Xi said Hong Kong should crack down on moves towards “Hong Kong independence”.

“Any attempt to endanger China's sovereignty and security, challenge the power of the central government ... or use Hong Kong to carry out infiltration and sabotage activities against the mainland is an act that crosses the red line and is absolutely impermissible,” Xi said.

China said this week that it views the handover deal of 1997, which was scripted to set out a pathway towards universal suffrage for Hong Kongers, as a historical document with no relevance in the present day.

And Beijing reacted bitterly to comments from UK Foreign Secretary Boris Johnson ahead of Saturday’s anniversary, who called on all parties with an interest in the city to “progress towards a more democratic and accountable system of government”.

Chinese officials responded to the statement by chastising the Conservative front bencher for speaking about their “internal affairs”.

“Hong Kong is a special administrative region of China, and therefore Hong Kong affairs are China’s internal affairs,” said foreign ministry spokesman Lu Kang.