Hong Kong (CNN) Terence Tam's dad fled to Hong Kong from China during the political chaos of the Cultural Revolution to make a better life for himself and his family.

Then a British colony, the city became home to a huge number of refugees fleeing Communism in the 1960s and 1970s.

Now 39, Tam says he's ready to make another escape, 20 years after the UK handed sovereignty over the city to China.

While Hong Kong is one of the most affluent cities in the world, he says it's now in the throes of its own political upheaval.

"I want to see if I can do the same for my family (as my father did for us)," he says. "I'm not satisfied with the situation in Hong Kong. The political situation, the government. We have so many complaints."

'I feel like we are refugees'

By most yardsticks, Tam's done well for himself. He's an assistant IT manager at one of Hong Kong's universities and owns an apartment -- no mean feat in a city that has the world's most unaffordable property prices.

But Tam objects to China's growing grip on the city.

Under the principle of "one country, two systems," Hong Kong was supposed to have preserved the rule of law, freedom of speech and the right to protest as enjoyed under the British -- at least until 2047 -- but Tam, and others, feel that China hasn't held up its part of the bargain.

"Some people say that we are emigrating but I feel like we are refugees escaping from Hong Kong," he says.

In two years, once his newborn daughter is ready for kindergarten, Tam plans to uproot his family and move to Taiwan.

Terence Tam and his wife on vacation in Hualien, eastern Taiwan.

His wife has already secured citizenship there through an investment visa scheme and Tam plans to cash out his Hong Kong pension pot and start a business selling Hong Kong snacks.

Tam is one of a growing number of people in the city looking for an exit. According to an opinion poll conducted by the Chinese University of Hong Kong last year, two in five Hong Kongers want to live overseas -- up from one in five in 2014 -- although only 10% of those who said they wanted to leave had put such a plan into action.

What's changed?

It's not the first time that political upheaval has had Hong Kongers scouting for boltholes.

From 1989, in the wake of the Tiananmen massacre, almost 60,000 people left the city every year, primarily for Canada, the United States and Australia, according to one academic study.

However, many came back as the nightmare scenarios some had imagined didn't emerge and Beijing largely left the city alone during the first 15 years of Chinese rule.

But in 2014, Beijing denied Hong Kongers the universal suffrage promised to the city under its mini-constitution, the Basic Law, triggering massive streets protests that paralyzed much of the city for more than two months.

Police fire tear gas upon pro-democracy demonstrators near the Hong Kong government headquarters on September 28, 2014.

Since then, fears have grown that Beijing neither cares about, nor pays heed, to the city's supposed autonomy.

In 2015, five men involved in publishing gossipy books on China's ruling elite -- two of whom had European passports -- disappeared only to reappear in Chinese custody weeks later.

China has also moved to bar pro-democracy lawmakers from taking office, using a rarely invoked power.

Many fear that Hong Kong's new leader Carrie Lam , who was selected by a 1,200-strong committee and takes office July 1, will now enact a controversial security law, known as Article 23 , that would "prohibit any act of treason, secession, sedition, subversion against China."

JUST WATCHED Next Hong Kong leader on missing bookseller Replay More Videos ... MUST WATCH Next Hong Kong leader on missing bookseller 03:51

Looking to Taiwan

Andrew Lo, an immigration consultant at Amlex in Hong Kong, said his company has seen an upturn in business since the 2014 Occupy protests -- largely down to the deteriorating political situation, though he says the city's pressure cooker school system and cramped high-rise living conditions also play a role.

"The main reasons why people leave Hong Kong is because of political instability -- the future is uncertain -- and poor education in Hong Kong," he says.

Schooling starts young in Hong Kong and there's lots of pressure to make the grade.

Taiwan, and southeast Asian countries like Malaysia and Thailand, have emerged as popular destinations -- particularly for younger couples and retirees, says Lo. It's also now much harder to emigrate to places like Australia and Canada than it was in the 1990s.

Tam says he chose Taiwan because it's a place he fell in love with on vacation and it has the same cultural roots as Hong Kong.

He knows that life won't be easy. Tam's native language is Cantonese and says he speaks Mandarin, the language most Taiwanese use, haltingly. Jobs are also lower paid on average in Taiwan than in Hong Kong.

"I love the culture in Taiwan. I've been to the UK to study for one year. When I landed there, the first step, I felt like I didn't belong. But when I went to Taiwan I felt different," Tam says.

Supporters shout during Tsai Ing-wen's speech on her presidential election victory on January 16, 2016 in Taipei, Taiwan.

It's also the world's only Chinese-speaking democracy -- an important drawcard.

According to official figures, the number of Taiwan permanent residents hailing from Hong Kong has more than doubled since 2014 -- though the absolute number is still very small at 1,086.

Taiwan, however, is not without its own political issues and complexities . Despite Taiwan governing its own affairs for more than six decades, China views the island as part of its territory , to be taken by force if necessary.

The two -- officially the People's Republic of China and the Republic of China -- split in 1949 following the Communist victory on the mainland after a civil war.

"I have a stable job, income and property, but I find the environment here very uncomfortable," Tam says. "We were born before 1997. Our birth certificates, our passports were issued by the UK. After 1997 we didn't immediately see any changes but we do now."

On July 1, thousands braved the city's sultry summer humidity to mark the 20th anniversary of the handover by marching through Hong Kong's streets to demand greater democracy. The annual event took on extra significance this year with Chinese President Xi Jinping visiting the city for the very first time.

Tam has joined the protest in the past but didn't this year -- he sees it as a futile act: "I have a choice, I can leave. It's time to get out."