Thousands of Chinese nationals have been deported from Ghana in a crackdown on the plunder of the country's gold by illegal miners.

Immigration authorities say more than 4,500 Chinese nationals have been repatriated after a series of swoops on illegal goldmines. The Guardian has learned that Ghana's government, which depends heavily on China for billions of dollars in loans and as a major trading partner, believes Beijing may be retaliating, damaging relations between the two countries.

"Of late we have seen a tightening of the visa regime at the Chinese embassy for Ghanaians. We don't know whether this is a manifestation of our actions to deport illegal Chinese goldminers," said Alhaji Inusah Fuseini, Ghana's minister of lands and mines.

"This is a matter for concern. Relations between China and Ghana go back a long way, and they had been on the rise until we started the expulsion of foreigners from our illegal mining sites. We didn't think China would take it to this extent."

China has denied that any of its actions are a retaliation for the deportation of illegal immigrants from Ghana. Ghana's government said claims that delays in Ghana being able to access a $3bn loan facility agreed with China recently were related to the current events. So far this year 1,577 illegal foreign miners have been arrested by the Ghanaian authorities, and more than 3,800 have submitted to deportation on a voluntary basis, according to the ministry for lands and mines. Immigration authorities said that in June and July alone 4,592 Chinese illegal immigrants were deported.

"All these Chinese were involved in illegal employment in Ghana, the majority in illegal goldmining," said Michael Amoako-Atta, a spokesman for Ghana's immigration service. "This is only the first phase of our operations, we will be continuing in due course."

This year the Guardian exposed the scale of illegal Chinese goldmining in Ghana in a film that showed widespread excavating and use of toxic chemicals, and allegations of human rights abuses. Following the film Ghana's president, John Dramani Mahama, established a high-level taskforce, saying it was necessary to bring "sanity" to the mining sector.

Since then the government has launched a series of raids through the combined military, immigration and police taskforce. Towns that had swelled with the presence of illegal miners have been significantly affected, officials say. In Dunkwa, where the Guardian filmed this year, sources said the departure of Chinese had drastically affected the local economy.

"A very big change has happened in Dunkwa," said one source, who did not want to be named. "They were the ones who provided the mining equipment – most of the Ghanaians left behind cannot continue their operations. It has really affected the mining. The local people are complaining because they say the Chinese were good for business. Now everything has slowed down."

But many Ghanaians have lauded the government's efforts to curb illegal mining. The Chinese have attracted heavy criticism from Ghanaians for taking local jobs, wielding weapons such as AK-47 rifles, and polluting lakes and rivers.

"This illegal goldmining was compromising the environment. It was compromising the security of this country. It had a lot of social consequences and the government was losing a lot of resource revenue," said Fuseini, the mines minister. "The Chinese were just plundering the resources. So what we have done is take action to stop the plunder and ensure that the resource needs of this country are not any way depleted in ways that breaks resource revenue of the state."