Immigration minister to signal more selective policy under which only the right kind of migrants are allowed to enter Britain

Critics have accused the government of paving the way for a selective immigration policy whereby only the wealthy will be able to marry who they want from abroad, and only migrants earning more than £31,000 a year will be able to settle in Britain.

The immigration minister, Damian Green, will confirm on Thursday that ministers want to move to a more highly selective policy under which only the right kind of migrants are allowed to enter Britain.

"We need to know not just that the right numbers of people are coming here but that the right people are coming here. People who will benefit Britain, not just those who benefit by Britain," Green will argue in a speech to the Policy Exchange thinktank.

Green also wants to move the immigration debate on from the single issue of numbers that has dominated for the past decade and instead focus on the benefit to Britain of allowing only the "brightest and the best" into the UK.

He says the pressure to reduce net migration numbers to "tens of thousands" will continue as it has since the general election.

His desire to "raise the tone of the debate" follows the latest figures showing net migration to Britain reaching a record 252,000 in 2010. Green insists there are "the first small signs" that numbers have been falling since then.

"What we need is a national consensus on how we can make immigration work for Britain. We are evidently a long way from such a consensus but I want to start to build it … the legitimate question in today's world is how we can benefit from immigration," he will say.

Green will promise that by May – the second anniversary of the coalition – the government will have announced or implemented changes to all the main routes of immigration and broken the link between migration and staying permanently in Britain.

He will confirm that major changes in family visas are about to be announced which will mean that husbands, wives or fiancés who cannot speak English and are likely to be dependent on benefits will be barred: "Importing economic dependency on the state is unacceptable," the immigration minister will say. "Bringing people to this country who can play no role in the life of this country is equally unacceptable."

Critics say a proposal to ban any British resident from bringing an overseas-born spouse into the country unless they have a minimum household income of £25,700 a year will cover half the UK's working population.

Green will also confirm that he is considering setting a minimum income threshold of between £31,000 and £49,000 a year, below which migrants who legally come to work in Britain will lose the right to apply to stay permanently after five years.

"We will end the assumption that settlement is an option for all those who come to work. Instead, we will accord it to the brightest and the best," Green will say, adding that pay levels will be used to define who fits into that category. The minister will hint that he intends to replace the "post-study work route" for graduates with a new more limited "graduate entrepreneur" specialist category.

He will also acknowledge recent recurrent visa problems for visiting artists, entertainers and business people that have discouraged even world-class performers from coming to Britain: "I am aware that this has been a sore point for some time, and we are taking action. The system does work well for most people," he insists.

Matt Cavanagh, associate director of the Institute for Public Policy Research, said it made sense to give some priority to wealthy migrants and to ask whether someone on benefits should be able to bring in a spouse.

He said: "But the government is going much further: essentially their approach is, if you're a wealthy migrant, you can come, you can stay as long as you like; if you're a wealthy resident, you can marry whoever you like; but for everyone else, it is going to get much more difficult.

"And we're not talking about people who are destitute or living on benefits, we are talking about people who are working and getting an average wage. Migrants will still be invited to come and work at these wages [between £20,000 and £30,000] to fill jobs where we lack the skills or nobody else wants to do the work, but after five years they will be asked to leave, regardless of the contribution they have made or could make in the future.

"Likewise, not just people living on benefits but almost half the British population, could lose the right to marry and live with someone from abroad."

Habib Rahman of the Joint Council for the Welfare of Immigrants, said it was very worried about Green's plans on family reunion visas.

"The rights of British people to live with their loved ones here, and the splitting of families in the coalition government's reckless pursuit of lower net immigration figures are the human cost of this insane numbers game, the Home Office is playing," he said.