The immigration minister, Peter Dutton, has said refugees aren’t numerate or literate and would take Australian jobs in inflammatory comments arguing against increasing the refugee intake.

Labor has said the comments are deeply offensive and called on Malcolm Turnbull to “unequivocally reject” them.

On Tuesday evening, Dutton made the remarks on Sky in response to a question about the Greens’ policy to increase the refugee intake to 50,000.

The show’s host, Paul Murray, said 90% of Afghan refugees don’t have a job and asked: “What on God’s Earth are these people suggesting about 50,000 being taken from every hell-hole of the world?”

Dutton responded: “For many people – they won’t be numerate or literate in their own language, let alone English.”

.@PeterDutton_MP says more refugees will take Australian's jobs or languish in unemployment queues #ausvotes #pmlive https://t.co/TdFlaQc8Tb — Sky News Australia (@SkyNewsAust) May 17, 2016

Dutton said the Greens were “very close to the [Construction Forestry Mining and Energy Union]” and Labor had “obviously well-known” affiliations with the union movement, implying that increasing the humanitarian intake would be unpopular with unions. “These people would be taking Australian jobs, there’s no question about that.

“For many of them that would be unemployed, they would languish in unemployment queues and on Medicare and the rest of it, so there would be huge cost and there’s no sense in sugar-coating that, that’s the scenario.”

In July the Labor national conference approved a policy mimicking the Coalition’s government’s strategy of turning back refugee boats. At the same conference it approved an increase of the humanitarian intake of refugees from 13,750 to 27,000 by 2025, which was supported by many union delegates.

The shadow treasurer, Chris Bowen, rebuked Dutton for the comments: “There are hundreds of thousands of refugees in Australia who have worked hard, who have educated themselves and their children and they will be shaking their heads at their minister today in disgust. He owes an apology, and it’s not to the Labor party, it’s them.

“Australia takes refugees for the right reasons, because of humane reasons, but we have also benefited over the years from the contribution of refugees. Most Australians would tell him he is just plain wrong.”

Peter Dutton's scare mongering on refugees over jobs exposes the Liberal policy as being steeped not in care for ppl in need, but xenophobia — Sarah Hanson-Young (@sarahinthesen8) May 17, 2016

Asked about Dutton’s comments on the literacy and numeracy of asylum seekers, the foreign minister, Julie Bishop, said: “Let’s have a reality check here – of course the cost of ensuring people who come here to Australia as a refugee, [or on] a humanitarian visa, is very high.”

She said the government’s plan to take 12,000 Syrian refugees was costed at $700m.

“Peter Dutton is pointing out the self-evident fact that it costs a great deal of money to settle people in Australia. And, of course, the Greens never have to account for a budget, the costs of doing so.

“It’s just another example where they are so out of touch with reality. There is a cost and the Australian people bear it.”

Bishop said the costs included education, to teach refugees English. She said these costs ensured people could contribute to Australian society by speaking English, getting jobs and making a contribution.

When asked whether taking Australian jobs was a consideration when granting people asylum, Bishop said: “It’s not one of the considerations, what he’s pointing out is we would want such people to have a job if they are going to be in Australia, we don’t want them to be on welfare.”

“We bring about 13,750 refugees and those on humanitarian visas to Australia each year, in addition the 12,000 Syrians we promised to resettle and then that 13,750 will increase to 18,750 and that has to be paid for – Peter Dutton is talking about the very real cost of doing so.”

.@JulieBishopMP responds to Peter Dutton's comments on refugees not being literate and numerate #ausvotes https://t.co/TFhbNhkbep — Sky News Australia (@SkyNewsAust) May 17, 2016

The opposition’s immigration spokesman, Richard Marles, said the comments were regrettable and deeply offensive.

“It’s a test now for Malcolm Turnbull: he needs to unequivocally reject these comments. This goes well beyond the question of expense as Julie Bishop kept referring to,” he said.

“What Peter Dutton referred to was illiterate, innumerate refugees, refugees taking Australian jobs. Now, we all know what Peter Dutton is trying to do with those sorts of comments.”

“Malcolm Turnbull must come out and unequivocally reject those comments, he can’t squib it like Julie Bishop ... if he doesn’t, we all know the transformation is complete, Malcolm Turnbull is simply leading the Abbott government.”

Asked about Dutton’s comments in Cairns on Wednesday, Turnbull said Australia had the most successful multicultural society in the world and a very generous humanitarian program.

“Now, the reason we are successful is because we invest an enormous amount of money into the settlement services, to make sure that our refugees who come to Australia get the language instruction, all of the support to enable them to integrate into Australian society and move into employment and take up those opportunities,” he said

“So we put – we invest more in settlement than many other countries do. So it is – it’s very expensive. We don’t begrudge the money but it’s important to get it right.”

Turnbull made no further comment on Dutton’s remarks, but suggested he might take more questions on it at a doorstop on Wednesday in Townsville.

The deputy prime minister, Barnaby Joyce, said: “We are an incredibly compassionate nation … we are a strong nation, we are a good nation, I am very proud of the work we do in getting refugees in our nation and we do that in making sure they don’t arrive by sea because we don’t want to be responsible for people drowning.

“We want to show our compassion by finding people who are under threat of their life somewhere else and giving them support of our nation and allowing them to come here.”