Labour has called on David Cameron to take action against a Conservative councillor who has made what appear to be "openly racist" remarks about the prospect of ethnic-minority schoolchildren from London studying in West Sussex.

The Durand Academy in Stockwell, south London, plans to open a boarding school for 600 students in Stedham village. The move is being backed by the Department for Education, which describes the plan as an "inspirational project". However, some Stedham residents are reported to be bitterly opposed to the academy, which will occupy an 8-hectare (20-acre) site that used to be a school for children with special needs.

John Cherry, the district councillor for Stedham ward, told the Mail on Sunday he had serious concerns about the effect the London pupils would have on the area.

"Ninety-seven per cent of pupils will be black or Asian," he said. "It depends what type of Asian. If they're Chinese, they'll rise to the top. If they're Indian they'll rise to the top. If they're Pakistani they won't."

He said while there were certain nationalities who valued hard work, there were also "certain nationalities where they are uncertain what this hard work is all about".

Cherry also appeared to suggest that keeping the students confined to the school premises would prove very dangerous.

"If the children are not allowed out of the site then it will make them want to escape into the forest – it will be a sexual volcano," he said. "Has anyone asked whether these children want to be plucked from their natural surroundings? They have never done boarding before, so they won't know how it works."

The trauma of transplanting the students to a new environment, he said, would be considerable. "Stockwell is a coloured area – I have no problem with that," he said. "To be honest, I would far rather Durand took over a secondary school in London, rather than shoving everybody here."

His comments were attacked by the Labour education spokesman, Stephen Twigg, who urged the prime minister to act. "When a Tory councillor makes openly racist comments like these, it's no surprise people still think of the Conservatives as the nasty party," he said.

"David Cameron must condemn his councillor's words and take immediate action against Councillor Cherry to show that he will not accept racism in his party."

A spokeswoman for the Department for Education praised Durand's "rigorous curriculum, great teaching and sky-high expectations for all pupils" and said it had a superb record on helping disadvantaged pupils achieve excellent results.

"Durand's boarding school is a bold experiment and a chance to give inner city youngsters a truly world-class education," she said. "It is difficult to believe that anyone would want to obstruct such an inspirational project."

Cherry, who has lived in the area for more than 40 years, says he is committed to education, the environment and political transparency. "My Christian beliefs lead me to putting people first," he says on the Chichester Conservatives website.

"I continue to work on keeping costs and bureaucracy down and keeping council tax down; also protecting our wonderful environment from over-development; maintaining the best schools we can afford and, last but not least supporting honesty and clarity in local government."

Kate Hoey, the Vauxhall MP whose constituency contains the Durand Academy, said on Twitter: "I invite Cllr Cherry to visit Durand Academy in Stockwell to meet those who will be boarding. Stedham will gain hugely from their presence."

A Conservative party spokesperson, said: "These comments are totally unacceptable and do not reflect the views of the Conservative party and we are investigating."

Cherry declined to comment on the matter. A woman answering the phone at his home would not let the Guardian speak to him or say whether the comments attributed to him were accurate.