Rory Stewart has said it would be “heartbreaking” to leave his job as international development secretary were Boris Johnson to become the next prime minister.

Stewart, an anti no-deal candidate who was knocked out of the Tory leadership contest after last week after a television debate, has vowed not to serve in a Johnson cabinet.

“It’s heartbreaking,” he said. “It’s the proudest I’ve been. It’s the greatest job I’ve ever had in government. It’s an amazing department that makes a huge difference around the world. And it’s greatly admired.

“I’ve worked with the Department for International Development in Iraq, Afghanistan going back over 15, 16 years. It really was the dream of my career, but unfortunately we will have to see who the next prime minister is.”

The former prisons minister, who had worked previously at DfID before he took over from Penny Mordaunt last month, said Johnson’s stance on Europe would make it impossible to stay. Stewart, who has backed Johnson’s rival Jeremy Hunt in the Conservative leadership contest, warned that a no-deal Brexit would make it harder for Britain to attain UN targets to lift people out of poverty.

“Unfortunately, I wouldn’t be able to remain in this job if Boris was prime minister,” said Stewart. “I don’t think we he would want me, for a start, but it’s also true that I’m afraid on the very fundamental issue of the day, which is Brexit.

“I have said repeatedly that he is trying to deliver something that he can’t deliver and he’s going to disappoint people deeply. I wouldn’t be able to be a member of a cabinet arguing for a policy that I think is misleading.”

Stewart was at Torriano primary school in Camden, north London, to face questions from schoolchildren and to talk about the UK’s progress towards the UN’s global goals. A DfID review said that while Britain has made “significant strides” on the targets, much remained to be done – particularly on the key UK priorities of reducing poverty and ensuring “no one is left behind”.

Stewart said avoiding a no-deal Brexit was essential to achieving these goals. Asked how the government could better achieve its aim to reduce poverty, he said:“From my point of view, we have to avoid a no-deal Brexit. It sounds like a daft thing to say, but I think the economic damage it would do would reduce the amount of money that we have available. If we can deliver a sensible solution to Brexit there should be £26m or £27m more available, and I would like to see that invested firstly in education, and we should put more into policing and the NHS.”

The international development secretary, who pledged to put the climate crisis and the environment at DfID’s heart, said the world was at risk of becoming “much more difficult and unpleasant” within 50 years without sweeping changes in sustainability.

Stewart said he expected Johnson to retain the 0.7% target of gross national income for overseas aid, and hoped DfID would remain a standalone department despite Johnson’s previous suggestions that it should be brought into the Foreign Office.

“My understanding is that Boris is committed to keeping 0.7% and that’s really important, because there were candidates in this race that were trying to get rid of 0.7%. That is central, partly because climate and the environment is something that is so intermeshed in the way we think about development and poverty, and the scale of the challenge we’re facing on climate and the environment is literally multiple trillions of pounds.”

Esther McVey, an early leadership candidate, has suggested the 0.7% target should be cut.

Stewart said that he felt close co-ordination between DfID and the Foreign Office would help in conflict zones like Somalia and South Sudan, but added: “I would definitely want DfID to be a standalone department. It’s also an achievement we have put into law and I would hope that whichever leader keeps this.”

During a tour of the school, which banned single-use plastic two years ago, Jonah, 11, asked Stewart about introducing a charge for bottles similar to the one introduced for plastic bags. Stewart replied: “Plastic bottles I think would be a great opportunity and so would plastic coffee mugs”.

Stewart suggested that a charge of 1p for every bag of mixed waste would lead to more people recycling.

He said if he “really had money to spend”, he would put £1bn of research and development into universities to work on technology.

“If you could develop light spectrum technology or solar film technology so the next time China builds a power generation station, instead of building it powered by coal, they built it, powered it by solar, that would probably make a bigger difference than almost anything else we did,” he told pupils.