Boris Johnson is yet to win over some of those he expelled from the Conservative party to vote for his EU exit deal, including the former chancellor Philip Hammond, who fears doing so could lead to a much harder Brexit than expected.

Johnson will need the backing almost all of the 21 who had the whip removed, plus Eurosceptics and either the Democratic Unionist party or a bloc of Labour MPs, if he is to have a chance of passing any Brexit deal he achieves.

However, No 10 has yet to secure promises from many of the anti-no deal rebels that they will vote for an agreement. Senior government figures have been asking the former Tories what they would need to vote for a deal and prevent an extension, but many have worries about being pressed into voting for an agreement with insufficient scrutiny.

A source involved in the group led by the former chancellor Philip Hammond said: “There are versions of it that are pretty suboptimal. Philip, [David] Gauke, Amber [Rudd] and the others have said a few times this needs to be something that works for the whole of the UK, and if this breaches the union there will be some concerns about that.

“On some things there will be a strange sort of common concern that draws from the ERG to DUP to former Tory rebels along the lines of the union, perhaps for different reasons.

“I don’t think it’s a dead cert for any of them that they will vote for it … I will be very much studying the substance but there are concerns about the shape [of] what has been submitted so far. It is also that Theresa May made a commitment to parliament over its role on the future relationship and the big question will be: what shape does the political declaration take?

“They are only going on what they’ve read in the media, but none of them are people who will just vote for any deal. There are obviously different concerns within the group about what they want to see but it’s broadly along the lines of something that has to work for the whole of the UK.”

Hammond later told the BBC’s Andrew Neil he may not support Johnson’s deal. He said his mind was “certainly not closed to supporting a deal the PM brings back”, but he was worried about proposed changes to the political declaration that could lead to a more distant trading relationship with the EU in future.

“What I am hearing from Brussels is that the government is seeking to take out things from the future relationship,” he said. “I don’t think we can judge any package until we see the full package. We can’t even know what it will mean for Northern Ireland.

“If GB has tariff-free trade with the EU then goods from GB to Northern Ireland will flow freely, but if it doesn’t have tariff-free trade then there will be a hard border in the Irish Sea and that will be problematic.”

So far, only a handful of MPs, including Anne Milton and Caroline Nokes, have been invited into Downing Street for private talks on what they could accept. Others, such as Hammond and Gauke, have not yet been summoned.

Another source close to the 21 former Tory MPs said: “A deal for Northern Ireland and a very hard Brexit for the rest of the UK is much worse than what Theresa May got.”

Despite reservations, Margot James, a member of the group, said she would probably vote for the deal even though she has concerns about what is being proposed.

“I’m very keen to avoid leaving without a deal and therefore I will certainly give whatever proposals can be agreed with the European Union a very favourable look, no matter how many reservations I have about what I’m hearing,” the former business minister told BBC Radio 4’s The World at One.

“All I can say at the moment is I would give it the benefit of the doubt. I don’t want to vote against it, but yes I am very concerned.”

Dominic Grieve, Justine Greening and Guto Bebb are likely to vote against a Brexit deal unless it has a second referendum attached.

Grieve said on Wednesday that he would push for an extension to article 50 to go ahead regardless of whether the prime minister brings back the outline of a deal from the European summit this week.

“It’s clear that a final Brexit deal will not be agreed this week … That means the Benn act comes into play, and an extension to the article 50 deadline must be requested by law,” he said after meetings in Brussels.

There have been suggestions that the former Conservatives could have the whip restored if they vote for a deal, although Sam Gyimah has defected to the Liberal Democrats and is committed to a second referendum, while Rory Stewart is running as an independent to be mayor of London.

At the same time, Tory sources have suggested Eurosceptics who refuse to back any deal proposed by the prime minister could also have the whip removed.

James said it would be “good for them not to operate a system of double standards” and for No 10 to sack any Tories who do not back Johnson’s withdrawal agreement.

“I think that’s the thinking behind why the whip has not been offered back to us,” the independent MP said. “The government wants to make clear this is a highly disciplined environment and all members of parliament on the Conservative side should follow the Conservative whip in a vote as important as this one.”