Pro-Brexit Conservative MPs walked out of a meeting of parliament’s committee on leaving the EU after objecting to what they felt to be the negative tone of a report prepared by its Labour chairman, Hilary Benn.

About a third of the committee left the meeting, which was convened to discuss a report on the UK government’s white paper on Brexit.

About two-thirds of the committee were relatively happy with the 150-page draft version of the report, which is understood to have been highly critical of the prospect of the UK leaving the EU without a deal.

However, all the pro-Brexit Tory MPs present, including Maria Caulfield, Karl McCartney, Dominic Raab and Craig Mackinlay, decided to boycott the process after about an hour of discussion. Michael Gove and Peter Lilley were among those absent.

The move calls into question the ability of the crucial scrutiny committee to do its job without a degree of consensus among its members, who are deeply split between those who supported leave and those who voted remain at the referendum. The boycott also means the committee may have to present a report that is not unanimous unless substantial elements are changed to allow the Brexiters to support its conclusions.

One source present said the walkout was polite but many of the Conservative MP committee members who campaigned for Brexit could not support some of its conclusions and felt the chairman was trying to “bounce” them into supporting it.

The source said Benn had been told the report was flawed in such an overarching way that none of the pro-Brexit MPs would bother to put down amendments to try to make changes. They said it was pointless to spend another eight hours considering the report when there were such fundamental disagreements about its focus on the risks rather than the opportunities of Brexit.

Following the walkout, the other committee members continued their meeting. Alistair Burt, a Conservative MP who backed remain at the referendum, said the “rest of committee [is] still sitting and doing its job to consider [the report]”.

Another pro-remain MP told the Guardian that it was more a “shuffle-out” than a “walkout” after a fruitless discussion for about an hour and 15 minutes.

The report was written by parliamentary clerks and Benn, who supported remain at the referendum but voted in favour of triggering article 50.

The senior Labour MP had told the committee he had tried very carefully to tie the report to the evidence, but one of the pro-Brexit Tories argued it seemed like the “Sir Ivan Rogers” report – claiming it was too heavily reliant on evidence given by the former UK ambassador to the EU.

Rogers told the committee last month the consequences of the UK securing no deal and relying on WTO rules would be “nuts” and like falling off the “cliff edge” into a “legal void”.

“If you had an abrupt cliff edge with real world consequences, you’ve seen what Mark Carney [governor of the Bank of England] has said about the financial stability risks to the eurozone of an abrupt cliff edge. There are other consequences in other sectors which would make it an insane thing to do,” he told the MPs in February.

One MP said that this appeared to have riled the Tory MPs, as Theresa May has said no deal is better than a bad deal with the EU. “A number had emailed Hilary beforehand to say the report should not be published at all but it was clear they did not have the numbers to do anything about it, so they made their point,” he said. “There was no point where they were actually able to point to anything that was just opinion. It was very carefully referenced.

“If consensus can’t be reached and people are not even prepared to accept the principle of making a report, then there is not much you can do. It was an unfortunate turn of events as we managed to find agreement in the two previous reports.”

Another source close to the committee said the draft report was excoriating about the prospect of no deal based on the evidence given by David Davis, the Brexit secretary, who ended up admitting that he had not calculated the cost of not having a deal with the EU.

Davis told the committee earlier this month: “Much of this is about mitigation. Any forecast that you make depends on the mitigation. As a result, it is rather otiose to do the forecast before you have concluded what mitigation is possible.”