Leave supporters are losing faith in the Conservatives to lead the Brexit negotiations, according to the latest Opinium poll for the Observer.

With Brexiters in the cabinet also fearing that the issue is being mishandled by Downing Street, the poll showed that only 41% of leave voters trust the Tories most to lead the negotiations – the lowest level since January 2017.

Overall, 32% of voters trust the Conservatives most to handle the Brexit negotiations, compared with 21% who think Labour would do a better job. The Conservative lead of 11 points on this issue is the lowest recorded by the pollster so far this year.

There have been growing complaints among pro-Brexit supporters that Theresa May has set a course for a soft Brexit that could see Britain stay close to EU customs arrangements and regulations.

In terms of overall voting intention, the Conservative lead has been cut to two points from the four-point lead it enjoyed last month. The Tories are on 42% of the vote, with Labour on 40%. The Liberal Democrats record 7% support.

Jeremy Corbyn’s personal ratings have improved slightly, from -18% last week to -13%. May’s lead over Corbyn on who would make the best prime minister has dropped from 13 points to 10 points.

The poll shows the public is shifting towards wanting to stay in the EU’s single market. Currently 38% would prefer to stay in the single market even if it means allowing free movement of Labour, while 34% would prefer to end free movement of labour even if it means we leave the single market.

However, the public remain unmoved on whether or not to have a new referendum. Asked if there should be another in-out referendum on the final deal, 38% said there should be, while 48% said there should not be.

In terms of May’s future, if it looks unlikely that she can deliver Brexit by the original timeline of March 2019, half (50%) think she should resign as prime minister in favour of someone who has a greater chance of success.

Opinium interviewed 2,005 adults online between 5 June and 7 June.