A former Conservative minister who resigned over the government’s handling of Brexit has lost a vote of no confidence in him by his local association, after he was outspoken in his support for a vote on the final deal.

Phillip Lee, MP for Bracknell in Berkshire, said he recognised that a majority of his Conservative association’s members opposed his view on Brexit, but argued he was elected to represent the best interests of everyone in his constituency.

Lee supported remain in the EU referendum and was the first government minister to resign over Brexit.

Bracknell Forest – an area including the towns of Bracknell, Sandhurst and Crowthorne – voted to leave the EU in 2016, with 53.9% of voters in favour of exiting the bloc.

“I have never been in any doubt that a majority of my Conservative association’s membership have opposed my view on Brexit, mostly at the same time as agreeing with me that the government’s withdrawal agreement would be bad for our country and should be rejected – as I have repeatedly done in parliament,” the former justice minister wrote on his website.

“However, as a Member of Parliament, I do not represent only the views of Conservative association members. I do not even represent only the views of Conservative voters.

“Rather, I am elected to represent the best interests of everyone in the Bracknell constituency – young and old, of every political persuasion and none and including those of future generations who are still too young to vote.”

He added that he would not be “forced into taking a decision” on his future by an “orchestrated, destructive campaign from outside the party”.

Bracknell Conservative Association called the vote after 53 members signed a petition.

The association chair, Gerry Barber, told the BBC: “The result of the vote was that a majority of members present were in agreement with the motion, which was therefore passed, and the result has been communicated to Dr Lee and to the full membership. I will be discussing the meeting with Phillip later this week.”