Getting the draft Brexit agreement approved by a largely hostile Parliament will be a "hard job" for U.K. Prime Minister Theresa May, and a second referendum is out of the question, according to a former advisor to the late Margaret Thatcher.

On Thursday, a flurry of ministers resigned from their positions, including Brexit Secretary Dominic Raab, in protest at May's proposals. Their resignations came hours after May said she had obtained enough support from her senior ministers for her draft Brexit deal to move forward.

It will be a "hard job" to get the draft proposal through Parliament "because Prime Minister May has upset people from all sides, including the 'Remainers' ... they feel that we're still in the European Union with no seat on the board," John Browne, who was also a former member of the British Parliament, told CNBC's "Squawk Box" on Friday. Remainers are people who voted for the U.K. to stay as part of the European Union.

"I don't think there's a chance of a second referendum. The referendum is already been done. It's like trying to re-run the derby if you didn't have the winning horse — so, it's already been done. There's Brexit," he added.

May could also potentially face a leadership challenge if enough Conservative lawmakers submit letters of no confidence in the prime minister.