The former Tory attorney general has been called a ‘traitor’ in the rightwing press for his refusal to accept a hard Brexit. But he remains committed to softening our exit from the EU – and says it is time for the foreign secretary to go

The pseudoscience of physiognomy has long been discredited, and with good reason. It seems absurd that criminologists, for example, ever believed a scoundrel could be identified by the incline of his forehead. And yet, in Dominic Grieve’s company, it is almost possible to believe a man’s character can be inferred from his facial features.

The Oxford-educated son of a QC turned Tory MP, who followed in his father’s footsteps from the bar to parliament to become attorney general under David Cameron, so closely resembles an artist’s impression of the British establishment that the only thing missing from the picture is a top hat. I don’t think I have ever met anyone who looks so exactly like his biography.

It would be hard, in other words, to imagine anyone looking less like a rebel. The 61-year-old has never picked up a placard in his life and the only march he ever joined was for the Countryside Alliance, for the right to hunt foxes. In his 21-year parliamentary career, until last December the member for Beaconsfield had voted against his party just once – over HS2 – and as he says himself: “I came into parliament to do things, so I don’t particularly relish being a rebel. Apart from everything else, I’ve got my friendships with my Conservative colleagues. I am a Conservative, so I don’t wish to be seen as a rebel, particularly, at all.”

I feel greater loyalty to the prime minister than, apparently, some members of her cabinet may have

Nevertheless, since Grieve voted against the government over the EU withdrawal bill in December, the Daily Telegraph and the Mail have branded him a “mutineer” and a “traitor”, his office has received death threats and a furious constituency activist has suggested he “go and join the Liberal Democrats”. The Sun even reported that the Commons tea room emptied when Grieve walked in for breakfast the morning after the vote. “As soon as he sat down,” according to a source, “everyone scarpered.” There is an obvious irony, therefore, in the context of our meeting this week, after Boris Johnson’s denouncement of Theresa May’s “crazy” customs partnership proposal. “I feel greater loyalty to the prime minister than, apparently, some members of her cabinet may have,” observes Grieve drily, before spelling out his contempt for Johnson’s disloyalty.

“The prime minister has a difficult task and I don’t think it’s helped by this tendency of the foreign secretary to express himself. If you are making policies through speeches that are contradicting some of the policy development your colleagues are embarked on, you are destroying collective responsibility. And, ultimately, you will utterly undermine the ability to provide democratic governance. That’s what he’s doing – and he shouldn’t do it. If he continues doing it, what he does is make government impossible. I think it’s a profound mistake and hugely damaging. If you don’t like a policy, you leave the government. That’s what you should do. If there are problems, you either accept them or you have to go. That’s your choice.” It sounds as if he is calling on Johnson to resign. “He should resign. Yes.”

Most of the time, Grieve isn’t this forthright. His parliamentary office, lined with paintings and rugs, has the restful air of chambers; if robes were hanging over a chair, they would not look out of place and Grieve speaks largely in the rather dry, considered register of legalese. His tone is appealingly understated, deploying terms such as “odd” and “strange” to denote disapproval, but the lawyerly style, although precise, is prone to drift into obfuscation, which can be frustrating.

Facebook Twitter Pinterest Giving the speech that left him branded a ‘traitor’. Photograph: AFP/Getty Images

I ask if it’s true, as was reported this week, that Grieve and like-minded colleagues would now support the prime minister’s proposed customs partnership, an infernally complicated arrangement designed to allow the UK to leave the customs union but maintain frictionless trade with Europe – which Brussels has called “magical thinking”.

“Well, what I see as absolutely essential is to try to ensure we can negotiate a deal with the EU that enables us to enjoy frictionless trade. In reality, we won’t get that without a high level of regulatory alignment.” But his Brexiter colleagues won’t accept that, will they? “The irony is that when I keep saying, ‘Well, what regulatory misalignment do you want?’, nobody can give an appropriate example.”

The whole problem with their position, according to Grieve, is that it’s “fundamentally ideological. It’s not a practical position. They think the liberty – the ideological purity – of complete liberation is more important than the risks in losing frictionless trade. I don’t subscribe to that because, as a lawyer, I’m so aware that we’re tied in by numerous international agreements of one form or another. The idea that by shedding the EU we recover this untrammelled national sovereignty is nonsense. The world doesn’t work that way. It never worked that way, even before we were members of the EU.”

Their objection to the European court of justice (ECJ) – “this hated supernational tribunal” – is, he adds, equally irrational. “I mean, the world is full of international tribunals arbitrating the interpretation of international agreements. I think we’re subject to about 800 of them, apart from the ECJ. But we don’t get het up about them – so why is it that we get het up about the ECJ?”

The economic and political consequences of indulging Brexiters’ irrational fantasies, says Grieve, will be dire. “If our country goes into relative economic decline compared with our EU partners, in five or 10 years’ time the electorate is not going to be thanking us for having delivered such an outcome. As a pragmatic party, it’s a very strange thing to do. It’s one thing to ask people for blood, toil, tears and sweat in wartime – but to go out and say to people: ‘We think you should give 20 years of blood, toil, tears and sweat and poor growth to achieve an ideological objective …’” He shakes his head in bewilderment. “Nobody in this country goes around saying: ‘I’m feeling very oppressed by the EU.’ Well, one or two people do, but they’re a bit odd. Ultimately, if they’re getting oppressed by the EU, they’re going to start to feel oppressed by something else and just switch to a new subject of oppression.”

Facebook Twitter Pinterest As attorney general in 2012. Photograph: Oli Scarff/Getty Images

Grieve’s clarity on the matter of Brexiters’ “emotional feelings about liberty” becomes less evident when it comes to specifying what he actually wants. I try to pin him down, but he keeps talking in generalities about guiding principles – as in, “an outcome that minimises the risks of leaving the EU as well as preserving some of the opportunities. That’s the broad aim.” But surely that’s everyone’s broad aim. “The single question for me is: do we want to sacrifice free trade with Europe for free trade elsewhere? It seems to me that logic dictates that free trade with Europe is more important to us. I’m afraid I think it’s very likely that you can’t have both, so that is the bottom line.” That means membership of the single market and customs union? “Yes, we could go as far as EEA [European Economic Area] plus.” His problem, of course, is that “there are some of my colleagues on the Conservative benches who disagree with me. They’ll take the other bottom line. And that is, I think, a fundamental disagreement. It’s not one I can easily avoid.”

As the intractable complexities of Brexit become clearer, it seems to me that a deal may in fact be impossible to achieve. “I don’t think it’s impossible,” he says. “It’s undoubtedly going to be very difficult. I’m not sure it’s completely insoluble, though, because I think there is an innate pragmatism in Britain.” Is he seeing much evidence of it at the moment? “No. Not at the moment. But are we seriously considering leaving the EU with no deal at all?” What odds would he give on that outcome? “I think the danger is there, but it’s very difficult to assess a percentage risk.”

When I ask if he agrees that no deal is better than a bad deal, he wrinkles his nose. “I never like expressing it that way. I don’t like these sorts of mantras. I mean, of course, suppose we ended up signing a mad deal with the EU that said we have to pay them £50bn a year in return for nothing, of course I’d be saying this deal isn’t worth the paper it’s written on and we might as well have no deal. But I think that the words ‘no deal is better than a bad deal’ are being used to try to coerce people into accepting the idea of no deal and to ignore just how damaging no deal would be. So I just don’t get dragged there.”

The fashionable default position became: if anything goes wrong, criticise the EU

After some more probing, Grieve finally states his preferred solution to the impasse: a second referendum. “This is a personal opinion and I have to stress that. But I think there is an argument, at the end of all this, for asking the public: is this what you really want? Because it seems to me that the difficulty with the 2016 referendum was that we asked an abstract question and people gave an abstract answer.” He pauses, looking uneasy. “I want to stress I’m not working to precipitate it. But it does at least offer a way through the problem that we’re facing, which is that we could end up in a situation where we can’t agree on an outcome at all. And I’m not sure where we would go from there.”

I ask who he blames for the mess we are in. “When the vote for a referendum was passed, I probably have to admit I didn’t give it enough thought. It was sprung on us by David Cameron; it was not exactly consulted upon, even within government. Maybe he never imagined he was going to win the 2015 election – but he did and he then had to implement it.” So he blames his former boss? “He is responsible for the decision to hold the referendum.” But Grieve also blames “a failure of many of us – and I plead guilty to this. The fashionable default position became: if anything goes wrong, criticise the EU. I don’t think enough of us – and I do regret it – went out and articulated the alternative until, arguably, it was too late. I think on that basis we should be blaming ourselves for what happened.”

Despite all the turmoil and upset, he says the Tory party still feels like his tribe. Talk of a new centrist party holds no interest for Grieve? “No, I’m a Conservative.” He has known May since they were at Oxford together and says they have always been “friendly”, so I ask if she would think of him now as helpful or unhelpful. “I don’t know. She may think of me sometimes as unhelpful. But do I overall think I have been helpful to her? I hope I have. But I accept that in the course of that, there has been some friction and she may think ill of me. But she’s too polite to have said it directly, and I haven’t asked.”

Would the party still feel like his tribe if Jacob Rees-Mogg were leading it? “Hmm. It would be very strange because ultimately all parties are coalitions of interest, and that tends to mean you coalesce towards a central figure who can unite the party. It’s difficult to see Jacob in that context.”

But, he adds quickly: “We get along extremely pleasantly.” In fact, the only thing that seems to make Grieve really cross is the portrait painted by the press of him in disgrace within his party. There is nothing remotely dry or understated about his tone as he says, with some feeling: “Newspapers love to think that divisions within political parties manifest themselves in acrimonious moments. It’s not true. That story about my colleagues refusing to sit with me at breakfast was completely fabricated. Total and complete nonsense. It just didn’t happen.”