Next week it will be my first day in a new job, after almost 20 years in public life in Europe and Britain. It could not be more of a contrast. Instead of the gothic splendour of Westminster, I will be surrounded by the gleaming glass and steel of Silicon Valley. Instead of the clout of the state in Whitehall, I will now experience the dynamism of the private sector in Palo Alto. Instead of the impact of Brexit, I will now be focused on the impact of tech.

A job at Facebook is – literally and metaphorically – a long way from SW1. I have mixed feelings about leaving the UK’s public debate about the future of our country’s relations with the rest of Europe. Since the referendum, I have devoted my energies to making the case – often in uncompromising terms unavailable to those still in office – for a rethink of what I believe may turn out to be the greatest act of self-harm committed by a mature democracy. It is being made against the interests and wishes of those who have to bear the heaviest consequences: the young.

The Brexit drama will soon move to – and possibly culminate in – the place where it arguably belonged all along, parliament. I am no longer a member of parliament and have never had any desire to sit in an unreformed House of Lords. So once I had decided to accept a new role at Facebook, there was little merit in delaying my move any longer.

It is, of course, deeply distressing to see the UK become marginalised in world affairs, distracted by the grinding Brexit psychodrama at Westminster. If the prime minister gets her way, we will soon take a step without precedent in the modern era. Britain will, voluntarily, enter into a protracted negotiation about our future relations with the EU, having thrown away the remaining leverage we possess as current members. It is unthinkable that statesmen and stateswomen of the past would have placed our country in such a weak position.

So I sincerely hope, when they finally come to pass judgment, MPs will put the long-term fortunes of our nation before the short-term prospects of their party. The case for a people’s vote on any final deal is overwhelming. But I will no longer seek to play a public role in that debate myself.

Profile Nick Clegg's political highs and lows Show Nick Clegg: political highs and lows Even though Nick Clegg spent five years as deputy prime minister, his probable political highlight came about a month before he took the post, in the unlikely arena of the pre-election party leaders’ debate of April 2016. Pitted against David Cameron and Gordon Brown, the then 43-year-old Liberal Democrat leader showed himself telegenic, engaging and politely baffled at his opponents’ tribal enmity. Not only did the audience polling declare him the winner, the debate also sparked the catchphrase: “I agree with Nick”. It’s perhaps unfair to say things only went downhill from then. After the election, the Lib Dems had 57 MPs – enough to gain a share in government with Cameron’s Conservatives and get Clegg an office adjoining Downing Street. But, inevitably, with a minority share in power came compromises, most notably ditching the party’s pre-election promises on tuition fees, as well as a tacit approval of the government’s austerity programme. Clegg was never as politically popular again. In the 2015 election, the Lib Dems were decimated to a rump of eight and out of power. Two years later, Clegg lost his Sheffield Hallam seat to Jared O’Mara of Labour, a party which had never held the constituency since it was created in the 1860s. It was a rapid fall for a man who ended up spending just 12 years in the Commons, becoming Lib Dem leader little more than two years after becoming an MP, following his work at the European commission and five years as an MEP. Since leaving parliament, Clegg has doubled up as a Lib Dem grandee and a leading voice opposing Brexit, which has seen him lead unofficial and controversial diplomatic efforts to try to stop the process. His political fortunes were placed into some context after it emerged last year that Clegg’s eldest son, Antonio, then 15, had been diagnosed with the cancer Hodgkin lymphoma. He responded successfully to treatment. Photograph: Nils Jorgensen/Rex Features

Instead, as vice-president of global affairs and communications at Facebook, I hope to help it navigate the numerous challenges it faces, in common with other leading tech companies, as the data-driven technological revolution continues to affect every aspect of our everyday lives: the control we have over our personal data; the integrity of our democratic process; the power and concerns about artificial intelligence; the tension between the global internet and national jurisdictions; the balance between free speech and prohibited content; and the wellbeing of our children.

I do not arrive in Silicon Valley with a monopoly of wisdom on these crucial questions, but I have been impressed in my numerous conversations with Mark Zuckerberg and Sheryl Sandberg in recent months by the seriousness with which they recognise the profound responsibilities that Facebook has – not only to its vast number of users but to society at large.

As with most big decisions I took in politics, I expect my decision to move to Silicon Valley will be shouted down by people from the left and the right. As concerns about the impact of technology grow, the left has tended to condemn “big tech” as representative of everything that is wrong in an unleashed market economy. The right regards it as a threat to traditional family values. Some criticisms – especially the tech world’s libertarian indifference to the wider responsibility of governments and regulators in its early days – are fair. Others are wide of the mark.

But, as a general rule, I believe in engaging and grappling with difficult questions. If Silicon Valley were to disappear overnight, the technology invented in it would not. Global tech leadership would instantly transfer from the west coast of the US to China. We cannot wish away technological progress. Nor can we simply bemoan the social, ethical and political dilemmas raised by the tech revolution. Every major technological breakthrough in human history has created challenges as well as opportunities. We need to find solutions, not succumb to the easy emotions and polemic of a “tech-lash”.

I remain a stubborn optimist about the progressive potential to society of technological innovation. It can transform how we work, play and build relationships. It can help to protect our environment and keep our streets safe. It will fundamentally change how we teach our children at school and at home. It is transforming healthcare and transport. If the tech industry can work sensibly with governments, regulators, parliaments and civic society around the world, I believe we can enhance the benefits of technology while diminishing the often unintended downsides.

The worlds of politics and tech too often speak past each other. The temptation for one side to condemn or ignore the other is great. But this standoff will, in the end, leave everyone worse off. It is time to build bridges between politics and tech so that tech can become the servant of progress and optimism, not a source of fear and suspicion.

•Nick Clegg is a former UK deputy prime minister and former leader of the Liberal Democrats