Who would now dare to find hope in among the Brexit nightmare? Halloween is just a month away, and there is still no sign of any breakthrough in negotiations. The Conservative party is dealing in dangerous rightwing populism, a development that looks all the more abject because its new leader has taken this turn solely in the interests of his own career.

The idea that an election could end the mess is undermined by wildly uneven polls, many of which have pointed to another hung parliament. And after his behaviour in the Commons last week and lack of contrition since, the prospect of Boris Johnson somehow agreeing some kind of compromise deal and getting it through parliament with the help of Labour MPs now looks like a fantasy. If you want a vision of the future, imagine an upper-class voice yelling, “Get Brexit done” for ever, as the paralysis and failure deepens.

Last week I was at the Labour party conference, where Brexit was a reliable source of angst and division, and the often rancorous atmosphere regularly suggested yet more impending doom. But I also started to feel pangs of optimism, and a renewed sense of something too often overlooked: the fact that our national meltdown is full of political complexities; and that amid the mess, there are also the stirrings of a politics that might eventually answer the 21st century’s challenges.

The list of policies announced and adopted in Brighton cohered around a central spine of ideas that were much bolder than the party’s 2017 manifesto. There was a Green New Deal with a 2030 zero-carbon target; John McDonnell’s proposals for a four-day working week and set of universal basic services and plans for a state-owned pharmaceuticals producer, the abolition of private education and a new national care service.

Whether these will make it into a manifesto is an interesting question, but when proceedings were at their most exciting there was a sense of an embryonic leftwing politics that might at last speak to a future oriented around three key things: the roots of inequality in society and the economy; the upturning of life and work by technology; and the need for a green approach to almost everything.

In part, these were the result of hard work and creativity among a network of thinkers and policy wonks, and the kind of activists who are less interested in the bureaucratic grind of conference than the annual The World Transformed festival, which was once again full of energy and optimism. Some of the party’s new ideas reflect the radical environmentalism of Extinction Rebellion and the school strikes. But they also highlight the way the 2016 referendum has changed our politics in ways only just being understood.

It is easy to forget that the vote for Brexit was initially taken as being about much more than Europe, on both left and right. Jeremy Corbyn explained what had happened in terms of “economic dislocation” and people being “betrayed and marginalised by successive governments”, themes he has repeated ever since. Not long after, Theresa May said that support for Brexit expressed many people’s belief “that the world works well for a privileged few but not for them” and that the biggest sacrifices after the crash had been made by “ordinary working-class families” – a view now being echoed in Johnson’s cynical trumpeting of the wonders of public spending.

‘Some of Labour’s new ideas reflect the radical environmentalism of Extinction Rebellion and the school strikes.’ Students take part in a climate crisis rally in Rome. Photograph: Massimo Percossi/EPA

In addition, something even more game-changing has happened. Since 2016 Brexit has been transformed from the modest and endlessly beneficial readjustment promised by the pro-leave campaigners – £350m extra a week for the NHS, and all that – to a revolutionary project in which national harm might have to be accepted in pursuit of sovereignty. This change does not seem to have significantly affected levels of support for leaving the EU. To many people – me included – this might appear irrational. But it also calls time on the pre-referendum idea that politics can never be about big, radical ideas. If politicians can defy every ounce of political and economic common sense and insist we leave the EU, why should there be any limit on the kind of red-green ideas that we saw last week?

This is not to say that success for the left is imminent. Standing in the way, obviously, is Brexit itself. Labour has arrived at a messy compromise that will send the party into an election advocating another public vote, while declining to say which side it will back. Whatever the logic of holding its base of support together and waiting for the outcome of new negotiations, Brexit-weary voters will feel wearier still at the prospect of having to vote again; and devout leavers, in my experience, already see the drive for another referendum as tantamount to backing remain.

Even if those problems can be set aside, if the party really wants to embrace a programme even half as thoroughgoing as the one hinted at in Brighton, it will need to ally with people, institutions and parties well outside the tent – something regularly thrown into doubt by the fact that to anyone looking in from the outside, Labour looks like a party struggling to get on with itself.

As evidenced by yet another Corbyn speech that lacked a stirring, broad-sweep account of how Britain got into this mess and the national attributes that might get us out, the left’s eternal suspicion of anything that smells of patriotism means that its visions are too often expressed in a dry, mechanistic vocabulary that will never compete with the crass, emotional rhetoric we will hear from the Tories at their conference this week.

The face Labour presents to the country is still sullied by a shrill self-righteousness, the weird stuff of leader-worship, and the sense that many energetic, outward-facing, essentially pro-European members are now in awkward alliance with some people who behave, procedurally and politically, as if it were 1974.

But such is British politics. The big parties usually move in long and uncertain cycles, the old tends to get in the way of the new, and changes in thinking can take years to cohere. In Labour’s case, the long build-ups to 1945, 1964 and 1997 are all cases in point. For all Corbynism’s talk of a radical break from the party’s past, these tendencies are still ingrained – and in times like these, the party’s delay in getting its act together is maddening.

But one of the ironies of our time is that Brexit has blown the lid off conventional notions of political possibility, allowing the left’s grassroots to start offering grounds for optimism. In the midst of all-enveloping darkness, that might provide just enough flickering light to keep us going.

• John Harris is a Guardian columnist