The Tories have parked their tanks on our lawn, and they intend to take the house. This week they claimed they were ripping up Treasury spending rules to put more investment into former Labour heartlands. Their self-interest and self-confidence are painful to see, but there is no room for pain. To defeat this and raise the bar for the people we have always fought for will take a Labour party that can come out fighting, sustain that fight, and flip the old rules on their head.

When we set up the Centre for Towns three years ago, we had felt the tremors under our feet in those Labour heartlands. After four decades of underinvestment, there was a drumbeat of concern for the lost bus networks, decaying high streets, and young people who had to leave home to find good work. A political system that shrugged its shoulders when people spilled their hearts out was already in serious trouble.

The Tories have won the right to speak for those areas, but there are already signs they haven’t got a clue what they are doing or any humility about the responsibility they now have. The new spending rules simply give greater discretion to decision makers in central London. They do nothing to empower the people who know best what problems they have, and more importantly how they would solve them, given the chance.

Boris Johnson has all the hallmarks of a Prime Minister who does not get it. Nobody wants handouts. He is deeply uninterested in the root and branch changes we have spent years exploring – real and long-term investment in transport, digital and skills, in the right places and done in the right way; devolution of power to the right levels giving decision-making to people who can see the potential in their areas, not just the problems; greater accountability of those who seek the trust of the people to govern. These are the details that will make all the difference.

His first act after the election, using his new mandate to attack child refugees, showed his ignorance of communities that have fought off the far right for decades. His arrogance, as much as his intolerance, will find little support in those former Labour heartlands. This is ripe territory for Labour, to win back trust, and to win again.

We can be the party that has the confidence to disperse decisions and power much further, wider and deeper than a small group of people with similar experiences in central London. We could be the party that really hears what is happening across different parts of the country, from Bexley to Livingstone and Wrexham to Birmingham Northfield.

To do this, we have to change. Welsh and Scottish colleagues tell a long and powerful story of being side-lined and disrespected by the national party. If Labour wants to win consent to govern for the whole nation, we have to walk the walk. As a first step, Labour HQ should move out of central London, our party conferences must move out of the major cities, and we must empower good people at the local, regional and national level to make spending decisions and decide on the strategies that will anchor us in our communities again.

This means no time for factions – full stop. Factions sweep away talent. I saw how it cost us a broader range of views and led to some catastrophic decisions during the New Labour era. We cannot afford it now. A defeat on this scale demands a level of humility and a recognition that infighting kills us. We will not convince the public we can build a more equal, compassionate society when we are at each other’s throats. Our breadth is our strength and without it we have no claim to be a national party.

Now is not the time for licking wounds or for one last heave. We are in a new world, which demands new thinking and new rules. No more false binaries. No more divisions. No more jockeying for position. We need the widest range of energy and talent to defeat a newly invigorated Tory party and it can be done. We go forward together or we do not go forward at all.