London’s mayoral contest is slowly limbering to life. A fierce battle is breaking out on the Labour side with Tessa Jowell, Sadiq Khan and David Lammy all in with a chance to pull through the primaries. For the Tories, Hammersmith’s former leader Stephen Greenhalgh and MP Zac Goldsmith are in the frame. Sol Campbell has declared his intention to run for the Tories and rumours continue to circulate about other celebrities – Karren Brady, Nick Ferrari or even Jeremy Paxman – throwing their hat in the ring on the Tory side.

That’s as it should be. The leadership of a great city should be fought over as if it mattered. So what’s missing? At the moment what’s missing is what makes London great – ideas. Just as the general election was bereft of exciting ideas, so is London’s contest.

This is surprising. By any standards this is an era when cities are setting the pace, and particularly world cities. As national governments groan under the strains of austerity, cities are on a roll. A few years ago they passed the point when they contained half the world’s population. Many are moving ever further ahead of the rest of their countries and a few have become magnets – with the likes of London, Paris, Los Angeles, sucking in talent, migrants and money to an unprecedented degree.

This dynamism isn’t without its problems. Some are external. The rest of Britain is jealous of London’s magnetism and of its privileged access to the lion’s share of public capital, from Canary Wharf in the 1980s to Crossrail today. Some of the problems are internal – a few streets have become ghost towns, owned by the “non-dom” super-rich but only rarely visited, while other parts of London are still among the poorest in the country.

For prospective mayors, the central challenge of leadership flows from this dual character. On the one hand, they need to generate wealth and jobs, and keep on attracting the energetic and ambitious. On the other, they need smart ways to share the opportunities and benefits beyond a small elite.

Achieving that double success is far from easy. Los Angeles combines vast wealth with entrenched poverty and crime. Paris is divided between its glossy centre and its crumbling and troubled banlieues. New York swung left in 2013 because too little was being done to bridge the gap between the haves and have-nots. But everywhere cities are once again where ambitious politicians want to make their names, where companies want to launch and where universities seem to thrive best.

Twenty years ago, things looked very different. Then, many of the most famous cities were struggling with deindustrialisation and mass unemployment. Crime was soaring in New York, London and Paris, and the affluent were keen to move out. A mid-1990s piece of futurology from Arthur Andersen was typical in predicting that by 2015 the inner parts of cities like London would be populated only by the poor and unemployable.

Small towns beyond the greenbelts looked more likely to grow in the age of digital technologies when no one had any need to be physically close to the people they worked with. Given the choice, attractive natural environments seemed preferable to smoke and congestion, and many hundreds of science parks were being built well away from city centres, just as in the 1960s most of the new universities were built far from the chaos of city life. Now, all of that has turned around and cities look uniquely well placed to forge the combinations and connections that are driving growth, whether through the Internet of Things, big data, social innovation or genomics, all of which are flowering first in cities.

It’s not surprising that politics has shifted in tandem. Over the past decades the rollcall of dynamic city leaders has become much more impressive. It’s probably not a coincidence that Boris Johnson is Britain’s most popular politician, even if people sometimes struggle to remember what he’s actually achieved. In Berlin the gay, socialist mayor Klaus Wowereit turned his city into a cosmopolitan powerhouse. Another gay leader, Paris’s Bertrand Delanoë, introduced rentable bikes before London and then went further with rentable cars. Barcelona has had many dynamic mayors, the last one reinventing his city as a hub for mobile technologies and maker spaces, presented as a 21st-century version of the public library and a tool for bringing manufacturing back into the city centre. A succession of mayors in Copenhagen have made it a green icon, while Reykjavik’s comic/clown mayor pioneered the most advanced citizen decision tools in the world.

Sheila Dixit in Delhi didn’t only provide smart airports, but also delivered good public health and jobs for the poor. Jakarta’s Jokowi so impressed that he was catapulted into the presidency. Fabulously creative mayors in Latin America, like Antanas Mockus and Enrique Peñalosa in Bogotá, redefined what the role could mean, with spectacle, civic behaviour and the arts, using mime artists to change the behaviour of bad drivers or asking the men to stay at home so that their wives could have a night on the town.

Sergio Fajardo transformed Medellín from its association with cocaine barons and industrial-scale murder to be ranked by Citigroup as the world’s most innovative city in 2013. Seoul has seen one of the most remarkable transformations of city politics under mayor Park Won-soon, a civil society campaigner who ran against the party machines, won on a wave of social media enthusiasm and is transforming his city into a leading example of the sharing economy, mixing feverish citizen engagement with the fastest broadband infrastructure in the world.

The most visible city leader of all has been Michael Bloomberg, the billionaire businessman. His 12 years in charge of New York included its continuing renaissance, with strong economic growth and falling crime. His rule was marked by attention to detail, a love of data and a passion for innovation. That translated into aggressive policies to turn around the city’s often lamentably bad schools and iconic projects such as the High Line park (a renovated elevated railway). Perhaps having run a media company he was more aware than most that leading is about more than issuing press releases.

So where is Britain in all of this? Some cities are undoubtedly making a mark. London has had two characterful mayors and, for the first time in a century, a great city’s leader – Boris Johnson – could be on track to be prime minister. But London’s mayoral election to succeed Johnson is one of many signs that Britain’s cities are not playing a full part in this urban renaissance. So far there have been relatively few signs of original thinking from the politicians lining up to take over.

This is surprising because London is doing extraordinarily well on many counts, especially as a cauldron of ideas. It has more world-class universities than any other city on Earth, and more air travel. Its booming digital start-up scene is being matched by a nascent life sciences economy, soon to be helped by the Francis Crick Institute Centre next to St Pancras and a new Alan Turing Institute for Data Science. Huge amounts of US investment are pouring into London-based tech firms – more than £600m last year alone. It houses the world’s most developed social investment industry, which barely existed a decade ago, and fields like food and fashion have probably never been as creative.

But apart from arguments about mansion taxes and a few thoughtful speeches on social policy from David Lammy, you’d be forgiven for not having noticed any of this in the speeches by would-be candidates. London’s next mayor could be pioneering the next generation of sharing economy ideas (as Seoul has done), opening up part of the budget to public ideas (like Paris), using big data to target public services in creative ways (as New York has done with the fire service), or mixing digital technologies and community support to overhaul care for the elderly (as Barcelona is doing). Instead, for now, the debate feels parochial.

Elsewhere in England there are some green shoots. Of all Britain’s city bosses, Sir Richard Leese in Manchester has probably been the most adept at the craft of leadership – and his years of lobbying paid off in 2015 when he was given control over much of the health budget. Bristol has become a lot more fun under the leadership of its red-trousered mayor George Ferguson, and some cities have strikingly talented leaders, like Newcastle’s Nick Forbes. But the big picture is of weak leadership, compounded by insufficient powers; the major parties still do little to encourage their best and brightest to prove their mettle in cities, encouraging them to flock to Westminster.

More of the real leadership has come from below. Cities are places where the huge shift towards social innovation is most visible. The best ideas aren’t being dreamed up by policymakers in capital cities, but are emerging more organically from the grassroots. These are the kinds of projects celebrated in “Britain’s new radicals”, inventing new approaches to healthcare, education and the arts. It’s a world largely invisible in the platforms of major parties, but is full of energy and excitement – some of which has been tapped by the newer political parties, like Spain’s Podemos, which just won power in Madrid and Barcelona.

For the most imaginative cities, the challenge is to link the top down and bottom up in creative ways. Park put a huge ear outside his city hall as a symbol of two-way government. In Paris, mayor Anne Hidalgo has promised that 5% of the city budget will be opened up to participatory budgeting, while Helsinki is using open APIs (application program interfaces) to let the public engage with decisions at every stage.

These are much more attractive redefinitions of what it means to be a smart city than the ideas with which the label is usually associated, which range from sensible schemes to manage energy or traffic more efficiently, to technocratic utopias that often seem almost like parodies of the worst habits of city planning, in which glossy blueprints for cities packed with sensors are designed and built with no inputs from the people who’ll live in them. Songdo in Korea was meant to be the symbol of future cities, packed with smart hardware, but has lacked soul and failed to attract new residents. Masdar in Abu Dhabi was another very costly failure – which, again, didn’t grasp that what makes cities buzz is their openness and messiness, their chaos as well as their order. China’s 192 smart city projects are now trying to avoid repeating these mistakes, involving citizens in things like measuring air pollution or coordinating car pools. For them, the great challenge is to find routes to inclusive growth that spread opportunities far beyond the central business districts and the creative clusters.

This is the heart of the challenge for any great city: how to link the top and bottom, the rich and the poor, the fast and the slow. Not long ago, everywhere wanted to copy Silicon Valley, and it remains true that every city would love to give birth to the next Facebook or Twitter. But they also now recognise that the Silicon Valley model achieved very little “trickle-down” of wealth. Most Americans now earn less than they did a few decades ago, even as a tiny proportion have become immensely rich. And so the search is on for better models of economic growth that overhaul schooling to better fit where the jobs are coming from, and get big firms to open their doors to the poor and marginalised.

London’s mayor will have to be plausible both for the people in the City, and in “tech city”, who have never had it so good, and for others who watch on from the sidelines, either out of work altogether or trapped in insecure low paid work that’s going nowhere. A few ideas are floating around about affordable housing. But we risk ending up with a campaign that confirms the fear that politics has fallen behind society, stuck in a rut of predictable positions and low expectations.

Devolution was meant to be about unleashing ambitions and energies – and in Scotland that is exactly what it’s done. But for London and for England’s other cities, we’re still waiting.

Geoff Mulgan is chief executive of the National Endowment for Science Technology and the Arts

Berlin

Mayor Klaus Wowereit

In office 2001-14

Politics Social Democratic party

Greatest achievement reinventing Berlin as a cool, multicultural city, attracting tourists, migrants and investors.

Biggest challenge popularity levels plummeted after endless delays in building the new Berlin Brandenburg airport.

Paris

Mayor Bertrand Delanöe

In office 2001-14

Politics Socialist party

Greatest achievement introduction of rentable bikes scheme (before London) and rentable cars which transformed the city centre. Succeeded by one of his deputies, Anne Hidalgo.

Biggest challenge Paris’s continued social divisions with poor, predominantly Muslim suburbs not benefiting from prosperity of the inner city.

Seoul

Mayor Park Won-soon

In office 2011-present

Politics Democratic United party (2012-14), New Politics Alliance for Democracy (2014-)

Greatest achievement nicknamed the ‘listening mayor’, pioneering citizen engagement, the sharing economy and new welfare policies, helped by the fastest broadband infrastructure in the world.

Biggest challenge coping with hostility from national government and big business.

NEW YORK





Mayor Michael Bloomberg

In office 2002-13

Politics Republican (2001-07), Independent (2007-13)

Greatest achievement achieving strong economic growth and lower crime, as well as projects such as the High Line park.

Biggest challenge high levels of inequality which contributed to the victory of Democrat Bill de Blasio as Bloomberg’s successor.

MEDELLIN

Mayor Sergio Fajardo

In office 2004-07, now running the wider regional government

Politics Independent as mayor, now Partido Verde (Green party)

Greatest achievement turning around the city from association with cocaine cartels and violence to be named ‘Most Innovative City of the Year’ in 2013.

Biggest challenge sustaining the turnaround while peace process remains unresolved.

• This article was corrected on 18 June 2015 to delete an erroneous reference to Accenture.