We’ve all been there. You want a short cut – to the bus stop, office or corner shop – but there’s no designated path. Others before you have already flattened the grass, or cut a line through a hedge. Why not, you think.

So goes the logic of “desire paths” – described by Robert Macfarlane as “paths & tracks made over time by the wishes & feet of walkers, especially those paths that run contrary to design or planning”; he calls them “free-will ways”. The New Yorker offers other names: “cow paths, pirate paths, social trails, kemonomichi (beast trails), chemins de l’âne (donkey paths), and Olifantenpad (elephant trails)”. JM Barrie described them as “Paths that have Made Themselves”.

Reddit has desire path threads, tens of thousands of people strong, delighting in the more mysterious or illogical-seeming of them. They can form anywhere from apparently forgotten corners of cities to the grounds of national governments, as has happened around the National Congress of Brazil; some are so well established that they are visible on Google Maps.

Desire paths are evident in the grounds surrounding the National Congress of Brazil, Brasília. Photograph: Alamy

Desire paths have been described as illustrating “the tension between the native and the built environment and our relationship to them”. Because they often form in areas where there are no pavements, they can be seen to “indicate [the] yearning” of those wishing to walk, a way for “city dwellers to ‘write back’ to city planners, giving feedback with their feet”.

Desire lines present evidence about movement, which is important Riccardo Marini, urban planner

But as well as revealing the path of least resistance, they can also reveal where people refuse to tread. If you’ve been walking the same route for years, an itchy-footed urge to go off-piste, even just a few metres, is probably something you’ll identify with. It’s this idea that led one academic journal to describe them as a record of “civil disobedience”.

Rather than dismiss or even chastise the naughty pedestrian by placing fences or railings to block off “illicit” wanderings, some planners work to incorporate them into urban environments. This chimes with the thinking of Jane Jacobs, an advocate of configuring cities around desire lines, who said: “There is no logic that can be superimposed on the city; people make it, and it is to them … that we must fit our plans.”

Unofficial pathways, such as this one through Highbury Fields, London, can be seen to “indicate [the] yearning” of those wishing to walk. Photograph: Alamy

Architect and urban planner Riccardo Marini takes desire lines very seriously. He has seen plenty of evidence of what happens when you don’t. “Somebody has spent a fortune putting in granite steps with a piece of landscape next to it, and people have gone up the slope because their brain tells them that’s the quickest way to do it, even if they get muddy,” he says. “Desire lines present evidence about movement, which is important.”

Marini has spent decades working with the Danish urban planner Jan Gehl, famous for putting people and the way they interact with a space at the core of his work. Desire paths are about “listening to a place” – and listen Marini does: he has even drawn up bubblegum and cigarette butt maps of Regent Street in London to help inform where to place benches.

Michigan State University didn’t put in pavements when new buildings were created. Instead, it waited for students to create their own paths. Photograph: USGS © 2011 Microsoft Corporation

Using this kind of evidence to inform planning is not entirely new. Many college campuses have waited to see where their students and teachers choose to tread before adding paving. Michigan State University was one such institution, and its campus is now a pleasing etch-a-sketch board when seen from above.

Desire lines tell us about the endless human desire to have choice Professor Andrew Furman

In the 90s, the Dutch architect Rem Koolhaas spearheaded one of the best-known examples, letting the footfall of students inform his plan for the Illinois Institute of Technology. Hospitals, too, have been known to respond to organically produced footpaths: at the US National Institutes of Health, the footpaths that developed in the 1960s, 70s and early 80s went on to be paved.

There is historical precedent, too. According to some urban planning experts, Broadway was New York City’s earliest desire line, following as it does the Native American-made Wickquasgeck Path, which is thought to have been the shortest route between pre-colonial settlements in Manhattan that avoided swamps and hills. Broadway is the only remaining one path, according to Marini, that “wasn’t wiped out by the European grid being overlaid on it”.

Some believe that Broadway was New York City’s earliest desire line. Photograph: Mary Altaffer/AP

As such, desire lines can reveal the roots of cities. “When they do not fade, or until they fade,” writes Erika Luckert in a paper mapping the desire lines of the Canadian city of Edmonton, “[they] serve not only as paths, but also as historical records – making past patterns of footsteps apparent, and allowing the past to directly inform the present”. Marini points to Edinburgh’s Cowgate, which he says was originally “a desire line expressed by a shepherd or a herder and their animals [travelling] from the farm to the market”.

For Andrew Furman, a professor in interior design and architecture at Ryerson University in Toronto who has spent years looking at desire lines, they tell us something about “the endless human desire to have choice. The importance of not having someone prescribe your path”. In a heavily constructed city, there are “rules as to how public and public-private spaces are used”, he says. Desire paths are about “not following the script” – he calls it a “resistance”. “An individual can really write their own story. It’s something really powerful if you do have that agency to move.”

He points me to a passage in Rebecca Solnit’s book about the history of walking, Wanderlust: “Walking is a mode of making the world as well as being in it.”

Follow Guardian Cities on Twitter, Facebook and Instagram to join the discussion, and explore our archive here