In the dead of night, in spite of the electric lights, London seems an alien city, especially if you are walking through it alone.

In the more sequestered streets – once the pubs are closed, and at a distance from the 24-hour convenience stores – the sodium gleam of the street lamps, or the flickering striplight from a sleepy minicab stand, offers little consolation. There are alleys and street corners and shop entrances where the darkness appears to collect in a solid mass. There are secluded squares where, to take a haunting line from a poem by Shelley, night makes “a weird sound of its own stillness”. There are buildings, monuments and statues that, at a distance, and in the absence of people, pulsate mysteriously in the sepulchral light. There are foxes that slope and trot across the road as you interrupt their attempts to pillage scraps from upended bins.

And, from time to time, there are the faintly sinister silhouettes of other solitary individuals – as threatened by your presence, no doubt, as you are by theirs. “However efficiently artificial light annihilates the difference between night and day,” Al Alvarez has remarked, “it never wholly eliminates the primitive suspicion that night people are up to no good.”

It is easy to feel disoriented in the city at the dead of night, especially if you are tired from roaming its distances, dreamily or desperately somnambulant. For in the darkness, above all perhaps in familiar or routine places, everything acquires a subtly different form or volume.

Ford Madox Ford, in The Soul of London, lamented a century ago that, “little by little, the Londoner comes to forget that his London is built upon real earth: he forgets that under the pavements there are hills, forgotten water courses, springs, and marshlands”. It is not quite the same at night. At 2am, in the empty streets, no longer fighting against the traffic of cars and commuters, the solitary pedestrian’s feet begin to recall the “real earth”. In the abstracted, monochromatic conditions of the nighttime, it becomes more apparent that a sloping road curves over the sleeping form of a hill and tracks the course of an underground stream. The city is at its most earthly and unearthly at night.

Piccadilly Circus with the lights off, 1960. Photograph: Associated Newspapers/Rex

A prehistoric landscape comes to seem more palpable beneath the pavements of the city. And in this half-familiar environment it is difficult to eliminate entirely the archaic conviction that, as for our ancestors, the night itself remains ominous, threatening. Residues of a primal fear of the dark begin to trouble you.

The nighttime city is another city. Rhapsodising the public parks of the French metropolis in Paris Peasant (1926), the surrealist Louis Aragon commented that “night gives these absurd places a sense of not knowing their own identity”. It is a point that applies to all aspects of the city’s architecture or terrain. The nighttime self, moreover, is another self. It too is less certain of its own identity.

Who walks alone in the streets at night? The sad, the mad, the bad. The lost, the lonely. The sleepless, the homeless. All the city’s internal exiles. The night has always been the time for daylight’s dispossessed – the dissident, the different. Walking alone at night in the city by both men and women has, since time immemorial, been interpreted as a sign of moral, social or spiritual dereliction.

Solitary women, because of a long history of discrimination and patriarchal oppression, have been especially susceptible to this sort of suspicion. If women appear on the streets of the city at night alone they are commonly portrayed as either predators, in the form of prostitutes, or predatees – the potential victims of sexual assault. In both cases, they are denied a right to the city at night.

The historian Joachim Schlör has pointed out that, in terms of the freedom to inhabit the nocturnal city, “women’s needs and wishes are not fundamentally different from men’s”, since for both it is a case of entering it and circulating inside it freely and independently – “through the whole city, during the whole night, and not just in certain spatial and temporal reserves”. But he has rightly insisted that, historically, “men’s freedom of movement has [had] a real restrictive effect on that of women”.

Old Compton Street, Soho, 1936. Photograph: General Photographic Agency/Getty Images

If solitary men on the streets at night have exercised a right to the city denied to solitary women, then they too have often been identified or represented as pariahs. People who walk about at night with no obvious reason to do so, whether male or female, have attracted suspicion, opprobrium and legal recrimination from patriarchs, politicians, priests and others in authority, including the police, for thousands of years. In 1285, Edward I introduced a specific “nightwalker statute” in order to police the movement of plebeian people – especially migrants, vagrants and prostitutes – after the 9pm curfew. But long after this statute became impossible to implement, because of the rise of “nightlife”, the authorities continued to construe nightwalking as deviant.

Today, more than ever, solitary walking at night in the streets of the city does not necessarily mean deviant movement. It may well be perfectly legitimate, purposeful. Contemporary capitalist society requires what Jonathan Crary has identified as the despoliation of sleep in the interests of maximising the individual’s potential – both as a producer and a consumer – for generating profit. The political economy of the night, in this dispensation, means that plenty of people have to commute after dark, sometimes on foot, sometimes across considerable distances.

This is the daily, or nightly, reality of post-circadian capitalism, as it might be called. For the city’s army of nocturnal workers, many of whom are recent immigrants forced to perform the least popular forms of labour, travelling at night is in effect travailing at night. Sex workers and the police (or its precursors) have, for their part, always had to patrol pavements at night for professional reasons. So have street-cleaners and others employed to collect and dispose of the city’s waste.

Not all walking at night, then, is nightwalking. But most forms of solitary walking at night are nonetheless tainted, sometimes faintly with dubious moral or social associations. Indeed, even apparently purposeful walking in the city at night is not exempt from the assumption that it is suspicious. To be alone in the streets, even if one walks rapidly, determinedly, is to invite the impression that one is on the run, either from oneself or from another.

A view down Dean Street, Soho, London, August 1955. Photograph: BIPS/Getty Images

The late Chilean novelist Roberto Bolaño alludes to these conditions of being in the night – those of the haunted and the hunted – in a reference to the life, or half-life, of the city “at an hour when the only people out walking [are] two opposite types: those running out of time and those with time to burn”. In fact, these types are not really opposite: many people who are running out of time or resources, paradoxically, have time to burn. This contradictory state, of idling and hastening at once, is a comparatively common experience. It is even more potent on the streets at night.

To use a Dickensian phrase, nightwalking is a matter of “going astray” in the streets of the city after dark. Dickens is the great heroic and neurotic nightwalker of the 19th century. In 1860, in the guise of the Uncommercial Traveller, he made a crucial distinction between two kinds of walking: one that is “straight on end to a definite goal at a round pace”; another that is “objectless, loitering, and purely vagabond”. If the point of the first kind of walking is to travel from one point to another, the point of the second is that there is no point at all. Its purpose is its purposelessness. Nightwalking, according to this logic, is pointless. It is uncommercial.

In an economy in which time, including nighttime, is money, wandering the streets after dark – when most people are sleeping in order to prepare themselves for the next day’s labour – is in symbolic terms subversive. In the aberrant and deviant form celebrated by Dickens in the 19th century, and surreptitiously practised by innumerable others before and since, nightwalking is quintessentially objectless, loitering and vagabond.

Matthew Beaumont’s Nightwalking is published by Verso.