London at night is another place entirely to its daytime incarnation, with different rules and different inhabitants. If this remains true of today's 24-hour, 21st-century metropolis, in which commuters setting out for work rub shoulders on the Tube platform with clubbers coming home, how much more so was it of the past versions of the city that lie buried beneath its streets?

In this magisterial, perambulatory survey Matthew Beaumont excavates strata upon strata of literary sources to help us find the answer. In the process he both explores the night side of some of English literature's greatest writers and resurrects many unjustly forgotten voices, who in their turn give flickering life to the denizens of London's darkness: the vagrant, the fallen, the alienated and the dispossessed. Above all, he releases an ancient, urban miasma that rises from the page, untroubled by electric illumination, allowing us to inhale what Shakespeare's contemporary Thomas Dekker called "that thick tobacco-breath which the rheumaticke night throws abroad".

In the 13th century, cities were far from the connected, open spaces they are today. At sunset all fires and lights were extinguished. Great peals of bells heralded the closing of the gates in the city wall until dawn. Night air was known to be unhealthy. Those who walked in it were, at best, eavesdroppers at neighbours' windows; at worst they might be intending to "walk abroad a-nights and kill sick people under walls", like the serial murderer Barnabus in Marlowe's play The Jew of Malta.

The night watch patrolled the streets for 500 years before the first Peelers were recruited. Given authority to arrest "strangers", such watchmen were unarmed, corrupt, often pimps or fences of stolen goods and subject to ridicule and attack. By 1283 Cornhill boasted a prison, The Tunn, purpose-built to accommodate "Night-Walkers and other suspicious persons" so they could be held until morning to be brought before a judge. Indigents born a century or two later might find themselves incarcerated in a Bridewell, or workhouse.

Yet still there were those who refused to stay abed or, more often, had no bed to stay in. Beaumont's night-walkers are divided, just like the diurnal world, by class and gender. "Throughout London's history", he writes, "the homeless and the bohemian, the socially and the spiritually disenfranchised, have coexisted in its obscurest spaces." Differences between them were coded in language. Aristocrats on drunken revels, accompanied by servants with torches, were noctambulants, noctambulists or noctambules, while the lumpenproletariat, walking merely to keep warm and with no place to go, were noctivagants or noctivagators.

Navigating between these parallel worlds stalked those authors who have haunted the streets of London at night, in search of sleep, subject matter or themselves and who populate the pages of Beaumont's book with anecdote and quotation. The insomniac and "violent" night-walker Charles Dickens is here of course, fleeing his wife and his relentlessly demanding fictional characters, but so is Ned Ward, the first issue of whose periodical The London Spy of 1698 boldly declared its intention to report on the urban night with the words "A fig for St Augustine and his doctrines, a fart for Virgil and his elegance, and a turd for Descartes and his philosophy", before embarking on a picaresque journey ending in a delicious-sounding "smoky boozing-ken" of a pub.