I am not old enough to have lived through a yellow or black, dirty and suffocating London fog, but like many others I feel I have experienced it, especially through TV adaptations of the classics, with their inevitable use of fog as a backdrop to tales of mystery and evildoing set in 19th-century London. No representation of Sherlock Holmes or Jack the Ripper would be complete without it. Switch on the fog machine and light a dim gas lamp in the street and you have immediately told viewers what to expect.

But for many writers London fog was much more than a simple scene-setting device. Charles Dickens first conjured the image of foggy Victorian London in fiction. You can almost feel the clammy, tactile greasiness of the vapour as you read the opening passage of Bleak House (1853), with its evocation of fog coming “down the river, where it rolls defiled among the tiers of shipping, and the waterside pollutions of a great (and dirty) city”.

Facebook Twitter Pinterest Fog at Ludgate Circus, London in 1922 Photograph: Getty Images

Of course, in Bleak House Dickens uses fog to symbolise the mystification caused by the endless obduracy and complexity of the law. But he employs it in many different ways in his other works, too. In his last finished novel, Our Mutual Friend (1865), he defines the city by its fog. The surrounding countryside may be foggy, but there it is “grey, whereas in London it was at about the boundary line, dark yellow, and a little within it brown, and then browner, and then browner, until at the heart of the City … it was rusty-black.” Fog is employed here as a metaphor for the moral corruption caused by a lust for money. Yet contemporary newspaper reports indicate that this description, fantastic though it may seem, is far from inaccurate: London fog really was yellow, coloured by the sulphurous fumes emitted by millions of domestic coal fires, and in places it was indeed black, filled with countless tiny fragments of soot that were fatal to those with weak lungs, the elderly or babies.

Dickens’s gloomy view was taken up by other writers. In 1880 William Delisle Hay, a pioneer of science fiction, published a novella entitled The Doom of the Great City; Being the Narrative of a Survivor, Written AD 1942. In this a black fog that has lasted from Christmas until February 1882 destroys all life in the capital. “The murky atmosphere, the dingy gloom” of London hides “the odious colours of the evil that lies hidden behind the awful pall”. Fog brings moral retribution to the city’s inhabitants for their sins. London has become “a black obscurity” and the dead are everywhere. When the narrator arrives home, he finds a picture of domestic normality, with the cat on a chair, his mother and sister seated next to each other, but all are dead.

Hay’s story was the first of many in which anxiety about London’s future was probed through fiction. Other writers explored the potential dangers London fog could pose to lone women. HG Wells’s novel Love and Mr Lewisham may have been written in 1899 but the author chose to set it in the 1880s to accentuate the dangers: “Thick fogs … the horses looming up suddenly out of the dark … the street lamps, blurred smoky orange at one’s nearest, and vanishing at twenty yards into dim haze, seemed to accentuate the infinite need of protection on the part of a delicate young lady who had already traversed three winters of fogs, thornily alone.”

Facebook Twitter Pinterest Anna Maxwell Martin in Bleak House Photograph: BBC

Henry James employs London fog as a metaphor to explore the state of mind of his heroine, Isabel Archer, in The Portrait of a Lady (1881). Her courage has been questioned by her sister and, as if in answer to the criticism, Isabel walks back to her hotel in the fog unchaperoned. James lists the reasons why this is an act of “absolute boldness” in a mock-romantic style: “The early dusk of a November afternoon had already closed in; the street lamps, in the thick, brown air, looked weak and red; our heroine was unattended and Euston Square was a long way from Piccadilly.” She wishes to choose her own fate, and plunges in.

The 1880s proved to be a high point not only of fog’s density and frequency in the capital, but also of social and political anxiety. On 8 February 1886 a protest meeting of the unemployed in Trafalgar Square turned nasty when “roughs” began to overturn carriages and throw stones through the windows of gentlemen’s clubs in Pall Mall. The following day a dense fog descended on London. The fog thickened on 10 February and “the disorderly classes” gathered again. A mass panic broke out in the West End, with rumours flying that 10,000 dockers were marching on Trafalgar Square, destroying property as they went.

The 1880s also experienced the panic created by Jack the Ripper, a figure indelibly associated with London fog. One of the first fictional accounts of the Ripper murders was written by Marie Belloc Lowndes, who had worked with WT Stead on the Pall Mall Gazette, which had extensively covered the crimes. Her book, The Lodger (1913), turned into a film by Alfred Hitchcock in 1927, revolves around a similar series of murders of women by “The Avenger”. Like the Ripper, the murderer is never revealed, though a landlady strongly suspects that her lodger may be responsible. London fog sets the scene for many of the murders. This theme recurs in subsequent retellings of the story. Decades later, an episode of Star Trek entitled “Wolf in the Fold” (1967) features a homicidal alien that comes from the foggy side of its planet. The TV series The Avengers also has a Ripper-type villain, known as the “Gaslight Ghoul”, in an episode entitled “Fog” (1969).

Facebook Twitter Pinterest Ivor Novello in Alfred Hitchcock’s The Lodger, 1927 Photograph: PR

As the Ripperologists Gary Coville and Patrick Lucanio note, “a writer or director merely needs to add a soupcon of fog, a dram of yellow gaslight … The result screams Jack the Ripper.” Despite such associations, however, contemporary meteorological reports reveal that the real Ripper actually committed his crimes on clear nights. There was no fog around to conceal his dark deeds.

In a similar way, Sherlock Holmes is now closely associated with fog. Yet there is very little of it in the Conan Doyle tales, almost all of which are set under clear skies. “The Bruce-Partington Plans” is almost the only Holmes story in which London fog is employed as a plot device, concealing the criminal as he hides the body of his victim on the roof of an underground train.

Fogs persisted in the capital all the way up to the 1960s, but they have become associated only with a brief few years in the late 19th century, reduced from a metaphor to a cliche.

• London Fog: The Biography by Christine L Corton is published by Harvard (£22.95). To order a copy go to bookshop.theguardian.com or call 0330 333 6846. Free UK p&p over £10, online orders only. Phone orders min. p&p of £1.99.