In 1957, VUE magazine described Soho as “the world’s wickedest square mile”. Back then, like its Parisian equivalent Pigalle, the neighbourhood traded on its bad reputation, luring the curious and the bohemian with its after-hours strip clubs, shebeens, peep shows and sex workers. That the gullible or the inebriated could fall prey to pickpockets, hucksters and conmen only added to the area’s illicit glamour.

Soho’s seediness survived, albeit in diluted form, into the 1980s and beyond, with such wayward songwriters as Shane MacGowan summoning its dark heart in song. In The Old Main Drag, he evokes the rent boys who once haunted the back alleys, charging a fiver “for a swift one off the wrist”. All gone, all gone. In today’s Soho, a fiver might just get you a detox smoothie or a couple of flat whites served by an underpaid, overworked millennial. Seediness has been supplanted by rampant development and Darwinian gentrification, with Crossrail currently cutting a swathe though the heart of the square mile, decimating entire blocks.

Overlooked treasure … one of William Klein’s characteristically dramatic shots. Photograph: William Klein

Against all this it is perhaps inevitable that Shot in Soho, the Photographers’ Gallery’s latest ode to its own neighbourhood, has an apologetically elegiac undertow. Apart from a few nods to contemporary Soho, this is an evocation of a lost world as distant as London fog and the once ubiquitous greasy spoon cafe. It is also, intriguingly, a show that avoids the obvious – no Bill Brandt, John Deacon or Daniel Farson, save for glimpses of their work in a vitrine filled with enticing ephemera.

Instead we have some overlooked treasures, not least a characteristically dramatic series by William Klein, commissioned by the Sunday Times in 1980 and published under the prescient heading: “Soho’s fight for survival.” The first image is the most dramatic: a group of men exiting a “sauna and massage” parlour, covering their faces against Klein’s flash. The street number on the canopy above is 69.

Klein roamed Soho, alert to moments that caught the tenor of the times: a trio of shaven-headed post-punks from Stevenage seem both beatific and oddly sinister; eager crowds of teenagers queue beneath a neon-lit cinema hoarding for Emanuelle: Queen Bitch. That these fresh-faced youngsters may be queuing for something else entirely was beside the point for Klein, for whom atmosphere was an essential element.

Photograph: William Klein

It is illuminating to compare Klein’s brash colour shots of Soho’s everyday people with Anders Petersen’s stark monochrome portraits from 2012. Swedish-born Petersen befriended his sitters before creating his moody, high-contrast portraits, all dark shadows and pale skin tones. Traces of the old Soho are detectable in the “otherness” of many of his sitters: tattooed faces, hard drinkers, after-hours clubbers. But there is also a sense of engagement with the subjects that reflects Petersen’s collaborative approach.

Elsewhere, two series from 1968 reveal how photojournalists were looking to find the humanity and desperation beneath the stereotypical representations of Soho. East Londoner Kelvin Brodie was a hard-boiled photographer who had covered conflicts in Northern Ireland and India before returning home. For a series called Passport to Soho, he accompanied special police teams carrying out late-night “sweeps” in a bid to clean up the area. His images shift from the everyday (shopkeepers and street-sweepers) to the shadowy and clandestine (young women being questioned and arrested by police). They often have the shadowy patina of snatched surveillance photos, perfectly complementing the subject matter.

Altogether more life-affirming is John Goldblatt’s series The Undressing Room, shot over four consecutive nights in a dressing room of a strip club for the groundbreaking photo-mag Creative Camera. Here, intimacy is all, the women seemingly at ease with Goldblatt’s often up-close presence, the unruly mundanity of their surroundings offset by their casual nudity and, in one instance, the appearance of one of the women’s infant sons, who seems perfectly at home among the greasepaint and G-strings. Only the radiant beauty and glamour of one interrupts the abiding sense of routine. Otherwise, Goldblatt captures the banality, even boredom, of the performers’ downtime as they smoke, chat, yawn and read magazines.

Banal glamour … a shot from The Undressing Room series by John Goldblatt. Photograph: John Goldblatt

Less appealing is the late Corinne Day’s diaristic series shot in her flat on Brewer Street in the 1990s, which seems more interesting for the fashionable people she shot – including a young Kate Moss – than for, as the catalogue insists, their “unparalleled” documentation of “the uninhibited 90s”. They seem too slyly staged for that. Likewise Clancy Gebler Davies’s late-90s, fly-on-the-wall shots of the often famous denizens of the Colony Room Club on Dean Street: Tracey Emin, Sarah Lucas, Gavin Turk. Again, it is the subjects rather than the style that fascinates, the bohemian bonhomie somewhat suspect in retrospect, given the avarice that was to follow.

The most recent series here is Daragh Soden’s Looking for Love, a title that promises a lot more than the images deliver. The most haunting piece is his short 8mm film projection in which an anonymous older gay man talks matter-of-factly about love over images of bars, clubs and street scenes. In the same room, screenshots from Tinder and Grindr speak of another kind of interaction, stripped of flirtation. Soden’s traditional street photography of present-day Soho – couples gathered beneath Eros, glitter balls, posed portraits – are disappointingly empty in contrast, particularly when compared to the vibrancy and intimacy elsewhere.

Perhaps in Soho, it just takes time for today to look as melancholically entrancing as yesterday.