Near the end of In No Great Hurry, a 2014 documentary about Saul Leiter, we see the 88-year-old photographer kicking back in his chair, illuminated by cool New York light. “I may be old-fashioned,” he says slowly. “But I believe there is such a thing as a search for beauty – a delight in the nice things in the world. And I don’t think one should have to apologise for it.”

Beauty and delight are not nouns one associates with New York street photography, a hard-edged genre popularised by the combative monochromes of Garry Winogrand and the flamboyant grotesques sought out by Diane Arbus. The Gotham of glass and grime, of hustle and raw noise, has gifted many things to photographic history, but naked aesthetic pleasure has rarely been among them.

Yet Leiter – who died in November 2013 – was a photographer of an entirely different order: an artist whose boundary-defying work segued effortlessly from street scenes to formal portraiture, from fashion to architecture, and which also embraced painting and sculpture. Elliptical, poetic, beautifully crafted, Leiter’s images are impossible to pin down. They rarely show us the New York we think we know. Yet they could never be of anywhere else.

Saul Leiter’s Through Boards (1957)

One of the numerous puzzles about Leiter is the fact that, until recent years, few people had even heard his name. Born in Pittsburgh in 1923, he grew up in an stringently Jewish household, and was expected to follow his father, a towering figure in Talmudic scholarship, into what amounted to the family business. But when his mother gave him a Detrola at the age of 12 – “I can’t remember why,” Leiter later shrugged, “I thought I’d like a camera” – a different course was set. He eventually dropped out of rabbinical school, and, at the age of 23, boarded a bus for New York, intending to focus on becoming a painter. An apartment in the dilapidated Lower East Side became his home and imaginative locus; over the next 60-odd years, only rarely did he venture out of the city. The overwhelming majority of his photographs document a mere handful of blocks.

The earliest pictures were black and white: candid, quietly observed portraits of family, friends and fellow artists, often illuminated by gauzy natural light, with an offbeat compositional sense. Leiter takes sly delight in subverting the viewer’s preconceptions, or rerouting them in unexpected directions: a 1952 photograph of Marcel Duchamp is taken through what appears to be the plate glass of a cafe window, the master hunched over a table, offering nothing more than the back of his coal-black greatcoat and the edge of a hatbrim. A portrait of John Cage from three years earlier is even more droll – merely the leather sole of an elegant left shoe tucked under a chair, a scattering of hobnails glinting in the light.

A disciple of Matisse and Bonnard, and keen to continue his own experiments with paint, Leiter, in the late 1940s, began to do something he would later insist was unremarkable: shoot on the still-new technology of colour film. Technically unreliable and forbiddingly expensive, colour was regarded by serious photographers as a vulgar novelty, the medium of oversaturated advertising hoardings and gaudy magazines. It was the antithesis of an artform, which photography was effortfully striving to be.

Saul Leiter’s Red Umbrella (1958)

Yet, for Leiter, colour was a rebirth, a medium that could match the inventiveness and accuracy of his eye. In Red Umbrella (1958) a street is filled with a foaming river of old snow, offset by the splodge of a woman’s cherry-red umbrella. In Taxi (1957), the car is photographed with such shallow depth of field and cropped so brusquely that it becomes a miasma of salmon, persimmon and saffron-coloured forms. The only hint that this sensual, half-molten mirage is a real life city featuring real life people is a well-tailored, well-tanned arm reaching from the back seat.

Many photographs hover on the boundaries of abstraction, planes canting towards each other then cavorting away again; often they are riddles that never quite resolve. As with the Duchamp portrait, Leiter uses mirrors and windows to tease the eye, piling half-glimpsed images on top of each other – the sharp white of a woman’s shawl imprinting itself on to the palm-leaf design of a shop dummy’s dress, or, as in Reflection (1958), a chiming collision of reflected faces caught in glazing. Just as frequently, condensation, rain or snow films and fogs the frame. Often what we most want to see is held tantalisingly out of reach. The impression is of coincidence or confusion brought momentarily into harmony, but also of the reverse: the cosy clutter of Americana – candy-cane barber’s poles, workmen’s ladders, fire hydrants, neon signs, raffish hoardings – thrown into luminous disarray. The camera is not used as a tool for psychological or documentary investigation but as a means of conjuring mystery. The cityscape dissolves and deliquesces before our eyes.

Leiter's camera is used as a means of conjuring mystery – the cityscape dissolves and deliquesces before our eyes

One of the most remarkable pictures from this period, Through Boards (1957), reads from a distance like a Rothko, made up of sombre, tenebrous swaths of black, white and russet. Only when you get close do you see that it is in fact a street scene, taken inside what is presumably a boarded-up shop. The merest strip of sidewalk – the white chromed prow of a car, a dark-suited man strolling past in a straw boater – can be glimpsed, like a frame from a silent movie patiently awaiting a caption. “There are the things that are out in the open, and there are the things that are hidden,” Leiter once told an interviewer. “The real world has more to do with what is hidden.”

Masterful as many of these photographs are, for decades they remained unseen. Though some of Leiter’s work was included in an experimental show at MoMA by the pioneering photographer-curator Edward Steichen in 1957, Leiter was reluctant to play the game of self-advancement, failing to reply to gallery invitations or hobnob with curators, and was nagged by doubts about the quality of his work. Acutely short of money, in the late 50s he began to take fashion commissions, providing Harper’s Bazaar, Vogue and other magazines with a stream of elegant if chilly images of shoes, jewels, gloves and capes.

Saul Leiter’s Carol Brown, Harper’s Bazaar (1958)

The burnished palette of many of his more personal images has become a Leiter hallmark, but it chiefly came about because the photographer was forced to use cheap, out-of-date film stock on which the emulsion had degraded. He simply liked the effect. Never fussy about the technical side of his craft, he exposed hundreds of rolls of 35mm film, but could not afford to have them printed; a tidewrack of slide boxes piled up in his crowded apartment-cum-studio. Meanwhile the pictures remained stubbornly unexhibited. Only friends saw them in occasional projector screenings. When colour photography was belatedly championed by the art establishment in the 1970s, via the work of William Eggleston and Stephen Shore, Leiter was left entirely out of the frame – not so much neglected as never found in the first place.

Some of his fashion photography was exhibited at the V&A in 1991, but it wasn’t until 2006 – courtesy of dogged advocacy by the curator Martin Harrison and the gallerist Howard Greenberg – that Leiter was given his first solo exhibition, rapidly followed by a monograph, Early Color. To widespread astonishment, the book’s first printing sold out almost immediately and Leiter was hailed as a forgotten pioneer. With customary wryness, he himself remarked that this late, long-overdue success was all very well, but “when I am listening to Vivaldi or Japanese music or making spaghetti at three in the morning and realise that I don’t have the proper sauce for it, fame is of no use”.

Director Todd Haynes has been voluble about Leiter’s influence – Carol's painterly images pay tribute to his 1950s world

A decade on, Leiter – if he is not yet famous, not exactly – might at last be getting his due. Major exhibitions in Paris and Vienna are followed by a retrospective at the Photographers’ Gallery in London, his first in a major UK venue. Mad Men and the vogue for mid-century modern has no doubt played its part; the film director Todd Haynes has been voluble about Leiter’s influence, notably in Carol, whose painterly cinematography pays adoring tribute to his half-drowned 1950s world.

Yet the mysteries remain, and are only slowly being unveiled. In 2014, Early Color was followed by a beautifully produced diptych, Early Black and White, which restored to attention pictures from the 1940s and 50s that have never before been seen, sharper observations of New York life than emerge in the colour work, more ghoulish and sinister. Leiter’s portraits of women, many of them nudes – including of his partner and fellow artist Soames Bantry – are particularly startling, frank in their intimacy yet also tantalisingly enigmatic. A number of paintings, often based on or incorporating photographs, have never been exhibited. According to his assistant Margit Elb, thousands more negatives and slides are still in boxes, yet to be sorted through. The Leiter who awaits us still has a few surprises up his sleeve.