His stark masterpiece The Americans changed photography. Yet there was more to this countercultural hero who captured the debauchery of the Stones – and his own personal tragedies

‘People want to know so much,” Robert Frank answered wearily when I asked him back in 2004 about the lasting resonance of his classic photobook, The Americans. “All the time, this wanting to know. Where does it lead? Nowhere.”

We were sitting in the spartanly furnished kitchen of his apartment on Manhattan’s Bleecker Street, my tape recorder resting on a rickety table between a large brick of a mobile phone and a single bread roll. He was, it strikes me now, a man for whom fame and its comforts meant very little, whose sadness seemed more palpable than his genius. Almost 80 then, he seemed ineffably world-weary.

Almost 50 years earlier, funded by a Guggenheim Fellowship, Frank had set off to drive across America, accompanied by his young wife Mary, and their two young children Pablo and Andrea, on the first of three epic road trips. The 500 rolls of film he subsequently shot were rigorously edited down to just 83 monochrome images for Les Americains, which was published in France in 1958. A year later, the American edition appeared to a storm of critical disapproval, Frank being accused of everything from “general sloppiness” to anti-Americanism.

I wanted to present what I saw, pure and simple

Swiss-born Frank had photographed what he saw with his keen outsider’s eye, capturing what Diane Arbus would later describe as “the hollowness” of many American lives. In doing so, he challenged not just the prevailing romanticism of the American pictorialist tradition but also the easy certainties of photojournalism. “I was tired of romanticism,” he told me. “I wanted to present what I saw, pure and simple.”

With The Americans alone, Frank staked his claim to being one of the defining photographers of the last century, the monochrome starkness of his images alienating many but alerting others to a new brutally realist tradition. “Robert Frank, Swiss, unobtrusive, nice,” wrote Beat writer Jack Kerouac, in his celebrated introduction to the book, “with that little camera that he raises and snaps with one hand he sucked a sad poem right out of America on to film, taking rank among the tragic poets of the world.”

Facebook Twitter Pinterest Politics in the 50s … Democratic National Convention, Chicago. Photograph: Robert Frank, from The Americans

The visionary curator and critic John Szarkowski, who had created an earlier show of the work at the Museum of Modern Art in New York, wrote that Frank “established a new iconography for contemporary America, comprised of bits of bus depots, lunch counters, strip developments, empty spaces, cars, and unknowable faces”. One could see why, though, all those years later, Frank had grown tired of being asked about The Americans. It remains one of the most iconic photobooks of all time, albeit one which, as he told me, needed no explanation beyond the images themselves.

By the time of our meeting in 2004, he seemed to have grown tired even of photography, proudly showing me a catalogue of his second wife June Leaf’s painting. “I envy her freedom,” he said, “to sit down in front of a blank page with no machine to get in the way. That is freedom. Photography is not freedom.”

How Frank arrived at that conclusion is perhaps not entirely to do with his relationship to the medium that he redefined, and which came to define him despite his later foray into experimental film-making. His life was also marked by great tragedy: the loss of his daughter, Andrea, aged 20, in a plane crash in Guatemala in 1974; and the death of his son, Pablo, who died in 1994 after years of mental illness. “Life is not beautiful all the time,” he said, just after he noticed me looking at a portrait of Pablo and him, pinned to the wall of an adjacent room. “Life can be good, then you lie down, and stare up at the ceiling, and the sadness falls on you. Things move on, time passes, people go away, and sometimes they don’t come back.”

Facebook Twitter Pinterest Downbeat take … Mick Jagger, from the documentary Cocksucker Blues, 1972. Photograph: Robert Frank/Rex/Shutterstock

In The Lines of My Hand, a strange and beautiful photobook published in 1972, the commonplace images are marked by scratches and scribbles, a creative defacement that reflected the tidal wave of grief at his daughter’s death as well as his dissatisfaction with photography. In one of its most powerful pictures, his hand, reflected in a mirror, holds a tiny skeleton doll, the words “sick of goodby’s” are smeared on the glass. His work, he told me, had “shifted from being about what I saw to being about what I felt”. Photography, though, could not contain nor express the depth of his sadness. “I was really destroying the picture,” he says, staring at the floor. “I didn’t believe in the beauty of a photograph anymore.”

If Robert Frank’s legacy rests mainly on The Americans, it is worth remembering how restless his creative imagination was, from the freeform anarchy of films like Pull My Daisy, an unruly evocation of the Beat aesthetic featuring Kerouac and Allen Ginsberg, to the infamous Cocksucker Blues, his verite and decidedly downbeat take on life on the road with the Rolling Stones at their most glamorously debauched.

Frank’s singular vision did not sit well with Mick Jagger, who set out to suppress the film. “They sent lawyers, they sent planes, they sent the sheriff,” he told me, laughing, “It was out of proportion, like everything they did. It was comical really. I fled to Nova Scotia. I just wanted to be left alone.” In his absence, the Stones won a prohibitive court order that banned its screening unless Frank himself was present. Its infamy grew accordingly.

The Stones sent lawyers, they sent planes, they sent the sheriff. It was comical really. I just wanted to be left alone

Frank remained an outsider until the end, and an increasingly reclusive one. He had served his apprenticeship with the great Walker Evans, who, he said, never allowed him to watch while he worked. Frank’s influence eclipsed that of his mentor, which is saying something, and was such that it touched even those who seemed to reject his spare, documentary approach.

“I had Robert Frank’s The Americans as a teenager,” the photography pioneer Jeff Wall once told me. “In fact, I made pencil drawings from various photographs in it. Frank and Walker Evans closed the door for me. What they did was so well done, I could never have matched it and I don’t think anyone has. They nailed it once and for all. That was a huge realisation – that I could not follow in their footsteps.”

Artist Ed Ruscha concurred, once saying: “Seeing The Americans in a college bookshop was a stunning, ground-trembling experience for me. But I realised this man’s achievement could not be mined or imitated in any way, because he had already done it, sewn it up and gone home. What I was left with was the vapours of his talent. I had to make my own kind of art. But wow! The Americans!”

Facebook Twitter Pinterest ‘The kind of photography I did is gone. It’s old’ … Frank in 1954. Photograph: Dpa Picture Alliance/Alamy Stock Photo

Frank, alone, seemed less convinced by, or simply uninterested in, his own achievement. It was part of his past, photography’s past, and he was an artist who did not look back. “The kind of photography I did is gone. It’s old,” he told me in 2004, as the digital deluge gathered pace. “There’s no point in it any more for me, and I get no satisfaction from trying to do it. There are too many pictures now. It’s overwhelming. A flood of images that passes by, and says, ‘Why should we remember anything?’ There is too much to remember now, too much to take in.”

Yet we do remember – and revere – what he saw and what he achieved with The Americans. Through his deep looking, he taught us how to see the world anew, without sentiment or illusion. His alert and unforgiving gaze showed many Americans what was right under their noses yet invisible to them. It was, in retrospect, a wake-up call but, more than that, a beautiful, if bleakly poetic, vision of modern America, as evocative and unrelenting in its way as any work by Philip Roth or Saul Bellow.

The last time I saw Robert Frank, he was sitting outside his Greenwich Village apartment with June, on a warm late summer day. An old couple bathing in the last of the early evening sun. We chatted and, as I was departing, I told him they looked like they belonged in one of his early photographs. I think I detected the ghost of a smile. I think of him now, the instinctive outsider, the sad genius of American photography, and his words echo in my head … “Things move on, time passes, people go away, and sometimes they don’t come back.” Only the photographs remain, resonant and unsparingly truthful. In them, he lives on.