"Don't you know I'm a recluse?" Danny Lyon offers from his adobe home in New Mexico. It's a joke of sorts. The 70-year-old photographer, who established his reputation documenting the US civil rights movement, prisoners in the Texas penal system and motorbike gangs in Chicago, hasn't been so visible in decades.

Lyon may not be as immediately recognisable as predecessors such as W Eugene Smith or Walker Evans, or even near-contemporaries like Robert Frank, but his contribution to photography's "new realism" movement is no less significant, and a retrospective of his work currently at the Menil Collection in Houston goes a way to establishing that.

Lyon's anti-authority, 60s-warrior idealism is undimmed. He recently published a book on workers in the industrial Shanxi province of China and, late last year, turned his Leica on Occupy camps in nearby Albuquerque, as well as New York's Zuccotti Park, Oakland and Los Angeles.

"Occupy makes my heart beat!" he says on a trip to get hay for the horses. "I hope there's blood on the streets."

Intemperate, perhaps, but also unapologetic. At a recent awards ceremony at the Missouri School of Journalism, he warned students of certain "truths" – that a "puritanical code has eviscerated the US political system", that there's a growing attack on Mexican immigrants, and that good journalism is about truth not ideology.

"I'm a 60s guy," Lyon offers by way of explanation. "The civil rights movement was my coming of age." At the same time, Life magazine, the photojournalism colossus of the era, increasingly towed the establishment line. "I was against it and I knew in my heart of hearts there was a better way to take photographs of people and the world."

In 1962, aged just 20, Lyon left the security of his upbringing in Queens, New York, to join the Student Nonviolent Coordinating Committee in Selma, Alabama. The group, known colloquially as SNCC, had been established two years earlier after a group of black college students from North Carolina A&T University were denied service at a Woolworth's lunch counter.

"It was my good fortune to stumble into the story early," Lyon says. "Being in SNCC politicised me. Having said that, I wasn't black and I was free. My agenda was photography and books, and what is now called media."

A week after arriving to join the group in Albany, Georgia, Lyon was in jail with other protesters; Rev Dr Martin Luther King was in the cell next door. Though he regards them as immature, the pictures Lyon took foretells the style he would come to champion – formal in composition, rooted in realism, but also intimate. His only formal training, if it can be described as such, was studying Bruegel's mastery of composition in an introductory humanities course at university.

Lyon reflects that "the movement" was a natural place for the son of European émigrés (his mother fled the Soviet pogroms; his father, the Nazi) to gravitate. But more than any overt identification, he wanted experience and America, at least photographically, was wide open.

"I wanted adventure and I wanted dirt – all the things my parents didn't want. I wanted freedom for myself. I had a huge sense of adventure and I wanted to experience it."

But by the end of '64, Lyon was through with SNCC and any direct involvement with the civil rights movement. "I walked away and I didn't look back. I saw these things as subjects, and myself as a journalist."

It was Lyon's next adventure that would define him. He travelled north from Selma, bought a Triumph TR6 motorbike and joined the Chicago Outlaws motorcycle club. It's hard to point to a more complete vision of rebel freedom than The Bikeriders, a slim volume of photographs and conversations Lyon published four years later. While the book was a commercial failure, first edition copies now change hands for higher prices than Frank's The Americans.

Sky ride … detail from Crossing the Ohio. Photograph: Danny Lyons

The most famous images from the series – Route 12, Wisconsin and Crossing the Ohio, Louisville – are up there with the images of teenage rebellion Hollywood created for James Dean and Marlon Brando. But Lyon's subjects were real and dangerous.

As he wrote in the introduction, he joined the Outlaws to capture "the spirit of the hand that twists open the throttle on the crackling engines of big bikes and rides them on racetracks or through traffic or, on occasion, into oblivion".

"Photographers show character through how people look and the bikers were a perfect subject because they were what they looked like," Lyon reflects now. "They had leather jackets, they were dirty, they had weapons and boots."

He pulls out a letter from New Journalism compatriot Hunter S Thompson, who was at that time riding out with bikers on the west coast for what would become his work Hell's Angels. In it, Thompson advises Lyon against wearing a helmet in town but to wear one in the country where speeds are higher. Even counter-culture heroes, it turns out, make sensible road-users.

While the emerging counter-culture got high, Lyon turned his focus towards the Texas prison system for Conversations with the Dead (1971). He photographed inmates in six prisons and in particular Billy McCune, a rapist whose death sentence was ultimately commuted to life in prison. "I spent 14 months talking to these guys. They broke my heart. Prison is one story after another, and every one breaks your heart. I was young enough then to care."

By 1968 and only midway through his 20s, Lyon had completed his three signature projects. In a fourth, The Destruction of Lower Manhattan (1969), he documented the demolition of historic neighbourhoods to make way for the original World Trade Centre.

The reclusive Danny Lyon is, of course, less reclusive or defiant than he would like you to think. But he remains the dark romantic spirit who is finding new inspiration in this time of unrest and uncertainty.

"Photography is a lonely journey – that's what Robert Frank said – you really have to be by yourself because you're trying to interact with reality. I put myself through an ordeal in order to create something."

Forty years on, Lyon bemoans the loss of the America that opened up to him then. Homogeneity, he feels, is the modern curse. "If you're in love with the road, it doesn't really exist much any more – or you've really got to work to find it," he says. "Try going round America now. You're trapped in a nightmarish existence of food chains and chain motels. There are no more mechanics – try finding a guy with dirty hands who can change your tyre!"

China, by contrast, is now wide open. "I could go right in and they don't even look up at me. In the US, I'm used to being assaulted by people saying, 'I'm gone stick that camera up your ass, don't look at my daughter,' and so on ..."

Brightening slightly, Lyon recalls a conversation with Hugh Edwards, curator of photography at the Art Institute of Chicago, and a formative influence on both himself and Frank. Lyon told Edwards of a trip he'd made to Houston where he'd seen miles of new tract housing and ugly architecture. "I said to him, 'They're destroying America.' And he says to me, 'Nothing is ever complete.'"