‘For many years California frightened me,” Dennis Stock wrote in the preface to California Trip, first published in 1970. “For a young man with traditional concerns for spiritual and aesthetic order, California seemed too unreal. I ran.”

Stock, a naturally sceptical New Yorker who had served in the US Navy before hustling his way into the ranks of the esteemed Magnum photo agency, had instinctively picked up on the edgy undercurrents of the late 1960s Californian hippy dream. As the idealism of that decade peaked and faded, California became what Stock called a “head lab” – fomenting various radically alternative lifestyles fuelled by eastern mysticism, experiments in communal living, and all kinds of post-LSD mind expansion.

And, as the images in the newly reissued California Trip show, Stock’s initial wary incomprehension soon turned to fascination. In time, he came to see California as the frontier for a new kind of society where “technological and spiritual quests vibrate … intermingling, often creating the ethereal”.

Photograph: Dennis Stock/Magnum Photos

Almost 50 years later, and nine years after his death, California Trip now seems both prophetic and elegiac, Stock’s free-flowing approach allowing the contradictions of the time to speak for themselves. There are images of sun-kissed, back-to-nature hippy couples and marching black militants, missile bases and utopian communes, endless Californian beaches and a towering stack of rusting cars in a scrap yard. In one photograph, a tousle-haired infant frolics next to a Hells Angels motorcycle gang member. To Californians, he wrote, this was “all so ordinary as to be mundane”.

With hindsight, it is clear that California Trip upends our received notion of Dennis Stock, who remains most famous for his intimately observed images of the young James Dean in the months before his death in September 1955. Stock befriended the young Dean after seeing an early screening of East of Eden and subsequently photographed him on the wintry streets of New York and on a trip back to his family home in Fairmount, Indiana. When the ensuing photo essay appeared in Life magazine, it helped cement Dean’s status as a new kind of film star: moody, intense and ill at ease with the Hollywood fame factory. In the immediate wake of Dean’s untimely death in a car crash, Stock’s images attained an almost mythic aura that remains to this day, arguably overshadowing his other work.

‘The icon got in the way’ ... Stock’s shot of James Dean, New York, 1955. Photograph: Dennis Stock/Magnum Photos

“Dennis was not always happy about the prominence of the James Dean photos,” says Hanna Sawka, who directed the illuminating 2011 documentary, Beyond Iconic: Photographer Dennis Stock. “He made some quite bitter comments about the pictures, that people weren’t seeing them as they should because the icon got in the way.” Stock’s widow, author Susan Richards, who describes him as “the most confident person I ever met”, recalls that the prominence of the Dean photographs “maybe bugged him a little bit, but he also knew that the iconic stature of images enabled him to have the lifestyle he had”.

Stock had joined Magnum in 1951 and, the following year, shot an extraordinarily candid series about Emperor Haile Selassie of Ethiopia for Life magazine. Following the success of the Dean series, he began photographing jazz musicians, merging stark, monochrome portraits of the likes of Billie Holiday and Louis Armstrong with often dramatic images of their performances.

In a style that was unadorned and intimate, he set about capturing the reality of the nomadic jazz life as well as its drama. In one evocative image, a struggling musician, Bill Crow, lugs a bass across a Manhattan street in what looks like the early hours of the morning. In another, he captures an ecstatic Earl Hines pounding on the piano in a smoky club, the sense of the music’s joyous momentum palpable in a single stilled moment.

Against all this, the images in California Trip mark a dramatic departure, though one that had been taking shape in his work throughout the 1960s. The more free-flowing narrative style of Stock’s Californian pictures was surely informed by his dalliance with the moving image, which began when he left Magnum in 1968 to focus on documentary film-making. It also speaks of a relentless creative curiosity and open-mindedness that, as Sawka’s documentary shows, was not always immediately apparent in his everyday interactions with people.

In the film, as he teaches a photography class, his students often seem overawed by the sheer presence of a man whose opinions tend to be strongly held and forcefully articulated. “He was quite a personality,” says Sawka, laughing. “Sometimes people were offended by him, but the gruffness masked a deep sensitivity and integrity.”

Shadow play … Playa Del Rey, LA, 1968. Photograph: Dennis Stock/Magnum Photos

Richards concurs: “He took no prisoners. He could be harsh with people, including his friends and, the next moment, the gentlest, sweetest guy. If you didn’t know him, he could appear arrogant.” Richards, who was his fourth wife – “I met him when he was older and mellower and not travelling so much” – puts his combativeness down to a childhood in the Bronx that was marked by poverty and family dysfunction.

“His mother was a helpless person, and his father was absent a lot because his job as a house painter required him to travel. [Stock] was raised in a family that moved in the night a lot because they could not pay the rent. He told me that, when he was just seven, he was working odd jobs to support his mother. That kind of experience leaves its mark and I think that, to a degree, he was ashamed of his childhood poverty.”

It also made him resilient. He served his photographic apprenticeship with Gjon Mili, an Albanian-born pioneer in movement photography, who once brutally informed Stock he would never be a Life photographer. “Dennis did not see that as a bad thing,” says Richards. “It rolled right off his back. He interpreted it as that he would never fit the mould that Life required – which was fine by him.”

For all his combativeness, Stock was essentially a liberal New Yorker who was instinctively drawn to the promise of the Californian counterculture of the late 1960s and early 70s. The most well-known image from California Trip is also the most instinctive and intuitive. Shot from behind, his vibrant portrait of a young woman in a cotton dress dancing on stage at a rock festival in Venice Beach in 1968 exudes all the exuberant optimism of the time. “This kid just marched up on stage and started dancing,” he would later recall, comparing her to “a contemporary ballerina” – and himself to his hero, Henri Cartier-Bresson.

The hippy dream ... Novato, California, 1968. Photograph: Dennis Stock/Magnum Photos

California Trip, though, perhaps owes more to an American tradition of road photography that stretches back to Robert Frank, Walker Evans and beyond. Stock’s east coast outsider gaze settles on the darkness of the California dream as well as the light: bikers, anti-war protesters, the disenfranchised as well as the visionary. In one arresting image, a black couple in a parade in Watts, Los Angeles, have created an ornate tableau in which they are chained to the Liberty Bell. An idyllic image of a hippy couple on horseback gives way to a portrait of Anton Szandor LaVey, the self-styled high priest of the Church of Satan, who poses theatrically in front of a pentagram and a human skull.

“If there is a thread to be observed throughout my work,” Stock later said, “it’s that I’m relatively affirmative, I’m not inclined to make fools of people and I love beauty.” As the reissued California Trip attests, he had an acute eye, too, for the shadows cast by the unforgiving Californian light, the darkness beyond the surface dazzle.

• California Trip is out now, reissued by Anthology Editions.