In 1993, when I first read Vineland, Thomas Pynchon's great novel about washed-up 60s radicals, I was living in northern California with two middle-aged hippies. A certain bohemianism and lawlessness still lingered in their creaking house in Berkeley. The cable TV service was siphoned off from a neighbouring property. One housemate drank rank-smelling wheatgrass for her breakfast. The other disappeared at weekends on unspecified operations against the logging companies in the redwood forests up near the Oregon border. When she was home, she almost never left her basement room. She emphatically instructed me to deny her existence if anyone called.

I was an inquisitive male postgraduate in my early 20s: a classic potential Pynchon reader. One day after the winter had set in, with its low skies and week-long rains, I went down to one of the bookshops on Telegraph Avenue in Berkeley, where some of the legendary 60s student demonstrations had taken place, and bought Vineland.

The book had come out three years earlier, to approving but subtly disappointed reviews. Pynchon's previous novel, the seemingly all-encompassing second world war adventure and postmodern box of tricks Gravity's Rainbow, had been published in 1973; during the 17-year wait for a follow-up, all sorts of rumours had spread about what the famously brainy and reclusive American prodigy, only 35 in 1973, would produce next. "We heard he was doing something about Lewis and Clark," Salman Rushdie wrote in the New York Times in 1990. "Mason and Dixon? A Japanese science-fiction novel? . . . A 900-page Pynchon megabook about the American civil war?"

In fact, Vineland was less than 400 pages long, largely American rather than international in its settings, realistic in style for long stretches, and relatively earnest, even sentimental, compared with what Pynchon had previously written. In a typical response in Time magazine, Paul Gray compared an early, lovingly drawn scene of greedy birds stealing food from a dog bowl left outside by the absent-minded main character, Zoyd Wheeler, with the dazzling opening panorama in Gravity's Rainbow of a V-2 rocket descending on London ("A screaming comes across the sky . . ."). Gray concluded: "There seems to have been a little downscaling going on." In the Nation, John Leonard suggested an explanation: Vineland was "a breather between biggies", a John the Baptist of a novel preparing the ground for "another, darker, [more] magisterial" Pynchon production.

Pynchon's post-Vineland career hasn't quite turned out like that. But in 1993, as soon as I started reading the book, I forgot all about his professional arc. Vineland begins in a small northern Californian city of the same name, fictional but incorporating elements – misty weather, marijuana production, a freewheeling, end-of-the-road population – of existing local settlements. It is 1984, the height of Ronald Reagan's rightwing ascendancy, and the tentacles of the American state are reaching into the crevices where former 60s rebels such as Wheeler have been scratching a living relatively undisturbed for the past decade and a half.

Zoyd is a typically cartoonish Pynchon character, equal parts Homer Simpson and the Dude in The Big Lebowski, but unlike previous Pynchon protagonists, there's a depth and a sadness to him. He lives with his teenage daughter, Prairie, who regards him with a mix of fondness and contempt and is more at home with materialistic Reagan-era values. Prairie's mother, a radical 60s film-maker called Frenesi, is long gone. Wheeler yearns for her. In one of the book's many lyrical passages, he remembers their wedding: "afternoon on a smooth green gold California hillside, with oak in darker patches, a freeway in the distance, dogs and children playing and running, and the sky, for many of the guests, awriggle with patterns of many colors, some indescribable."

By the time Vineland came out, looking back at the 60s from the vantage point of the 80s was a well-worked theme in American and British culture, particularly in films, such as Return of the Secaucus Seven (1980), The Big Chill (1983) and Withnail and I (1987). There was nothing like a few years of Thatcherism or Reaganomics to make communes and experiments with chemicals look alluring again.

But the fact that Pynchon had published nothing between 1973 and 1990, between the oil crisis that marked the effective end of the 60s and the decade when, with communism having just collapsed, rightwing capitalism seemed at its most all-conquering, gave his thoughts on how the world had changed real weight. His earlier novels, V, The Crying of Lot 49 and Gravity's Rainbow, were packed with clever facts and speculations about secret power networks and European colonialism and the American military-industrial complex. Vineland marked a maturing. Instead of a precocious cynicism about politics, Pynchon, now 53, expressed anguish about America's trajectory from Nixon to Reagan: "the Repression went on, growing wider, deeper, and less visible, regardless of the names in power."

Frenesi, it is revealed, has been a government informer since the 60s. In a series of cinematic flashbacks, Pynchon shows her journey: from wide-eyed witness of the decade's student upheavals, to on-off lover of a manipulative government operative called Brock Vond, to spectacular betrayer of her counterculture peers and a shameful, transient life in "a pale humid Sun Belt city", whose name would "be denied to civilian eyes [in her official file] by federal marker pens".

When the narrative returns to 1984, Frenesi, Zoyd, Prairie and Brock are all set on converging paths, with the showdown seemingly destined to take place in the last crumbling bastion of the American 60s, the city of Vineland. But this overarching, potentially melodramatic story develops instead in unpredictable fits and starts, regularly looping back on itself or disappearing beneath a tangle of other plotlines. There are extended sequences involving ninja warriors, possible extraterrestrials and a huge monster which stamps on a laboratory "belonging to the shadowy world conglomerate Chipco". Finding a concealed geometry behind such apparent digressions is one of the main pleasures for hardcore Pynchon fans, but for me, even at 23 and with time on my hands, there were wacky, lurching chapters I could not quite be bothered to work out.

I preferred the slower sections set around Vineland. In long, intoxicating sentences, with chains of adjectives and commas strung out across the page like telegraph lines through a forest, Pynchon mapped the city and its hinterland: a maze of old logging mansions and unmarked roads, sudden fogs and "long redwood mountainslopes where shadows came early and brought easy suspicion of another order of things".

Reading these passages late at night, as the same trickling winter murk hung over Berkeley and my housemates went about their unfathomable hippie rituals, Vineland spooked me like the best landscapes of Conrad and Dickens. I soon forgot precisely how the book ended but its atmospheres stayed with me.

A few months ago, short of something to read, I picked up Vineland again. Within a page I was committed to finishing it. The book seemed wiser and warmer; rather than its sense of foreboding, I noticed all the writing about mealtimes and family gatherings. While Pynchon's earlier novels are full of brittle parties and orgies, the participants feverishly trying to blot out the horrors of the outside world, Vineland offers a more solid conviviality. Even Brock Vond's climactic arrival in the city, as malign and otherworldly as Darth Vader in Star Wars, is delayed by a leisurely description of an outdoor family reunion involving the book's more benign characters:

Behind the mountains . . . light grew in the sky . . . Soon toasters and wood fires . . . working on bacon, links, eggs, flapjacks, waffles, hash browns, French toast . . . were sending out branching invisible fractals of smell . . . fat smoke, charring spices, toasted bread, just-made coffee.

One of Vineland's strengths, it becomes obvious second time round, is its ability to hold opposing themes in balance: a defiant optimism that some communal instincts had survived the end of the 60s, alongside a deep pessimism about the behaviour of the state. Even the book's heroes and villains, defined with romantic simplicity at first glance, blur together more interestingly as you get to know them properly. Frenesi is still half in love with the counter- culture she is betraying. Zoyd, the anti- Reagan dissident, is dependent on regular government disability cheques. Brock, the conservative zealot, is out-manoeuvred by rival Washington players. And the counterculture which Pynchon seems so to venerate? There are suggestions it always had a spivvy capitalist side: in Vineland "Prairie worked at the Bodhi Dharma Pizza Temple . . . twelve-grain crust with the lightness and digestibility of a manhole cover."

All these hypocrisies and mixed motives and contradictions, as well as being a good source of comedy and plot twists, have the feel of political lives as they are actually lived. This gives Vineland much more depth than most political novels. In recent years, the radicalism of the 60s and 70s and what happened to it subsequently has again become a favoured theme for American and British fiction; but the resulting books, such as Hari Kunzru's My Revolutions and Dana Spiotta's Eat the Document, while compelling as storytelling, have contrasted the past and present too crudely. As the 60s and 70s counterculture recedes further into history and some of its elements – early feminism, early environmentalism – come to seem ever more important and prescient, it is easy to paint the participants as a uniquely heroic generation. But Vineland reminds you that heroic and unheroic political eras, such as they exist at all, are always intertwined.

Its warnings about the capacity for repressiveness of US governments also read well now. Perhaps the most dramatic section in the book describes the abduction of Frenesi and other activists – in "field-gray trucks, locked shut, unmarked" – from a rebellious 60s campus to a secret old military base in a concealed California valley. Location aside, it sounds awfully like Guantánamo.

Yet since Vineland, despite the creeping authoritarianism of the Bush administration and the war on terror, Pynchon has not written so directly about contemporary politics again. Instead he has published the kind of books Rushdie and others were expecting to appear when Vineland did: Mason & Dixon (1997), a vast, semi-parodic historical novel about the surveyors of 18th-century America, and Against the Day (2006), an even vaster fiction about early 20th-century balloonists, anarchists and other characters so numerous and eclectic as to be almost beyond summary. Reviewers of these two books grew increasingly impatient with their archness and shagginess and scale. The relatively modest and heartfelt Vineland began to look more of an achievement: not "a breather between biggies", but perhaps Pynchon's last fully realised novel.

Then again, he is only 73, and his rate of production seems to be accelerating again: last summer, after only a three-year silence, he produced a Vineland prequel of sorts, Inherent Vice, set in some of the same California locations during 1969 and 1970, and involving a similar-sounding cast of hippies and government agents. I've not read it yet. But it's short by his standards, and the reviews were warm. Maybe all I need is the right Californian summer rental.