In popular imagination, at least, the 1970s was the golden age of excess in Hollywood. This was the decade when the studios were taken over by a new generation of maverick, mostly male, artists and film-makers, whose creative brilliance was matched by their wild feats of self-indulgent hedonism when they were off duty. Thirty years on, as the media buzzes with the news about the arrest in Switzerland of 76-year-old director Roman Polanski, and the revelations about the incestuous relationship between Mamas and Papas star John Phillips and his daughter Mackenzie, that golden era of excess begins to look more than a little shabby.

Of course, there was a large amount of myth-making in the way Seventies Hollywood has been memorialised. The excess wasn't only in the Hollywood players' private lives – it was in the wildly far-fetched stories they and others told about it. If the cocaine and orgies had been as rife as some chrociclers believe, then the decade surely wouldn't have given us films like The Godfather, Chinatown, Five Easy Pieces, The Deerhunter, Cutter's Way, Apocalypse Now, Being There and the other masterpieces that were produced with such apparent ease.

Nevertheless, books like producer Julia Phillips's You'll Never Eat Lunch In This Town Again, Peter Biskind's Easy Riders, Raging Bulls and former studio boss Robert Evans's The Kid Stays In The Picture provided enjoyably lurid accounts of the drug- and sex-fuelled decadence of the time.

Download the new Independent Premium app Sharing the full story, not just the headlines

This was a pre-Aids era. Directors were young and vain enough to think that promiscuity carried few risks and that the drugs wouldn't frazzle their brains. There was a sense of droit de seigneur – the notion that the best film-makers were the equivalents of the old feudal lords who used to have the right to copulate with their vassals' brides on their wedding nights.

Photo-shoots with starlets were just another example of the casting-couch syndrome that had prevailed in Hollywood since the earliest days of the studio system.

In Hollywood, the older man often played the role of Svengali as much as that of exploiter. As Marina Zenovich's excellent documentary Roman Polanski: Wanted and Desired (2008) shows, Polanski's relationship with the teenage Nastassja Kinski, who he met in the mid-1970s and photographed for a magazine, helped set Kinski on the road to stardom. There are many other examples of real-life couples whose story matches that of the characters in George Cukor's A Star is Born. The young starlet outstrips the star who preys on her. Her career blossoms. He drifts into obscurity.

You can hardly blame film-makers for behaving with such recklessness after Hollywood let them in during the 1970s. This year marks the 40th anniversary of the film that helped them break down the barriers, Easy Rider (1969.) Before Peter Fonda and Dennis Hopper burst onto the scene, the old Hollywood system had often seemed hermetically sealed to the outside world. The studios ran along strictly regimented lines. Actors were all under contract. Directors did more or less what they were told.

Read F Scott Fitzgerald's Pat Hobby stories or his "Crazy Sunday", and skim through Kenneth Anger's Hollywood Babylon books, and you will realise that alcohol, sex and drugs existed in Hollywood long before the 1970s. The 1920s had seen the Fatty Arbuckle scandal. In 1921 Roscoe Conkling Arbuckle, the hard-living, boulder-shaped, heavy-boozing comedian who had become hugely popular in silent comedy, was charged with the murder of a young starlet called Virginia Rappe, who died in murky circumstances after a wild party in a hotel in San Francisco. The sensationalist media coverage of the scandal throws into stark relief the idea that muck-raking, celebrity journalism is a new phenomenon.

In Locarno this summer, director William Freidkin (director of The French Connection and The Exorcist) spoke eloquently about the "vast change" that took place in American cinema in the wake of Easy Rider.

"It [Easy Rider] was made for very little money by people who were complete unknowns and it was a great success. It was about the American drug culture. The studios in Hollywood were looking for other young film-makers to make other such films."

Friedkin himself, Francis Ford Coppola, Peter Bogdanovich and Martin Scorsese were among the figures who gatecrashed their way into the new Hollywood. The films that they made were primarily influenced by the work of European film-makers. Audiences in their turn became more and more interested in the "new" Hollywood. Studio bosses may have felt incomprehension and even revulsion at the new Easy Rider subculture but that didn't stop them greenlighting films by young, iconoclastic directors who probably wouldn't have been allowed into their offices a few years before.

Jack Nicholson's story is emblematic of the sudden transformation in the fortunes of many young film-makers and actors. In 1968, Nicholson had been an out-of-work actor living in Harry Dean Stanton's basement and working on a very goofy script with Bob Rafelson for the Monkees vehicle Head. After he played lawyer George Hanson in Easy Rider, he was on his way to becoming an American icon.

Whatever the similarities between the scandals that 1920s and 1970s Hollywood yielded, the two eras were fundamentally different. By the 1970s, the old studio system had collapsed. Moreover as William Friedkin puts it, film-makers were passionately interested in and influenced by the events of the late 1960s and early 1970s.

"America was going through a national nervous breakdown. It started with the assassination of John Kennedy and then the assassination of Martin Luther King, then Robert Kennedy, then the onset of the Vietnam War in which America stumbled very badly and has never really recovered," Friedkin suggested. "The 1960s ended with the Charles Manson murders – the murder of Sharon Tate and a bunch of people for no apparent reason at all by a bunch of drug-infested people who were aimless and sort of adrift from the American culture."

Friedkin, Coppola, Scorsese and Co all emerged in this period, making movies that reflected the upheaval in the society around them. "We were reflecting what we could perceive, which was paranoia everywhere and irrational fear. Certainly, my films of the 1970s reflected just that."

What was happening in the early 1970s Hollywood of The Exorcist and The Godfather had been partially anticipated in the British film industry of the late 1960s. Swinging London had yielded a series of very bleak and unsettling films. Donald Cammell and Nic Roeg's Performance, which had been made in 1968 but not released until two years later, took the ingredients of an East End gangster film and of a 1960s rock-star movie and blended them together to make a phantasmagoric horror film. Whether it was Michelangelo Antonioni's Blowup (1966), Joseph Losey's Accident (1967) or Roman Polanski's febrile psychodramas Repulsion (1965) and Cul-de-sac (1966), the best British-made films of the era invariably had a very dark core.

Just because they made such intense and unsettling films didn't mean that the directors in swinging London or, a few years later in 1970s Hollywood, were going to live austere and monastic lives. Of course, they did quite the reverse. Polish director Jerzy Skolimowski, who met Polanski in film school and co-scripted his first feature Knife in the Water (1962) recently testified how startling it was for young film-makers from the communist-era Soviet bloc to experience life in late 1960s London. Skolimowski came to the city when his film Bariera (1966) was in the London Film Festival. He stayed in the Savoy Hotel. Polkanski took him around him the town.

"Roman was living in Belgravia. I was very impressed with his situation," Skolimowski recalls. "Roman introduced me a little bit to the London scene. He explained to me what was the King's Road." Polanski even took him to the Playboy Club, where he met Victor Lowndes. "It was quite an event for a young man. Suddenly, I was on a dance floor with those bunnies. As a young man, I had a wild life – as Roman did too. We were quite similar in our adventures."

It was the austerity of their backgrounds combined with their rebellious nature and the sheer wealth of temptation on offer that led so many other directors, too, to behave so wildly. There was something epic and even heroic about their self-abuse. There was also a sense of an Oedipal struggle. They often had distrust and contempt for their paymasters, who belonged to a different era in Hollywood, and saw it as a point of honour never to conform or obey.

Much that has been written about the 1970s has been wilfully exaggerated. Friedkin has complained that Easy Riders, Raging Bulls was based on "garbage and dirt" and "the ravings of ex-girlfriends and ex-wives." You're often aware that the biographers and even the protagonists themselves are engaged in a myth-making process. All those stories about Sam Peckinpah's misdeeds or Hal Ashby's drug-taking and erratic behaviour seem juiced up. When Julia Phillips goes to quite such lengths to tell us what was in her system the night she won an Oscar for The Sting, you can't help but ask how she can remember all the pharmaceutical details when she was quite so "high."

British writer Lucretia Stewart recently described how, when she was a schoolgirl in the late 1960s, it was fashionable "not just to admire Nabokov's Lolita... as a work of art but also to embrace it wholeheartedly, to regard it as exemplary." Stewart wrote of teenage British schoolgirls playing the nymphet to predatory, Humbert Humbert-like older men without quite realising what they were doing. This is the theme of Lone Scherfig's excellent and strangely melancholy new film, An Education, based on Lynn Barber's hugely enjoyable childhood memoirs.

It's a long way from the 1970s Hollywood of Polanski, Robert Evans, Jack Nicholson and Co to the early 1960s London suburbs shown in Scherfig's film. What the film captures, though, is the excitement, pathos and ultimately extreme seediness of the precocious schoolgirl's affair with the older, ostensibly much more experienced man. The irony is that the older man (played by Peter Sarsgaard) is really the naive one. Viewers are likely to ask how on Earth he thought he was going to get away with it, as they watch him pick up the teenage girl at the bus-stop. It's the same question that could be asked of the satyr-like 1970s Hollywood figures who often seemed to think that they had their own special licences to transgress.

Ironically, the best book about 1970s Hollywood isn't really to do with sex and drugs at all. David McClintick's Indecent Exposure: A True Story of Hollywood and Wall Street isn't about Jack Nicholson or Roman Polanski. Instead, it tells the grim and utterly compelling story of Columbia Studio boss David Begelman. A former talent agent who had represented Judy Garland, Begelman was an inveterate gambler. He was popular and successful. With him at the helm, Columbia enjoyed big successes with movies like Shampoo and Close Encounters of the Third Kind. Begelman had everything – status, money, power. He was even well-liked. The only problem was that he was a crook of the first order. His long and slow downfall, which ended with his suicide in a Los Angeles hotel in 1995, started when he was caught forging cheques for relatively insignificant amounts and embezzling money from actor Cliff Robertson. When Robertson complained, it was the actor who was vilified. Hollywood closed ranks around Begelman and Robertson was blacklisted.

What was fascinating about the Begelman story, expertly told by Wall Street Journal investigative reporter McClintick, is the way it shows graft and corruption as being endemic in Hollywood. A little embezzlement and forgery are seen as very minor misdemeanours. The Hollywood community reacts with disdain to what it clearly sees as petty score-settling by dreary-minded reporters and disgruntled actors. If a studio boss wants to steal, why shouldn't he? If a film-maker takes a shine to a starlet, that goes with the territory too.

"The occasional embezzlement, fraud, cheating, and chiselling – serious as they are – constitute symptoms of a more pervasive and subtle corruption, a corruption that is more difficult to combat than outright theft," McClintick writes in conclusion. "It is the corruption of power and arrogance... the corruption that inevitably pervades a large and glamorous institution when that institution is tightly controlled by a handful of people, and thousands upon thousands of other people are clamouring for entry."

McClintick is writing about Hollywood of the 1970s but his language is eerily familiar to that found in Polanski's Chinatown (1974), a film with incest and corruption at the core.

In 1970s Hollywood, the battle-lines were more complicated than they first seemed. The Easy Rider generation, disgusted by the Watergate scandal and the Vietnam War, wasn't interested in graft and embezzlement. This seemingly was the province of an older generation. Begelman may have flourished in the "new Hollywood" but his roots harked back to earlier showbusiness times. The wild living of the younger film-makers who came to prominence in the 1970s was, surely, at least partly motivated by their desire to show their disdain for the conservative old-style studio executives and corporate officers.

It was inevitable that the new Hollywood would be short-lived. Peter Biskind's book has a final chapter titled "We Blew It" in reference to a famous line from Easy Rider. The combination of profligate spending and profligate living helped sabotage the Hollywood careers of Coppola, Michael Cimino, Polanski et al. Their hubris was self-evident, whether it was the belief that they could do what they liked on screen, whatever the expense, or behave how they liked off-screen, whatever the cost in broken lives.

We, the public, were complicit too. Roman Polanski: Wanted and Desired makes it very apparent that the abusive excesses of the film-makers of the era were more than matched by the equally seamy behaviour of the media and judicial system. This, in turn, was fuelled by the public's appetite for hearing the dirt. The double-standards are astonishing. As has again become evident in the last week, we love to hear this stuff. We still buy into the idea that 1970s Hollywood was as close as the 20th Century came to Nero or Caligula's Rome. Our disapproval comes tinged with prurient curiosity. Right-wing commentators relish dredging up old misdeeds in all their grisly detail so that they can wax indignant about them all over again. We won't let the fact that a corrupt judge broke his word or that much of the reporting at the time (and since) was wildly inaccurate get in the way. Our minds about 1970s Hollywood were made up a long time ago, and we're not going to change them now.