Hollywood stars flock to causes. An A-list name can boost the profile of a charity, highlight a far-off tragedy or reverse a grave injustice. So when Oscar-winning director Roman Polanski was arrested in Switzerland on the way to a film festival, it is perhaps no wonder that the great and the good of the film world rushed to plead for his freedom.

The list of supporters giving Polanski their impassioned support read like a Who's Who of the cream of the movie-making world. It included, among many others, Woody Allen, Martin Scorsese, David Lynch, Harvey Weinstein, Pedro Almodóvar and Ethan Coen.

But rather than rallying mass public support for the beleaguered film-maker – director of such undoubted classics as Chinatown, Rosemary's Baby and The Pianist – they have provoked an extraordinary backlash.

Led by a handful of outspoken female voices, a rising tide of opinion has instead applauded Polanski's arrest for unlawful sex with a 13-year-old back in 1977. They have turned the focus on the crime itself, calling the director an accused rapist who abused a child.

That, they say, should be the focus of the story and of Hollywood's ire, not defending an old man who pleaded guilty to unlawful sex with a young girl then went on the run for 32 years to avoid prison. The backlash – not only against Polanski, but also against the Hollywood clan that rallied round him – has begun in earnest.

"Roman Polanski raped a child. Let's just start right there, because that's the detail that tends to get neglected," wrote feminist and author Kate Harding in an impassioned column in Salon. That article then went viral across the internet, gaining tens of thousands of page views and seeming to herald the reaction to come.

Harding, a liberal feminist, found herself being asked to appear on rightwing talk radio shows. Soon editorial after editorial, from the mighty New York Times to the smalltown Lowell Sun in Massachusetts, followed suit, welcoming Polanski's arrest as a case of long overdue justice for a serious crime.

It seems that the consequences of the dramatic development could now spread wider than just Polanski. Already some early supporters of the director, such as actress Whoopee Goldberg, have had to backtrack and clarify their positions. More are likely to follow suit in the weeks to come. Could it be that Hollywood – whose very existence rests on accurately predicting the public's taste – has made one of its gravest misjudgments? "The disconnect between Hollywood and the rest of the country seems enormous," said Anthony Mora, an author and founder of a leading Los Angeles-based public relations firm.

There is little doubt that the case is extremely complex. In many ways both sides are dealing in black and whites and not the shades of grey that too often more accurately describe reality.

For Polanski's defenders, that has meant ignoring the act that took place in 1977 and instead focusing on judicial wrongdoings that have plagued the case and Polanski's own tragedy-tinged life.

They point out that the director pleaded guilty only as part of a deal, which he then feared was being reneged upon. That is why he fled, they say. They also refer to his past – as a Holocaust survivor and a man whose wife, Sharon Tate, was brutally murdered by followers of Charles Manson – as evidence that he has already borne much suffering in his life.

Finally, his sterling record as a film director is held up as evidence of why he should be celebrated as a leading artist, not arrested for a crime where even the victim has asked for him not to be pursued after such a long time. Perhaps it is no wonder that many in Hollywood have described his plight in terms that make Polanski himself the martyr. Weinstein said the arrest was a "terrible situation". Actress Debra Winger said the Swiss had been involved in "Philistine collusion" in allowing the arrest. Goldberg, in now notorious remarks, said: "I don't believe it was 'rape-rape'."

But, as the outrage has grown, especially in the wake of Goldberg's remarks, the sheer scale of Hollywood's misjudgment in rallying so enthusiastically to Polanski's cause has begun to be exposed. One of Goldberg's fellow presenters on the ABC TV show The View, Sherri Shepherd, condemned Polanksi outright. Details of the victim's testimony in 1977 have been published and widely circulated through the media and via the gossip website The Smoking Gun. It makes for grim and unpleasant reading.

The girl graphically described being given champagne and a quaalude, a popular recreational drug in the 1970s, by Polanski before he had sex with her. She testified that she repeatedly said no but that he did not stop, committing numerous sexual acts as she protested.

Not surprisingly, it is feminists and women who have led the charge against Hollywood's support of Polanski. The Feminist Majority Foundation is in favour of his extradition. Katie Buckland, chief executive of the California Women's Law Centre, has pointed out the difference between Hollywood's attitudes towards Polanski's long-ago crime and the unearthed pasts of elderly paedophile Catholic priests.

Writer Vicki Iovine has also been outspoken, making the same point. Even some women members of Hollywood have broken ranks as actress Kirstie Alley loudly condemned Polanski and those who defended him. Nearly all have accused him in no uncertain terms of being a child rapist.

The ramifications of that will be difficult to measure. Polanski now faces a long legal battle that will span two continents. But in the arena of public opinion his image has been shattered. The words many people will now first associate with Polanski will be all to do with the sexual assault of a young child, not his film work. Even if he goes free, Polanski could now be hurt where it really matters to Hollywood: the box office. "Sex with children was, and always has been, anathema to Americans... the 'anything goes' cultural excesses of the time do not excuse Polanski from society's expectation that adults should protect kids, not exploit them," said author and sociologist BJ Gallagher.

The Polanski backlash has spread far and wide. He was never popular at all on the right wing of America's culture, but now middle America is firmly in favour of seeing him in a Californian courtroom. Talkshow hosts, radio commentators and newspaper editorials from coast to coast have all insisted that the arrest was long overdue and that Polanski needs to be brought to the US.

"Hollywood people really don't see the world in the same way as average people... that is why there is a backlash," said Mike Levine, a Hollywood PR expert.

But it is perhaps no surprise that the gap between Hollywood and the rest of America has grown so large on this particular case. Because of his long and illustrious career, Polanski is a friend and colleague of nearly all the main players in the film world. They are his confidantes and his peers. His movies have made them stars and helped them to earn millions. They live in the same rarefied world of global fame. "Elite Hollywood culture is protecting one of its own," said Alexander Riley, a professor of sociology at Bucknell University.

It is also speaks to a certain type of Hollywood culture which appears to insist that its top stars are in some ways elevated above the law and should be treated differently to ordinary members of the public.

If Polanski was just an ordinary man instead of a world-famous film director, the bare facts of his case would be likely to elicit little sympathy – especially from the world famous. Hollywood stars seem to be arguing, in some ways, that Polanski's talent should allow him some sort of free pass for his past behaviour. "Hollywood... looks at the Polanski case and says, 'You have to make allowances for genius'," said Gallagher.

Hollywood's elite also functions as a kind of club and Polanski, seen by the elite as a great European auteur director, is a firm member. That requires a certain degree of success but also a great deal of ideological conformity. It is a cliche that Hollywood is uniformly liberal in its politics, but one with more than a dash of truth in it. It is certainly interesting to see the reaction to Polanski's case and compare it with the reaction to Mel Gibson, when he was caught mouthing drunken anti-Semitic abuse.

Gibson, a rare conservative in Hollywood, was brutally condemned by his fellow stars and sent into virtual career exile. Polanski, whose crime is far more serious, has seen a vast outpouring of sympathy. Being a member of the Hollywood club certainly seems to have its privileges.

"The difference between the reaction to Gibson and the reaction to Polanski has been just huge. Huge!" said celebrity interviewer Gayl Murphy. "That says a lot about what Hollywood thinks is important to them."

But, more importantly, it has also exposed a huge fault line between what Hollywood thinks of itself and what Americans think of Hollywood. No longer is it just the right wing of America lambasting "Hollywood liberals" for their permissive and overly Democratic ways. It is Democrats too. And feminists. And conservatives. Polanski seems to have united the different strands of America in a way that few other things have.

As Harding blogged after her column exploded across the blogosphere and she was inundated with emails and requests for press interviews: "Who knew being disgusted with Roman Polanski would turn out to be the ever-elusive common ground between rightwing dudes and liberal feminists?"