Theatregoers who seek a warm communal chuckle have never been so well served by the West End. Rival comedies are selling out across the capital, from Alan Ayckbourn's revived Absent Friends to Michael Frayn's vintage farce Noises Off, to say nothing of the successful reimagining of the Ealing classic The Ladykillers or the Broadway-bound smash hit One Man, Two Guvnors. And over at the National Theatre, Oliver Goldsmith's She Stoops to Conquer is gavotting its way across the stage of the Olivier, earning big laughs.

But this glut of consoling fare is provoking unease in some quarters. Last week, established playwrights David Hare, Mark Ravenhill and Simon Stephens argued that comfortable, safe theatre was in danger of pushing more challenging new work off the stage for good. In response, Nica Burns, vice-president of the Society of London Theatre, questioned whether Britain really is harbouring such a wealth of undiscovered "avant garde" talent.

"The years have shown the really fine stuff survives," she said, defending the triumph of a tried-and-tested theatrical repertoire, before going on to wonder whether Ravenhill really believed "there's a list of Becketts and Ionescos that are lying unformed and undiscovered because British theatre is not adventurous enough".

So is there a British avant garde cultural movement out there? And, if it does exist, would it ever be visible to the mainstream entertainment industry or to orthodox media?

For Alistair Spalding, artistic director of the Sadler's Wells Theatre, where nude dancers in blonde wigs recently hurled themselves through the auditorium every night, you can recognise any truly innovative art because it prompts some sort of emotional discomfort. "You cannot call a performance or a work of art avant garde unless it gives the audience that uncomfortable, perhaps even painful feeling," he said.

When he staged the premiere of Canadian choreographer Dave St Pierre's show Un peu de tendresse bordel de merde! last year, many audience members walked out; critics too thought the dance venue had gone too far. "Repellently, self-indulgently dreary," wrote Mark Monahan in the Daily Telegraph, after witnessing 20 naked dancers shaking their genitals, and the Observer's dance critic, Luke Jennings, called it "vulgar, witless, repellent", although some reviews did applaud its boldness as a mark of the genuinely avant garde.

Defining the avant garde has never been easy. "The nature of the avant garde is always very much in question," said Will Montgomery, senior lecturer in English at Royal Holloway, University of London. "Some people think it is inherently oppositional, others think it is about constant formal innovation and still others think of it as a historical term for a movement that is now definitively dead."

In popular entertainment, if not in literature, yesterday's avant garde is often tomorrow's mainstream, so the term can function as a label simply identifying the next trend. As the American poet John Ashbery pointed out in an influential 1968 essay on the nature of the avant garde, where once an innovative artist had to wait a whole career to see their work absorbed into mass culture, in the modern age it happens with increasing speed. "Today the avant garde has come full circle – the artist who wants to experiment is again faced with what seems like a dead end, except that instead of creating in a vacuum he is now at the centre of a cheering mob," Ashbery wrote.

Even more confusing is that ideas that once shocked the public are now quickly picked up by commerce and used to sell mainstream entertainment and to advertise products, becoming what the New York art critic Clement Greenberg dubbed as "kitsch" as long ago as 1939.

For Tim Marlow, the art pundit who runs the White Cube art galleries in London, home to Damien Hirst and the Chapman Brothers, the concept of avant-garde art is defunct. "I don't think there has been art that you can call that for 20 years," he said. "The difficulty is that you have to know what you are going against if you are counterculture. And there are so many cultures now."

According to Ravenhill, however, theatre is being starved of fresh writing talent by a new emphasis on commercial success in cash-strapped subsidised theatres, venues where hit shows such as War Horse, The Ladykillers and One Man, Two Guvnors all began life. His own most celebrated work, Shopping and Fucking, also transferred from the subsidised Royal Court to the West End in 1996, but the playwright argues that the grant-receiving sector now pays too much attention to pleasing the crowd.

This weekend his view is supported by the playwright Charlotte Keatley, whose 1987 work My Mother Said I Never Should is one of the most regularly performed British plays. With a highly unconventional new play, Our Father, opening last week at the Watford Palace Theatre, Keatley complains that work such as hers, which often has an unexpected subject matter and an unusual structure, has a much harder journey to the stage.

"There are so few theatres producing new work," she said. "Twenty years ago there may have been 12 or so, now there are about three. Yet people really want to see new work. I am confident about that, despite the raft of literary managers and directors in London who say that you can't do anything too risky."

Keatley's case that the capital has lost its risk-taking culture is supported by the fact that plays by Ravenhill, along with the brutal works of the late Sarah Kane, are more widely performed and appreciated in Germany than they are in their native land.

But what counts as challenging these days? When an actor deliberately catches the eye of an audience member, it can be a nerve-wracking moment. If you are the unhappy patron picked on by James Corden in One Man, Two Guvnors then you are in trouble, though still protected by theatrical convention. If, however, you went to Belgian theatre group Ontroerend Goed's controversial production, The Audience, the Edinburgh Festival hit that transferred to the Soho Theatre in the West End last autumn, it would have been time to start checking the exits. The show started with a brief lecture on the conventions of audience behaviour then launched into a full-scale attack on individual audience members.

A show like this is clearly trying to push the envelope, but other work is harder to categorise. At the commissioning stage, the difference between an innovative show and a commercial hit is less clear. When Nicholas Hytner and Nick Starr at the subsidised National Theatre took a bet on War Horse, it looked like a long shot. Similarly, as the Observer's theatre critic Susannah Clapp points out, the NT's faith in writer Richard Bean's comedic prowess took a while to bear fruit with his updating of Goldoni's One Man, Two Guvnors. Hytner has often underlined his strategy: a hit like War Horse can fund difficult work such as the NT's musical about the murders of the Ipswich prostitutes, London Road.

Clapp, a resolute fan of the site-specific and pop-up theatre that has blossomed in recent years, does not believe in the avant garde as a movement: "I think of innovation in the arts as about particular pinpoints of excitement. I also don't necessarily agree there is some sort of recoil, or retrenchment from innovation because of all the comedy in the West End.

"The National Theatre has had a brilliant era, taking risks such as Jerry Springer: The Opera. If the site-specific moment has gone by, it is partly because the big theatres have taken up their ideas and partly because great companies such as Punchdrunk and Gridiron want to do new things."

The lurking danger is, perhaps, that culture begins to merely ape the shocking moments of previous eras, nostalgically recreating the avant garde just as lovingly as the set of Absent Friends recreates a 1970s living room.