Where do TV comedy hits come from? Nobody knows, of course. If telly's top brass knew the answer to that, we'd have been spared David Jason in The Royal Bodyguard, or Amanda Holden under her Big Top. But, until recently, some things were taken for granted: the networks' next big hit was unlikely to be discovered playing to an audience of elderly women at the Glasgow Pavilion. It was unlikely to feature a 57-year-old man in a frock making jokes about rectal thermometers, and – in the event that it did – the cast surely wouldn't solely comprise that man's extended family and close friends. Oh, and its star wouldn't cite as his main influence the 70s double-act Cannon and Ball.

Which just goes to show, nobody knows nuffink. Mrs Brown's Boys – which the quick-on-the-uptake among you will have recognised from the above description – is the biggest sitcom phenomenon in yonks. There was a fanfare over Christmas for the Doctor Who festive special, for Miranda and Downton Abbey – all popular shows, all talked up in the media. But the most-watched comedy or drama of the Christmas period wasn't any of those – it was the unheralded Irish family sitcom Mrs Brown's Boys, with its Christmas Eve audience of 11.7m. Its third series is currently outperforming Miranda on Monday nights on BBC1.

I should qualify that word "unheralded". Mrs Brown's Boys has the highest AI (audience appreciation) ratings of any BBC comedy. It has won UK and Scottish Bafta awards and last week's bagged the National Television Award for Best Sitcom (voted for by viewers). But the critics despise it. "A 'comedy' hewn from the dark materials that spewed forth [notoriously chauvinist 70s sitcoms] Love Thy Neighbour and Bless This House," growled Metro. "Lazy, end-of-pier trash … the worst thing [I've] ever seen," thundered the Daily Record, while the Independent crowned it "the worst comedy ever made ... It is Two Pints of Lager and a Packet of Crisps, but more than 10 times as crass and not as cerebral." Critical and popular opinion don't always converge, but the gulf between them rarely yawns so wide, or sloshes full of so much bile.

So should we be welcoming Agnes Brown into the pantheon of comedy greats, or changing the pantheon's locks and pretending there's nobody home? And what does the series' success tell us about 21st-century comedy? Brendan O'Carroll himself, speaking by phone as he passes through airport security the morning after the National TV awards, isn't giving much away. A moustachioed Barry McGuigan lookalike when he's not frumped up as Mrs B, he finds his recent success "amazing", he says. "To not be able to get from the check-in to the aeroplane without stopping for 40 photographs is really weird," he says. "I don't even think my mother if she was alive would want a fucking photograph of me." But he's not – at least initially – forthcoming with explanations. "Brian, do me a favour," he says. "Analyse this and tell me what the secret is, 'cos I don't have a fucking clue."

O'Carroll's story – which is also the story of Mrs Brown's Boys – is a remarkable one. As everyone who works on the show will tell you, key to its success is that "it arrived fully formed", as director Ben Kellett puts it. "The characters knew each other, the storylines were there, the gags were there. It was all right there." That's because O'Carroll had worked on the character for 20 years. Mrs Brown began life as a radio series for RTE in Ireland. O'Carroll – so goes the legend – had no intention of playing Agnes himself, until the actress he'd hired didn't show up to the first recording. On radio, this working-class Dublin matriarch was an instant hit – and yet O'Carroll considered her a sideline to his standup career. The 11th of 11 children of the pioneering Irish parliamentarian Maureen O'Carroll – the ur-Mrs Brown – young Brendan grew up poor in 1960s Dublin, left school at 12 and worked as a waiter before catching the performing bug at the late age of 35.

High on the success of Mrs Brown – by now heroine of a series of novels – in 1998 O'Carroll borrowed a fortune to fund his 1998 movie Sparrow's Trap. Made but never released, the film left him £2.2m in debt. To pay it off, he accepted an offer to bring his well-loved alter ego to the stage. The first Mrs Brown play ran for 16 weeks at Dublin's Gaiety Theatre, and was a sell-out hit. O'Carroll has toured with the play and its successors ever since. Bizarrely, Mrs Brown was also the subject of a Hollywood movie, Agnes Browne, starring, and directed by, Anjelica Huston. The film was adapted from O'Carroll's novel The Mammy; Huston's thoughts on her character's later transformation into a man-in-a-dress comedy favourite are not, alas, recorded.

It was in the theatre – the Glasgow Pavilion – that BBC producer Stephen McCrum discovered Mrs Brown's Boys, three years ago. On the recommendation of friends, including Rab C Nesbitt creator Ian Pattison, "I went along to see the show," says McCrum. "And it was very, very funny. I don't believe in God, but I nearly did that night. The audience was full of 200 old women laughing, alongside ushers who were about 16 or 17 and also pissing themselves. It was immediately clear: there's something happening here." By now, O'Carroll and his troupe had been honing their broad stage comedies for over a decade. "The touring O'Carrolls", as Kellett calls them, included O'Carroll's sister, his son and daughter, his daughter-in-law, his son's best friend, and three pals of more than 20 years' standing. All are still in the cast of the TV sitcom, as is O'Carroll's wife Jennifer who, in an arrangement to make Krankies-phobes shudder, plays his fictional daughter Cathy.

O'Carroll's collaborators on Mrs Brown's Boys cite this unfeigned family atmosphere as integral to the sitcom's success. "It must be one of the few programmes," says director Ben Kellett, "on which, every week, the first 10 minutes of rehearsal is given over to hugging. The entire cast comes on and hugs the entire crew. Every single person who works on that show knows all of the cast's names, and the cast knows all of the crew. And that is a very unusual thing." That camaraderie is certainly conspicuous – if only because Mrs Brown's Boys leaves in footage (of the actors making each other laugh, winding each other up) that most shows would dispatch to the cutting-room floor.

This is a sitcom, then, that keeps the outtakes in. Its star will scurry between stage sets to retrieve a misplaced prop, and – as he did in the first broadcast episode – puncture a sympathetic moment by turning to the studio audience and announcing: "It's just a man in a fucking dress." (Old-fashioned Mrs Brown's Boys may be; genteel, it isn't.) Its fans cite the show's formal playfulness as defence against the charge of regressiveness. What all this convention-busting is really about, though, is retaining the show's theatrical quality. The best thing about it is its liveness – which O'Carroll was anxious to preserve when it switched to TV. That wasn't easy to do, says director Ben Kellett. "Studio sitcom is usually very prescriptive: stand there, wait for that beat, don't move there. But we're now able to achieve what technically would have been very difficult before, [which is] harnessing the loose ad-libbing energy of this type of performance, and of scenes that are written at theatrical length."

But to Kellett – like everyone else I speak to at the BBC – the real secret of the show's success is O'Carroll himself. "He has a very good knowledge of comedy," says the director. "He's massively self-critical. He's hugely driven. He's got incredible energy. He's naturally and emotionally intelligent as well as being comedically intelligent. It's a great range of gifts." McCrum compares him to Leonard Rossiter: "To have that control of an audience, it's extraordinary." Others reference Buster Keaton. O'Carroll, by contrast, names Cannon and Ball and Are You Being Served? among his influences, which will make more sense to the army of Mrs Brown-sceptics quick to rubbish the show's claims to quality. "It's predictable, formulaic trash," wrote one commentator on the Guardian's website last week. Others have called Mrs Brown's Boys racist (for its cliched portrayal of homely Oirishness), sexist (the gags are frequently gynaecological) and homophobic (Mrs Brown's son Rory is camper than a Graham Norton Christmas special).

Judging by the half dozen episodes I've watched, those charges are hard to make stick. Yes, it's crude – but generous-spirited, and liable to upset only the extremely sensitive, or extremely humourless. Harder to refute is the allegation that O'Carroll uses a lot of second-hand jokes – but then, disarmingly, he doesn't even try. "I have absolutely no problem recycling old gags," he tells me. "The old is new if it hasn't been seen for a long time. And y'know, every time we have a new child, we tell them the same fairytales we were told when we were kids. And they sound great!" You can tell O'Carroll till you're hoarse that Mrs Brown's Boys is backwards-looking, full of whiskery gags and a bit coarse in its stereotyping. He professes to care not a jot. "Who gives a fuck?" he says. "It is what it is. There are people who will love it, and people who won't."

The people who love it are, says O'Carroll, "the audience that comedy forgot". Alternative comedy – the punk-comedy revolution of the 1980s – steered mainstream humour away from the working men's club and towards the student union. Mrs Brown's Boys is taking it back again. "Somebody at the BBC read in a magazine," says O'Carroll, "that comedy is the new rock'n'roll. And they actually believed that, and started pitching it only to the 18-25-year-old market. And left the rest behind." The show makes people angry, because its success suggests that the battles alternative comedy fought – aesthetic battles, political battles – were in vain. Those of us who want comedy to be innovative, provocative and progressive aren't going to like Mrs Brown's Boys. But millions respond to its old jokes and its sentimentality, to the rudimentary subversiveness of a Les Dawson-ish man in drag with a gleam in his eye, and to the show's palpable nostalgia for a (non-existent?) time when large families lived cheek by jowl and we all strolled freely in and out of one another's houses.

"It's got a confidence in itself, and a joy," says Kellett, "and a harmless revelling in the downright silly" – which even its detractors would grant. It's also here to stay: last week, O'Carroll hinted at a fourth series, which the BBC will eagerly snap up. The broadcaster has commissioned a gameshow featuring the Mrs Brown character, while a movie is also in the pipeline. "It just works," says Stephen McCrum, the man who found God when he found Agnes Brown in a Glasgow theatre. "People want a good solid laugh, and they want reassurance that everything's all right. Brendan delivers emotion, comedy, great performances, and he delivers a nice world to be in on a Monday night when you might be a bit fed up at the start of the week. You can criticise that as much as you like," he says – and many do, and will keep doing, furiously – "but it works."