I first saw Jackson's Way in 2003. The actor and theatre-maker Will Adamsdale played a spoof life coach called Chris John Jackson; the show was a half-silly, half-profound seminar on the value of pointless behaviour. (Trying to put your hand in two places at once, trying to rhyme non-rhyming words, and so on.) The show began life as a work-in-progress, then the following year, Adamsdale took it to the Edinburgh fringe. It played seven performances, word spread, its run was extended, and then it walked off with the most coveted prize in world comedy, the Perrier.

The strange thing is that – though we all laughed like drains – no one really thought of Jackson's Way as comedy, for all that it was later billed as such in the fringe programme ("I thought that'd be mischievous," says Adamsdale now). It was just really entertaining theatre. Wasn't it?

Now I'm not so sure. I've been watching and writing about comedy for more than a decade, and have often found that the best standup is the most theatrical, finding ways to make us laugh beyond the usual motormouth-at-a-microphone formula. And things are moving in the other direction, too. As a theatregoer – and more recently, as director of Camden People's Theatre in north London – I've noticed how much of the best theatre is starting to hijack standup techniques, seeking a more direct engagement with audiences. The two artforms are no longer as separate as they once seemed, in other words, and fascinating things are happening in the overlap. That's why, this week, we're launching a new festival, Beyond the Joke, exploring the fertile area where theatre and standup meet.

What we weren't interested in was straight plays with standups in the cast. This, too, is a booming subsection of theatre: it was recently reported that last year's five most popular West End shows had comedy stars in their casts (Tim Minchin in Jesus Christ Superstar, James Corden in One Man, Two Guvnors, etc).

Of course, it's not news that comedians can act. What's less familiar is the blurring of the distinctions between standup and theatre, as acts such as Adamsdale and Daniel Kitson flit between the two, fast-rising standups like Tom Basden and Sara Pascoe choose to develop their own theatre shows, and theatre-makers including Doctor Brown (2012's winner of the successor to the Perrier) become the hottest properties in comedy.

Comedy itself is becoming a popular subject for dramatic inquiry – perhaps because it's now so ubiquitous, from stadiums to bestselling DVDs. Our festival is headlined by an electrifying new show from the solo theatremaker and rookie standup Rachel Mars, called The Way You Tell Them. Mars created it to explore her compulsion to be funny, both on stage and off, and gradually realised that humour was, for her, an avoidance of responsibility, a fear of appearing earnest. For part of the show, she wears a wolf costume, as if daring the audience to take her too seriously. It isn't just herself she's examining, she explains: the question applies to wider culture. "What does the fact that there's loads of comedy everywhere mean about us as a society?" Mars asks. "What are we hiding from?"

Another solo artist appearing at Beyond the Joke is Daniel Bye, whose Edinburgh 2012 hit The Price of Everything is influenced by standups such as Kitson and Stewart Lee. But Bye doesn't bill his show as comedy, he says, because the laughs-per-minute count isn't high enough. "If people came expecting comedy, the fact that it addresses something meaningful might be actively distressing." That's not to say this performance lecture about capitalism and its constraints isn't playful – it culminates in a tongue-in-cheek suggestion that eBay could help solve the arts funding crisis.

These young theatremakers are attracted to standup's liveness, looseness and immediacy. "It has a way of meeting the audience, a directness of address, that I don't see in theatre," says Bye. Mars agrees. "Standup offers instant approval – or disapproval."

But comedy has its limitations, too – the most obvious being that you have to make people laugh. Pascoe is one of the UK's best young standups, a veteran of BBC1's Live at the Apollo and of sitcoms including Twenty Twelve. However, comedy won't allow her to explore or express the range of feeling she is trying to capture in a solo theatre show, Emily's Very Sad Play, which flirts with the boundaries between sober reality and colourful fiction.

Plenty of live comedy acts do dare to deviate: in any Tim Minchin set, several of the best songs are the heartfelt, non-comedic ones. But standup is still limiting, says Mars. "If you're saying something intelligent [in standup] but it's not funny – or not funny enough to make a group of people laugh – you've failed." Theatre is a safer bet if you want to put the laughs on hold – which is why Kitson has taken to creating scripted solo shows that he performs in theatres rather than presenting all his work as standup.

All of which raises an interesting question: how on earth do you decide which is which? Over the years, I've seen many standup-type shows marketed as theatre, and even more comic theatre shows, including Jackson's Way or the early Mighty Boosh shows, branded as "comedy" in brochures for commercial reasons. Bye tells a story about being buttonholed at last year's fringe by a producer asking him for his TV ideas. Bye didn't have any, and the producer, confused, wandered off. Is that the main difference between 21st-century standups and theatremakers? That the former want to be on TV, the latter want to make art? "Stewart Lee says that now comedy is so corporate, it's no longer attractive to non-mainstream people," says Bye. "Maybe theatre will benefit from that."

In fact, of course, there are lots of "non-mainstream" people in comedy, and abundant creative ambition, too – just as in theatre. "I've stopped recognising the boundaries between [standup and theatre]," says Adamsdale. "It's all just storytelling, and it's only because it's good for business that they've been separated."

Adamsdale also dismisses the argument that comedy is obliged to make its audience laugh. "No one would mind if a comedian suddenly whipped out a cello and played for half an hour – if it was done well. Not many people would say, 'I want my money back.' The only rule is: keep people in their seats and stop them being bored."

Maybe that's too idealistic – but surely we're underestimating audiences if we claim it is they who are forcing comedy to be nothing but funny. Our Beyond the Joke event aims to short-circuit all that, by presenting work as neither theatre nor standup, but both – or, if you prefer, neither.

"As far as I'm concerned," says Mars, "the best theatre and the best standup show is exactly the same. It's a show that says something really interesting, and also guarantees big laughs."

And if we want to pigeonhole that – well, the joke's on us.