In Edinburgh, where I live, the fringe and other festivals are turning this great city into a dressing-up box. The street life is heightened, as if everyone's waiting for their cue, their choreographed moment, the flash mob music to begin.

The arts are manifold these days, re-enchanting and rejuvenating cityscapes as best they can, which is great. The post-industrial world needed new reasons to get us out of the house and, when we noticed the stasis of online life, we again wanted to get out.

So I hate to rain on the parade, but there's a problem with this new cultureverse. As a filmmaker, I regularly tour around the major arts venues and cinemas in the UK and abroad and, despite their fighting the good fight and being run by lovely, committed people, too many of them feel exclusive. The taxi driver who drops you off at them has never been inside. They're too narrow, too defined by class.

Here's what I mean. I grew up in working-class Belfast. Despite being brainy – so I was told – I did not go to many of the city's arts venues. Our school took us once to the Queen's Film theatre and once to the Lyric theatre. They were fantastic, but felt foreign – it was like being on safari.

My parents didn't go to such places, which was fine. Their culture was Morecambe and Wise, Elvis, Tammy Wynette, Charley Pride, John Wayne – all great stuff. And I found lots of other things – James Joyce, Paul Cézanne, Echo and the Bunnymen – by myself or in the pages of the NME, the art department of our library and through the inspiration of a great art teacher. Finding them by myself was part of their thrill. As a teenager, I swooned at Cézanne's apples and felt like the first ever to have done so.

But still, some help out in the city would have been handy. The problem is that while the content of the arts venues in Belfast back then might well have been for people like me, their form was against us. By form, I mean their thresholds, architectures, their ways of welcoming – or not – the public and the assumptions that you already knew what they were and how they worked.

Now, most good arts venues have children's programmes and outreach and inclusion policies, and they really want to involve the whole community. But so often, their sleek lines, or facades that look like office buildings, their malbecs and chorizo-studded menus are too culturally thin.

I like malbec and chorizo too, but is it OK to mention the mirthless, un-tactile, cold-to-the-eye aspect of "cool" arts venues? I'm not naming names here because, as I say, these places are working hard, doing good and precipitating new communities, but they need to be broader.

Forgive my generalisations and my bluntness, but there's sometimes a deadness that comes when middle-class people run stuff. They get the content right, the ideas, the themes, the politics. But they haven't a clue about how to embrace things. They don't understand all that's caught in the lovely Danish word "hygge", a sort of cosiness. They don't understand the warmth and feel of buildings, so well described by the writer Gaston Bachelard in his book The Poetics of Space. In my copy of it, I have underlined a bit where he says that the spaces that we love are "especially receptive to becoming" – welcoming places reduced of inhibitions.

Too often, also, they talk about guilty pleasures, which to me alway smacks of people ashamed of their emotions. By contrast, the old movie palaces of the 1920s and 1930s made working-class people, as well as everyone else, feel glamorous, flirty, elsewhere and very much welcome.

I recognise, as I noted, these are generalisations, but there's truth in them. In the film magazine Sight and Sound, recently, I argued that cinema, which started as cheap entertainment, is at its best when it is a hi-lo country, mixing, if you like, working-class and posh elements. The best British films – Terence Davies's Distant Voices, Still Lives, Nic Roeg and Donald Cammell's Performance, This Sporting Life, the movies of Chaplin, Lynne Ramsay, Jonathan Glazer, etc, as well as the work of Martin Scorsese, François Truffaut and many other directors, are great precisely because they are hi-lo.

The same goes for venues. Yes, beam in opera, but have fishfinger sandwiches too. Yes, have Ethiopian goat curry, but screen Strictly too. In the little movie events that Tilda Swinton and I do, such as the Ballerina Ballroom Cinema of Dreams, we have consistently played with hi-lo. We have serious art and ideas, we hope, but use contrasting form, by keeping pricing rock bottom, making cups of tea for people in line, having a dance before the movie starts, etc. Mix up serious art with a children's party or the Grand Ol' Opry in Glasgow or a mosh pit. To define and design our arts venues exclusively with middle-class values is to deprive them of passion, brio, instinct – and noise.

This isn't only a problem in the UK. The social exclusions in Mumbai and New York, Sweden and Paris, São Paolo and Tehran are shameful. (It need not be this way. I'm just back from film-maker Michael Moore's Traverse City film festival in Michigan, which, from the start, had a no red-carpet, no velvet-rope policy, and whose central cinema, the State, still shows some films at 1940s prices – 25c.)

Those of us who crossed over to the "other side", who moved from working-class lives to middle-class worlds, found that Raymond Williams was right, that we were living in a second ghetto. I don't know if it's true for others, but I often feel that I am in drag in arts venues, masquerading, lip-synching to another lifestyle.

Let's not be coy. It's my world now – I've shown my films or curated seasons in such buildings around the globe – and yet it's still not entirely my world. At worst, our arts venues scare people or make them feel stupid or small. Even now, I am often intimidated by them. I remember my teenage trepidation, my worry that such places were not for the likes of me.

We are missing so much by not nicking the best ideas from working-class culture – the texture, the glow – and adding it to how our arts venues work. I am delighted to live in the cultureverse, but it needs to be a multi- not uni- verse. It needs to be the cake, not just the icing. It needs to mix its metaphors, as I've just done. It needs to make sure that arts venues aren't just places that taxi drivers drop people off at. Right, I'm off to Edinburgh's Summerhall, one of the world's great arts venues. But I'm having my beans on toast before I go.

Mark Cousins is a film director and writer. His most recent work is the documentary A Story of Children and Film