What's the point of literary festivals, I used to think. What happens at them? At film festivals, you watch films; at beer festivals, you drink beers. Does that mean literary festivals involve a lot of reading? That doesn't sound very festive. It sounds like a library. And the sounds of a library, if it's well run, are slight.

I certainly wasn't going to get intrigued about them. I made that mistake with the Dublin Writers Museum. "Interesting," I thought. "A museum about all those brilliant Irish writers! I long to discover how the creative team behind this famous museum has prevented it from being a dry collection of black and white photos, scraps of spidery handwriting and the odd typewriter in a glass case." Bad creative team! There wasn't even a video.

I assumed literary festivals would just involve authors being asked questions about their books by a combination of people who'll never read them and people who'll definitely read them and are having the experience tarnished in advance by a from-the-horse's-mouth spoiler session in a tent.

But I was wrong: they're a brilliant system for extorting gaffes from celebrities – the broadsheet equivalent of a gaggle of paps outside a nightclub. You simply take a darling of the middle classes who's plugging a book, give them free wine, push them into a marquee crammed with fans and the imprudent remarks start flowing into the notebooks of the hovering hacks.

At this year's Cheltenham literary festival, it happened to Stephen Sondheim, who slagged off Noel Coward and WS Gilbert, Susan Hill, who gave a rocket to second-hand book dealers, and Andrew Marr who said bloggers were: "socially inadequate, pimpled, single, slightly seedy, bald, cauliflower-nosed young men sitting in their mother's basements and ranting". You can sense Marr being egged on by the bookish audience, using him to extort their collective revenge against the baffling computer age, which has stolen their children's souls, consistently delivered the wrong shopping and caused them to spend all of last Tuesday on the phone to Dell.

Last year, at the Hay festival, it happened to Jeremy Paxman who called the British public "barbarians" for watching TV instead of going to art galleries. Good job for him that they do. But again he's playing to the crowd: it's a mantra of the bourgeoisie that people watch too much TV and hearing the familiar admonishment, from the fleshy mouth of someone they admire from their shameful forays into that medium, is probably the nearest some of them get to a kinky thrill.

On one level, the poor book-promoting celebs are only showing the audience a good time and get widely quoted out of context saying something they don't necessarily think. But another way of looking at it is that our society has become so self-censoring, cautious and judgmental that it takes a thousand expectant sycophants and cosy boozy literary hospitality to get public figures to express what they really feel.

Because, while I doubt that Marr thinks all bloggers are angry losers or Paxman that TV is without merit, I'm convinced that they both sincerely feel something along those lines. As an old-school journalist, Marr rails against the anarchic novelty of the internet which is undermining the very greasy pole he's just climbed. And intelligent, serious Jeremy Paxman must sometimes resent the medium which has given him mass appeal at the expense of intellectual credibility. His considered opinion about television may be more nuanced but he's still kicking the thing he wants to kick.

Maybe there should be more of this emotional honesty (believe me, this is not my usual line). Most of us are pissed off with how frightening and annoying life can be and it doesn't help that we have to pretend not to be. I think it stems from smiling in photographs – I don't know who came up with that idea. The Victorians were content to scowl or look rustically confused but, with the advent of shorter exposures, came the convention of looking happy. No matter what arguments or misery are the reality behind a family holiday, Christmas gathering or office party, we must present a united face of joy to the outside world and posterity: if they think we're happy, maybe we will become so. We cannot have outsiders smugly believing that they are happier than us.

I'm not defending those who, aware of the smiling convention, look moodily into the lens as if posing for an album cover. The convention exists and should be observed by anyone not wishing to seem like a dick. But a different convention would be more honest. God knows, I don't want photos to become an opportunity for people to show their inner pain, but maybe a neutral face would better express our collective struggle through this vale of tears.

Celebrities present a blissful image to the world, talking occasionally about miseries they've been through but only in order to proclaim how happy they are now. Politicians pretend that, unlike in any other job on earth, their day-to-day dealings with colleagues consist purely of constructive unanimity or platitudinous and mutually admiring debate. And the press claim an investigative coup when it inevitably finds these assertions to be false.

Meanwhile society's suppressed rage explodes online. In the anonymous shelter of comment sections, the brain's dominance over the spleen is overturned. Millions of words of insult and conspiracy theory get generated hourly. The opinions are nonsense but the anger, the disbelief in the placid smile of public life, is sincere.

The confidence to be outwardly stroppy comes with age. Look at Elton John. Last week, he disparaged TV talent shows' musical output saying: "the only way to sustain a career is to pay your dues in small, shit clubs", said his own success didn't come "until we'd had six years of hard graft and disappointment", and insisted that: "It's an outrage that Bruce Forsyth hasn't been knighted." He might as well have added: "Some of them don't know they're born" and "It was very different in my day". And it was: in his day, he was a gay man married to a woman. I'm sure Sir Elton would accept intellectually that some things have improved since then. But he's got the emotional honesty to assert that, to a 63-year-old man, that's just not how it feels.

I agree with him, intellectually, that Bruce Forsyth has had a remarkable career. It's taboo to think otherwise. But is that how I feel? You'd have to ask me at Hay-on-Wye.