The Booker Best Pony in Show row is an annual event that at least lifts novels off the books pages and into the public debate. This year's fight about readability tempts me to set up a new publishing house, funded by Sir Stelios. EasyBook could recruit the chair of the Booker judges, Stella Rimington, as CEO and offer a no-frills novel-reading experience that goes from A to B and does not tax the brain.

Nothing wrong with that. There are plenty of entertaining reads that are part of the enjoyment of life. That doesn't make them literature. There is a simple test: "Does this writer's capacity for language expand my capacity to think and to feel?"

Subject matter is not the point. It might be socially relevant, or it might not. It might be historical, science fiction, a love story, a crime novel, a meditation in fragments. There is no point judging a novel by its subject matter; what is in vogue now will be out of date soon. Nobody reads Jane Austen because we want her advice on marriage. And we don't care that she lived right through the Napoleonic wars and never mentioned them once. Who cares about the Napoleonic wars now?

Novels that last are language-based novels – the language is not simply a means of telling a story, it is the whole creation of the story. If the language has no power – forget it.

The problem is that a powerful language can be daunting. James Joyce is hard work. Virginia Woolf's The Waves is a very slow read. Schools teach language-friendly versions of Shakespeare.

Ali Smith's There But For The is a wonderful, word-playful novel, ignored by the judges this year because it doesn't fit their idea of "readable". It is better than anything on their list. Why? It expands what language can do and what fiction can do, and when a reader collides with that unruly exuberance, he or she has to shift perspective. That is what literature is supposed to do.

I don't see this row as one about dumbing down though. Rather, it is a misunderstanding about literature and its purpose. We are nervous about anything that seems elitist or inaccessible, and we apologise for the arts in a way that we never do for science.

Nobody blames maths for being difficult – and it isn't difficult – but it is different, and demands some time and effort. It is another kind of language. Literature is also another kind of language. I don't mean literature is obscure or rarefied or precious – that's no test of a book – rather it is operating on a different level to our everyday exchanges of information and conversation.

That's obvious in poetry and we welcome it. In fiction we seem to want a kind of printed television. Why?

The most unreadable books I have read recently were Stephenie Meyer's Twilight series. I did try to read Stella Rimington's own spy series but instead I began to wonder if we would choose an enthusiastic member of a painting-by-numbers club to chair the Turner prize?

The best thing for the Booker prize would be to outline its aims – is it about literature or is it a division of easyBook? Should we include first novels? I think not. Writers need time, and our restless search for novelty is not because we long for genuine new work but because we are easily bored.

I am sorry that the Booker judges were thrilled to be seen as champions of the jolly good read. I wish they had championed the power of literature instead. The so-called literati aren't insular – this from a woman who ran the security service – but we aren't going to apologise for what we believe in either. There is such a thing as art. There is such a thing as literature.

This year's Booker prize isn't about the power of the new – there's no experiment with form or strangeness of imagination. The winner may get on the bedside tables of middle England, but that's not as important as changing the way that even one person dreams.