Julian Barnes finally won the literary prize that has eluded him on three previous occasions when he was tonight presented with the Man Booker prize for his short novel, The Sense of an Ending.

His victory came after one of the most bitter and vituperative run-ups to the prize in living memory - not among the shortlisted writers, but from dismayed and bemused commentators who accused judges of putting populism above genuine quality.

But few of those critics could claim Barnes' novel is not of the highest quality. The chair of this year's judges, former MI5 director general Stella Rimington, said it had "the markings of a classic of English Literature. It is exquisitely written, subtly plotted and reveals new depths with each reading."

Much of the row over the shortlist has stemmed from Rimington's own prioritisation of "readability" in the judging criteria. But tonight, she said quality had always been just as important.

"It is a very readable book, if I may use that word, but readable not only once but twice and even three times," she said. "It is incredibly concentrated. Crammed into this short space is a great deal of information which you don't get out of a first read."

Accepting the prize, Barnes thanked the judges for their wisdom and the sponsors for their cheque. He also offered some advice to publishers: "Those of you who have seen my book, whatever you think of its contents, will probably agree it is a beautiful object. And if the physical book, as we've come to call it, is to resist the challenge of the ebook, it has to look like something worth buying, worth keeping."

Afterwards Barnes admitted a sense of relief at finally winning. "I didn't want to go to my grave and get a Beryl," he said referring to Bainbridge, who was shortlisted five times, never won and received a posthumous Best of Beryl Booker prize.

He said the "readability" row had been "a false hare" to which he had paid little attention, adding: "Most great books are readable. Any shortlist of the last ten years that I've read has contained nothing but what you would call readable books."

Barnes once called the prize "posh bingo" and he said he had not changed his view – it simply depended on who the judges were and what they liked. "The Booker prize has a tendency to drive people a bit mad," he said, not least writers with "hope and lust and greed and expectation" so the best way to stay sane, he said, was by treating it as a lottery until you win "when you realise that the judges are the wisest heads in literary Christendom". Asked what he would spend the £50,000 prize money on he said a new watch strap was first on his list. "I could buy a whole new watch."

The book, at 150 pages, is undoubtedly short, but not the shortest to ever win the prize – that record belongs to Penelope Fitzgerald's Offshore, which won in 1979 and is shorter by a few hundred words.

The Sense of an Ending, Barnes' 11th novel, explores memory: how fuzzy it can be and how we amend the past to suit our own wellbeing. It tells the story through the apparently insignificant and dull life of arts administrator Tony Webster.

"One of the things that the book does is talk about the human kind," said Rimington. "None of us really knows who we are. We present ourselves in all sorts of ways, but maybe the ways we present ourselves are not how we really are."

Rimington said the question of whether Barnes was overdue to win the £50,000 prize never entered her mind or figured in the debate. "We really were, and I know you find it very boring of me to say so, looking at the books that we had in front of us," she said.

Despite the sometimes hostile reaction to the shortlist, Rimington said she had enjoyed the process of judging. "I've been through many crises at one time or another in which this one pales, I must say. We've been very interested by the discussion. We've followed it sometimes with great glee and amusement. The fact that it has been in the headlines is very gratifying."

It took the judges (Rimington, MP Chris Mullin, author Susan Hill, the Daily Telegraph's head of books Gaby Wood and the former Spectator editor Matthew d'Ancona) just 31 minutes to decide on the winner, after what Rimington called "an interesting debate." They had been divided 3-2 at the beginning of the judging meeting, but were all agreed by the end.

"There was no blood on the carpet, nobody went off in a huff and we all ended up firm friends and happy with the result," she said.

Barnes, 65, had been shortlisted for the prize three times previously; in 1984 with Flaubert's Parrot, when he lost out to Anita Brookner; win 1998 with England, England, losing to Ian McEwan; and with Arthur & George in 2005, when he lost to John Banville.

What was particularly striking this year was that Barnes was the only seriously big hitter on the shortlist, and the only author to have been shortlisted previously.

The others on the shortlist were Carol Birch for her much-admired Jamrach's Menagerie, a historical high seas adventure; two Canadian writers - Patrick deWitt for The Sisters Brothers, a picaresque western, and Esi Edugyan for Half Blood Blues, which mixes the raw beauty of jazz and the terror of Nazism; and two debut novels – Stephen Kelman for Pigeon English, which tells the story of a Ghanaian boy who turns detective on a south London housing estate; and AD Miller for Snowdrops, a Moscow-set tale of corruption and moral decline.

The shortlist undoubtedly prompted a livelier debate about what makes a great novel with many commentators annoyed by judge Chris Mullin's belief that a book had to "zip along" to be worthy of being considered. Last year's Booker chairman Andrew Motion also weighed in, accusing the judges of creating a "false divde" between what is high end and what is readable, and questioning the absence of authors such as Alan Hollinghurst, Edward St Aubyn and Ali Smith.

The row has also led to a group of writers, publishers and agents announcing plans to set up a rival literary prize that would reward the artistic achievement of a writer above 'readability.'

Full details of the Literature Prize have yet to be announced but the agent Andrew Kidd said they felt "a space has opened for a new prize which is unequivocally about excellence." However, not everyone condemned the shortlist. Book sellers, in particular, were happy with a list that resulted in record Booker sales. A spokesman for Waterstones, Jon Howell, called the critical reaction "ungracious sniping" and said Barnes was a worthy winner.

If anyone is upset at the win, it may well be the bookies. William Hill said more than half of all bets had been for Barnes, a 6/4 favourite.

• This article was amended on 19 October 2011. The original described Matthew d'Ancona as editor of the Spectator. This has been corrected to former editor.