Zadie Smith spoke recently of the world of literature that her local library opened up for her as a teenager, but according to Tory MP John Redwood it's the middle classes, not disadvantaged children, who are using the UK's libraries.

With almost 500 libraries around the country currently under threat, according to Public Libraries News, Redwood – MP for Wokingham and former candidate for leadership of the Conservative party – paid a visit to a public library, only to find it virtually empty.

"During the half hour I was in or near it I did not see anyone borrow a book. I was the only visitor in the visiting party to go and look at what was on the shelves. There were not that many books on offer. It was predominantly a fiction library. The crime section seemed to be the single biggest themed area," he said on his blog.

The author of books including Singing the Blues, a personal history of the Tory party over the last 30 years, and I Want to Make a Difference, But I Don't Like Politics, Redwood lingered over the library's non-fiction shelves, which he said "seemed oriented to middle-class hobbies like antiques and foreign travel".

"I guess the book buying had been well judged to cater for the demand of a fairly affluent local community that said it wanted a public library," he said. "Some defenders of every public library imply that they are for a different clientele. They conjure images of children from homes living on low incomes developing a passion for reading serious books borrowed from the local library. The library is seen as a force for self improvement and the pursuit of knowledge. I fear that in many cases this is no longer true, if it ever was."

His comments follow Smith's passionate defence of her local library in Kensal Rise late last month, when the author said she could see "that if you went to Eton or Harrow, like so many of the present government, it is hard to see how important it is to have a local library. But then, it's always difficult to explain to people with money what it's like to have very little".

Calling the coalition government's library closure plans "a policy so shameful that they will never live it down", Smith said it was "all very well replacing local libraries with enormous libraries – but for those families for whom getting on a train to visit the British Library is inconceivable, having a local branch 100 yards from your front door can change your life".

Redwood, however, said that since most cities and large towns have secondary school and university libraries as well as public libraries, we need to re-evaluate "how many libraries we need in each community" in this "time of tighter spending controls".

"When it comes to general fiction libraries we need to see how many we need and where they best be located to maximise use whilst keeping down cost," he wrote. "Mobile libraries that bring the books from the big libraries to the public might be one way through, to improve the service at realistic cost."

The twice Carnegie-shortlisted author and library campaigner Alan Gibbons disagreed. "Where do you start? He does a half-hour visit to a library when he's clearly not a library user, and he doesn't understand that 320m visits occur to libraries every year," said Gibbons. "They are hugely used, and when they are not used it tends to be because people don't know when they're open, because of the successive cuts from the 1980s onwards, from politicians like him."

Gibbons was adamant that it is not only the middle classes who are library users. "The vast majority of people use libraries … Public libraries serve nearly everyone apart from incredibly rich people like John Redwood and the culture minister. Everyone else needs a library."