Almost 4 million adults never read books for pleasure, according to new research, with lack of time one of the most-cited reasons for not reading.

Men are more likely than women to avoid picking up a book, with 11% of men and 5% of women surveyed saying they never read for pleasure. A quarter of the UK's adult population – more than 12 million people – had picked up a book to read for enjoyment less than twice in the past six months.

The YouGov survey was carried out on a sample of 2,059 adults in January. Shortness of time for reading was a problem for 29% of respondents, 40% preferred to do something else and 26% said they "didn't enjoy reading very much".

"Reading can bring so much to your life, from giving you something to discuss with friends to helping you relax or taking you to another world, so it's sad to think that one in four people are in such a reading rut that they hardly ever pick up a book," said Cathy Rentzenbrink from Quick Reads, which has launched a collection of £1 "bite-size books" by authors including Andy McNab and Kathy Lette to encourage more people to read. "This is not polling people who are illiterate – it's a representative sample. And on top of that, the recognised statistic is that one in six adults of working age in the UK struggle with literacy. When you are a reader, it's quite difficult to get your head around the fact that someone might not find reading easy, that you can get to adulthood and not read very well."

The UK figures, however, compare favourably with recent statistics from the US, where Pew Internet found 25% of Americans over the age of 16 had not read a book in the last year.

"Possibly things have improved," said Rentzenbrink. "E-reading is opening up channels to people who might not have thought of themselves as a book reader. I hope things are getting better."