Members of Britain's literary establishment have combined to condemn Chris Grayling's ban on sending books to prisoners.

Alan Bennett, Salman Rushdie, Carol Ann Duffy, David Hare and even Jeffrey Archer – novelist, former Tory party chairman and ex-prisoner – declared support for a campaign by the Howard League for Penal Reform urging the justice secretary to drop his ban on family and friends sending books and other essentials to prisoners.

"While we understand that prisons must be able to apply incentives to reward good behaviour by prisoners, we do not believe that education and reading should be part of that policy," says a letter signed by more than 80 leading authors.

"Books represent a lifeline behind bars, a way of nourishing the mind and filling the many hours that prisoners spend locked in their cells. In an environment with no internet access and only limited library facilities, books become all the more important."

Frances Crook, the Howard League's director, said she hoped the intervention by a distinguished list of writers and dramatists would persuade ministers to make a U-turn.

"The Howard League is going to be seeking a meeting with Mr Grayling to see if we can resolve this problem and get prisoners reading," she said.

The signatories include Hari Kunzru, Irvine Welsh, David Edgar, Joanne Harris, Julian Barnes, Mark Haddon and Sarah Waters. The letter follows an attempt on Tuesday by Grayling to label opponents of the book ban as "leftwing" groups which he claimed were unable to deal with the government's rehabilitation agenda.

Writing on the ConservativeHome website, he strongly defended rules introduced in November allowing prisoners to receive only letters and cards from home as part of a new earned incentives and privileges scheme which involved better behaved prisoners getting better access to funds to buy their own books. The measures, in effect, mean prisoners can no longer receive books. Haddon told BBC Radio 4's Today programme that prisoners would have little money to spend on reading material and would have to order books from a special catalogue.

Grayling, however, said the measures were the kind of change the public wanted to see – "a regime that is more spartan unless you do the right thing".

He wrote: "It was never the case that prisoners were simply allowed unlimited parcels – books or otherwise … It would be a logistical impossibility to search them all, and they would provide an easy route for illegal materials. The only change over the past few months has been to ensure all prisoners are treated the same. They can receive one parcel of essential items when they first arrive, but after that they can only get letters and cards from home, unless the circumstances are exceptional."

David Cameron's official spokesman said the prime minister backed the ban on receiving books and entirely supported Grayling, whose department imposed the ban to preserve a rigid system of rewards and punishments for prisoners. He said there was no need for prisoners to be sent books as they could borrow from prison libraries and keep some reading material in their cells.

Jeremy Wright, the prisons minister, ruled out a change in the new rules. He told the Today programme the rules were not about banning books but "sensible security precautions" to prevent items such as drugs being sent in. Friends and families could always send money to prisoners so they could buy books, he added.

Yet even senior Tories, normally supportive of Grayling, have seemed bemused. One senior minister told the Daily Mail: "Chris Grayling wins the prize for the government's least enlightened minister. He has no backing for this from any quarter at all."

The row began after Frances Crook wrote an essay for the website Politics.co.uk saying how "from now on, any man, woman or child in prison will not be able to receive a book from outside", and called the situation "part of an increasingly irrational punishment regime orchestrated by Chris Grayling that grabs headlines but restricts education or rehabilitation".

Under the rules, convicted prisoners were required to wear prison uniform for the first two weeks of their sentence. Many prisoners also lost their automatic daily access to a gym and to daytime television.

The change in the punishment system in jails in England and Wales also meant individual prison governors lost much of their discretion over which perks and privileges could be used to reward good behaviour, which are now prescribed nationally by the Ministry of Justice.

The ban on books being sent to prisoners by families and friends is part of a new "incentives and earned privileges" regime, which allows prisoners access to funds to buy books and other items as they move up from "basic" level.

Grayling said the big change in the prison regime meant prisoners no longer got privileges just by "keeping your nose clean" but by engaging in "proper rehabilitative activity".

Penalties for bad behaviour were tougher too, including prisoners losing the right to wear their own clothes and having to wear a uniform instead.

"And of course it's the kind of thing that leftwing pressure groups hate. That's why they are trying to persuade the public that we have banned books and brought a halt to all rehabilitation. It is, of course, complete nonsense."

Andrew Neilson, director of campaigns at the Howard League, responded to the apparent attack from Grayling by saying: "Concern on this issue extends far beyond leftwing pressure groups or indeed a nonpartisan charity of almost 150-year standing and consultative status with the UN, which the Howard League for Penal Reform enjoys."

A former prisoner told the Guardian that although libraries existed, access could be severely restricted, particularly in closed prisons.

"I've been in places where prisoners only get 20 minutes a week to visit the library and change books." Cutbacks had meant, he said, that few new books were bought and local libraries were far less able to supply books on loan.

He said the new rules also set a strict limit on the number of books a prisoner was allowed to have at any time – 12, excluding the Bible or other approved religious texts. This meant prisoners studying for Open University courses or other qualifications often could not get hold of the study material they needed.