More than 40 people have been caught trying to break into prisons in the last five years, highlighting a growing drug problem in the penal system, according to new government figures.

The data released by the justice secretary, Jack Straw, showed that 19 people were involved in such incidents last year alone.

The figures appeared to confirm warnings from the Prison Officer Association.

The shadow justice secretary, Nick Herbert, who requested the information, said: "Whether these are offenders trying to return to jail, as prison officers have alleged, or dealers trying to traffic drugs, it is ludicrous that supposedly secure establishments can be breached in this way."

In April, the POA assistant general secretary, Glyn Travis, said: "Drugs are coming in at a rate that's so dramatic that [they] are actually cheaper than on the outside.

"It tells me there's something wrong in society when people are breaking into jail to bring in drugs and prostitutes, but the prisoners are quite happy to stay inside."

He highlighted one break-in at Everthorpe prison, in east Yorkshire, when drug dealers and prostitutes entered the jail using ladders.

The figures said ladders were used by 13 of the 42 people who were caught trying to break into prisons since 2003. Most of the incidents involve break-in to open jails.

The Prison Service denied inmates had any opportunity to escape in the Everthorpe incident.

"The prison was aware of a security breach back in January during which drugs were being brought in at night. At no time were prisoners out of their cells or able to access any other areas of the prison," a spokesman said.

"Immediate action was taken with extra fencing, the removal of trees, extra CCTV cameras, and the transfer of the offender involved to another establishment."

Frances Crook, the director of the Howard League for Prison Reform, said: "The majority of drugs being brought into prisons come through staff, who did it for money or are intimidated by prisoners. The quantities involved could not come in from a kiss [between inmates and visitors].

She added: "People trying to bring break in to prisons to bring in drugs are the stupid ones. People break in for a range of different reasons. They could be vulnerable people – if you get released from prison after 30 years, the only family and community you have is prison, it can be terrifying to be released because of the support on the outside."

A justice department spokesman said: "Whilst trespass into open prisons, by their very nature, is more difficult to control, there has only been one identified case of a member of the public breaking into a closed prison in the last five years."

He added: "Prison is anything but soft and it is absurd to suggest otherwise."