Chris Grayling, the justice secretary, has admitted the prison service is facing "pressures" from an unexpected increase in the number of inmates but denied suggestions of a crisis.

As inspectors warned about levels of violence at Isis prison in south-east London, Grayling insisted violence was at a lower level than five years ago.

But he told the Today programme on BBC Radio 4 that the prison service was having to cope with the pressure of hundreds of inmates convicted of sex offences after the Jimmy Savile scandal.

The justice secretary was speaking after Nick Hardwick, the chief inspector of prisons, warned that overcrowding and staff shortages in England were leading to an increase in the number of inmates killing themselves.

Grayling said the prison population in England and Wales had increased by unexpected levels in the past year but he denied there was a crisis. He told Today: "We are looking to make sure that we can continue to deliver a safe and appropriate prison regime in a world where budgets are much lower than they were in the past. We've actually got a prison estate where violence today is at a lower level than it was five years ago. We've got challenges from an increased population that was not expected in the last 12 months.

"We are meeting those challenges, we are recruiting more staff – but I'm absolutely clear there is not a crisis in our prisons. There are pressures which we are facing but there is not a crisis."

The justice secretary was speaking as Her Majesty's Inspectorate of Prisons reported that a third of inmates feel unsafe at the Isis prison, which saw 254 fights and assaults in 2013. Hardwick said of the prison in Thamesmead: "The location of the prison, the volatile population it holds and continuing staff shortages mean the risks and challenges it faces are significant. Opening any new prison is complicated and although Isis has now been open for three years it is still in a settling phase."

Critics have blamed government spending cuts for a reduction in the number of prison places and prison staff. The Howard league reported in July that prison officer numbers were at "breaking point" after a 30% cut in three years. The league reported that there were 19,325 officer-grade staff in September 2013, down from 27,650 in September 2010.

Figures published by the Ministry of Justice show an increase in the number of assaults by prisoners in England and Wales from 14,083 in 2012-13 to 15,033 in 2013-14. But Grayling said that over the past five years the number of assaults had fallen from 15,775 in 2008-09 to 15,033 in 2013-14.

Grayling said: "Those figures show that the number of violent incidents in our prisons is lower than it was five years ago under the last government. They have gone up in the last 12 months. Figures go up, they go down, they don't follow a scientific path. Any assault is too much in our prisons. We are now working to bring it down again."

Eoin McLennan-Murray, president of the Prison Governors Association, told the Today programme: "There is an increase [in the prison] population. We are used to dealing with that. What we have, though, is a number of things which have all coincided in a very short space of time. We have had massive changes in our management structures, we have this benchmarking programme being rolled out across the service, we have had changes for conditions for prisoners in terms of their incentives and earned privileges. And we have had this shortage of staff."