The outgoing chief inspector of prisons is explaining why he is so keen to get out of the job. It’s not the budget fights Nick Hardwick had with the Ministry of Justice, nor the fact that he wasn’t actively encouraged to apply for another five-year stint. It’s not even the fact that the previous secretary of state for justice, Lord Grayling, “robustly” tried to influence him – as Hardwick revealed to a select committee last week.

No, he simply feared that he was becoming desensitised; that he was getting prison-horror fatigue. “You shouldn’t do this job for too long because you get used to things you shouldn’t get used to,” he says. “I’ll give you an example of something that is objectively shocking, but how do you keep the outrage going? Take the level of self-harm and suicide. On one level, one bit of your brain is thinking, ‘Oh well, they’ve only had two suicides since we were last here, good.’ On another level, that is appalling.”

Hardwick, 58, barely takes a breath before providing another example: dirt. He recently did an inspection with a man who normally does police custody inspections. At the end, the inspectors discuss what they found. “We went through the physical conditions, saying, ‘Lots of people doubled up in cells mean, for one, toilets not adequately screened, next issue’ … and he went, ‘Hang on a minute, that’s disgusting.’ And we all said, ‘Yeah it is.’ But when you’ve seen it so often … In my early reports I used to bang on about two men in a small cell with an unscreened toilet, one of them eating their meals sitting on the bed next to a toilet, and I’ve stopped banging on about that now. But actually we should keep banging on about it because it is disgusting.”

If Hardwick has got compassion fatigue, he’s making a good job of hiding it. His outrage could not be more visceral. Throughout his five years as chief inspector, he has been an outspoken critic of the conditions he found in the prisons of England and Wales.

When Hardwick took up his post in 2010, the incoming coalition government promised a “rehabilitation revolution”. In his final annual report, in 2015, the chief inspector said he was “still waiting for this to happen”. The same report quoted an inspection at Wormwood Scrubs in which a guard urged him to look at cells the officer “wouldn’t keep a dog in” – broken windows, filthy, inadequately screened toilets and cockroaches everywhere.

But perhaps the most telling verdict on the system he leaves is the stream of reports on the young offender institutions he and his team visited towards the end of his tenure. To quote from his 2014 inspection of Glen Parva YOI, “this is a model of custody that does not work”. He said the same about virtually every YOI he inspected, citing rising levels of violence and self-harm; young men locked idle in their cells 23 hours a day; 30 minutes outdoor exercise the norm. The situation is so bad in the YOI estate that Hardwick took the decision to inspect them annually, instead of every three or four years, which is the norm across the penal estate.

Glen Parva young offenders institution in Leicestershire, which Hardwick described in 2014 as ‘a model of custody that does not work’. Photograph: Rui Vieira/PA

While the outspoken Hardwick has had his critics, he has also had plenty of supporters. Surprisingly, justice minister Michael Gove appears to be one of them. This week he announced that Hardwick would be the new head of the Parole Board. It’s an interesting choice, given that Hardwick does not believe that many of those incarcerated in England and Wales for minor crimes should be there in the first place. But, he makes clear, the new job will largely be dealing with very different people – serious offenders, who should have been sent to prison.

We meet at HM Inspectorate of Prisons in Holborn, central London, just a few days before he is due to clear out. In fact, he has already started. The spartan office is normally quite cosy, he explains. “I don’t want you to think I live in a bare cell. I normally have pictures on the wall, but at the weekend we came in by car and loaded it all up. That’s why it looks so bleak.” Hardwick is a big, bouncing ball of a man with a demotic accent that regularly drops Hs and Ts.

Whereas a number of former chief inspectors have been old-establishment figures who turned native (notably, former army general David Ramsbotham), Hardwick has always supported the underdog. Thirty years ago, he was made chief executive of the homeless charity Centrepoint. He then became chief executive of the Refugee Council before heading up the Independent Police Complaints Commission. Hardwick is no flibbertigibbet. He tends to put in a good shift of eight or nine years before moving on to the next worthy post. But this time, as chief inspector of prisons, he only managed five.

During that time, he discovered how much he hated being in prisons. “If you ask me how my view has changed, and this may be another reason why I’m not sad to leave the job, I’m surprised by how much I don’t like being in prison. Although I have keys and can get out at any time, and I regard myself as pretty resilient, it’s the noise, the echo, the clanging, the claustrophobia, the sense that even if you’ve got keys you’re shut in, and the unhappiness.”

He grits his teeth as he talks. “I didn’t understand the degree to which, once you lock someone up, even in the best prisons for a short period of time, that is a very severe punishment indeed.” He laughs at the notion that prison is soft. “It’s as bad as you could possibly imagine and possibly more so, and don’t think a little flat-screen television in the corner is going to alleviate it, because it doesn’t.”

Hardwick says there are so many problems with our prisons. For starters, they don’t prepare you to return to society. “What a good prison does is teach you to be a good prisoner, so it teaches you to be compliant, not to use your initiative, to do what you’re told, to rein in your emotions, and that isn’t necessarily what you need to do to be a good citizen, or a good parent.” Prisons are based on rigid rules, he says, and another problem is that most prisoners are no good at following rules – that’s why they ended up in prison in the first place.

And then there’s the issue of who is in there in the first place. The more time he spent in prison, the more he wondered whether many of those locked up should have been. “It is striking the number of people in prison who are obviously ill, who have either got mental health problems or substance-abuse issues.

“At one end of the spectrum, you have people who are clearly ill who definitely shouldn’t be in prison, and we need to find ways of diverting them out of the criminal justice system.” These are by no means the only prisoners he fears for. “Then there is a bigger group in the middle who may not be ill per se, but certainly struggle to cope. If we had better care in the community – not just in a sense delivered by the state, but actually if we all took a bit more care of each other – then some of those people could be managed much better in the community than prison.” He accepts such people can be difficult, that many are a “nuisance”, but he still insists that they should not be in prison for minor crimes.

Roughly how many of the 86,000 people locked up in England and Wales is he talking about? “I’m talking about a very large proportion of the prison population.”

Hardwick appearing before a justice select committee earlier this month, during which he said that his independence had been undermined by the Ministry of Justice. Photograph: BBC

Hardwick was born in Surrey but considers south London his stamping ground. He is an intensely private man. He has given few interviews during the years, and they contain virtually no personal details. He concedes that he graduated with a third in English from Hull university in 1979, that he was anything but a swot, and that he has a wife and son. But that’s as much as we’re getting. And even that’s a struggle.

He segues to another issue that has concerned him in his half-decade as chief inspector – asylum seekers detained in immigration removal centres. And here he is almost hissing with anger. Locking up asylum seekers is simply an abuse of power, he suggests. “These people haven’t been convicted of anything, and they’re detained on the say-so of a relatively junior civil servant. If you lock someone up in a detention centre, you are punishing them. Whether that’s your intention or not, you are. Right? Even if you’re trying not to run it like a prison. Even if you have the best staff in the world, right, it’s still a prison.” And the bottom line is, he says, most detained asylum seekers have not committed a crime. “It should be very exceptional that you lock someone up without going before a court, and at the moment, it’s simply not exceptional enough.” He returns to the junior civil servant. “It ought to be a huge decision to lock someone up, and the problem is that if you make a huge decision often enough, it becomes not such a huge decision; it becomes routine.”

Hardwick believes there are two major failings with policymakers – “lack of imagination and failure of empathy”. “Too many policymakers do not ask themselves the crucial question: ‘How would I react if I were in that situation, and why are people in prison in the first place?’”

For Hardwick, the most striking example is children. In fact, he says, if he had the clout to close down any institution and start again from scratch, he would do so with youth prisons. “If I could, I would bear down on the juvenile estate, because these are the buildings that are least fit for purpose. Children should be held in much smaller, more caring places, he says. He talks about the time he visited one institution and a boy asked him for help. “I said, ‘What can I do?’ and he said, ‘I want to go home to my mummy, and echhhhh … ’” He chokes up and can’t complete his sentence. “Then another boy in healthcare was just lying in his bed with his blanket pulled over his head. I went away thinking I wouldn’t be surprised if I got a call over the weekend saying any of those boys had hurt themselves.” He pauses, and drinks his coffee from a navy blue NYPD mug. “And that was not an uncaring place. But you think: oh God!”

He hates the way these kids are so often referred to as young people, as if to disguise the fact that they children. “I went on the warpath about the boy who wanted to go home to his mummy, and got him into a hospital.” Again, he pauses. “And they are boys – largely boys – and girls, they are not young people.”

We have the ultimate duty of care to the children we lock up, he says, and we are currently failing them. “The kids in custody are our most troubled children; school teachers, social workers, community workers, all these skilled professionals have not been able to get through to them, so what do we do? We put them in an institution with loads [of others] just like them. It’s bonkers, right? Bonkers. And we could afford to take these children and have them in smaller units with the skilled professional staff and staffing ratios to be able to make headway. There are not so many of them that it would break the bank.” You’re only talking about around 1,000 children, he says.

Again, he says, it goes back to empathy – or lack of it. “Once the state decides to take over the parental responsibility of a child, it has obligations to do all it can for them. If it was our own children in those places, or in that level of need, you would move heaven and earth to try to sort them out and help them. Well, these are our children; we should be moving heaven and earth to try to make a difference and we don’t.”

Last week, he told the justice select committee how his independence had been undermined. For starters, he had to have his budget cleared on a weekly basis for inspections, which resulted in him threatening to suspend all inspections. The solution to this “absurd” situation is simple, he says. “The inspectorate should not be sponsored by the department that has direct operational experience with the things that we inspect. Right?” He mentions a conversation he recently had with a team of Russian inspectors and laughs. “I was going on about how important it is to be independent, and they said, ‘Well, who appoints you?’ I said, the Ministry of Justice, and they asked, ‘Who sets your budget?’ Well, that’s the Ministry of Justice, I said. And they go, ‘Ah, you mean that kind of independence!’”

Chris Grayling, former justice secretary, who Hardwick says deliberately tried to influence one of Hardwick’s prison reports. Photograph: Mark Thomas/Rex/Shutterstock

The most alarming thing he said to the select committee last week was that Grayling “robustly” tried to influence reports. But he never went into details. Actually, he says now, that’s a slight misquote. “I said we had some very robust conversations with Chris Grayling – that was often after reports had been produced.”

But yes, he says, there is one particular report that he did try to change. “The only time he tried to deliberately try to influence a report was an annual report where he said, ‘This is what I think should be in your annual report’, and I said, ‘That’s very interesting,’ and then went away and wrote my report.” What did he try to change? “It was the last one, published this July, and I can’t for the life of me remember what he said should be in it, because I wasn’t taking much notice.”

Was it an issue of substance? “He was telling me the points he hoped I would make that were positive, and I didn’t think it was his place to say that.”

Can he give a general indication of the positive points he wanted made?

“I can’t remember, I didn’t take a note of what he said.” Really? He ums and ahs, looks at his PA, then finally comes out with it. “His general concern was that I had said the lack of staff, overcrowding and some of the policy changes that he had introduced had contributed to poor outcomes in prisons. I was very clear about that, and he disagreed very strongly with that conclusion.”

What does Hardwick he thinks he has achieved in the job? Well, he says, women’s prisons are better than they were, and now most inspections are unannounced, whereas previously only half were, but he’s struggling for positives.

“The other achievement is much more modest,” he finally says. “In the adult estate, the reality is [that] things have got worse, and I think they would have been even worse were it not for us.” He smiles. “That’s probably not a great claim to make, is it?”