The former justice secretary Chris Grayling was warned repeatedly by members of the Tory party not to part-privatise the probation service, according to an ex-minister.

Phillip Lee, who quit to join the Liberal Democrats earlier this month, has revealed that during his two years in the justice department there were “deep concerns about that policy” from within the party itself.

The probation service was part-privatised in 2014 under Grayling, the then secretary of state. It cost £500m and involved lower- and mid-risk offenders being supervised by 21 private companies.

After a series of failings from providers, including a sharp rise in offenders being recalled to prison for breaching their licence terms, the government decided to end the contracts early and received significant criticism from the National Audit Office.

Lee quit the government over Theresa May’s Brexit policy in 2018 after serving as a justice minister since 2016. He issued his criticism while speaking at his first Lib Dem party conference in Bournemouth at a fringe event on vulnerability in the justice system. He told party members that the Tory government “knew internally that [the system] was a mess”.

He said: “There were some market-sensitive issues because obviously they were private companies. What frustrated those of us on the liberal side of the Conservative party was the person who was responsible for introducing the original policy was told repeatedly not to do it.”

Grayling was at the helm of the department between 2012 and 2015. The quality of probation and the performance were regularly raised by people from within the Ministry of Justice and within the Tory party, Lee told the conference.

He said: “In fact, internally within the Conservative party, there were deep concerns about that policy.”

Lee’s frank appraisal of justice policies developed by the party under Grayling’s leadership also involved concerns about the relationships with lobbying groups for women and victims, which he described as “pretty awful” when he first joined the ministry.

“I spent most of my time in the first, six, nine to 12 months trying to get that relationship into a better position,” said Lee, adding that the department was a revolving door of ministers and not often viewed as a department for which politicians craved to work.

Johnson’s cynical ‘tough on crime’ talk shows the depth of his incivility | Simon Jenkins Read more

Support for certain policy ideas or reform wavered also among his Tory colleagues, he said, adding how “[I] often found myself at the despatch box, looking forward, getting more support than I did from behind me in my time at the justice department”.

One of those policies included trying to pass reforms on prisoners on temporary licence being able to vote, which he said had more support from opposition parties than among Conservatives.

He said he and the then justice secretary David Lidington had believed “we could have gone for it. We had parliament behind us,” but not the full support of the party.

Lee said it was striking that of the Tories who no longer have the whip, four are former justice ministers.