A mandatory minimum four-month prison sentence for 16- and 17-year-olds who are found guilty of "aggravated" knife offences is to be introduced despite the open opposition of the justice secretary, Kenneth Clarke.

A new "two strikes and you're out" mandatory life sentence is also to be introduced for anyone convicted of a second serious sexual or violent crime – extending mandatory life sentences for the first time to cover crimes other than murder.

The definition of the "most serious sexual or violent offences" that will be covered by the new "two strikes" policy is to be extended to child sex offences, terrorism categories and "causing or allowing the death of a child".

The surprise announcement of this new sentencing regime marks a return to a more traditional "lock 'em up" approach to law and order by the coalition government and deals a further body blow to Clarke's hopes of a more liberal penal policy that would stabilise the prison population.

The package represents a major extension of the use of US-style minimum mandatory sentences into the British legal system and comes after a fierce cabinet battle.

Clarke is reported to have repeatedly clashed with the home secretary, Theresa May, over this issue behind the scenes.

It is believed to have only been settled on Wednesday by the intervention of David Cameron. Clarke appears to have won a concession that children under 16 will not be affected.

Clarke made clear his personal opposition to the use of mandatory sentences at a hearing of the Commons home affairs committee only on Tuesday, indicating he preferred to give judges unfettered discretion to set sentences based on the facts of the cases for nearly all crimes but murder.

He also made clear his view that a minimum mandatory sentence for juveniles under 18 was not part of the traditions of the British criminal justice system.

The sentencing regime announced on Wednesday night includes replacing the much-criticised indeterminate sentence for public protection, which has left 6,500 prisoners without a set release date, with new fixed-term sentences. Dangerous criminals will in future serve at least two-thirds of the new sentence.

The new sentencing regime detailed by the Ministry of Justice includes:

• A four-month mandatory custodial sentence for aggravated knife possession for 16- and 17-year-olds but not for younger children. Those convicted of using a knife or offensive weapon to threaten and endanger will be given a four-month detention and training order. Adults are to face an automatic six-month sentence for the same offence.

• A "two strikes and you're out" mandatory life sentence for anyone convicted of a second very serious sexual or violent offence, extending the life sentence beyond murder for the first time.

• An extended determinate sentence [EDS] for dangerous criminals convicted of a serious and violent sexual crimes, who will serve at least two-thirds, scrapping the current consideration of parole at the halfway point. Release for those in the most serious category serving this sentence will require the approval of the parole board. Those paroled will be under recall licence for at least 10 years.

• An extended licence period: those who have served an EDS will have to serve a further period on licence – an extra five years for sex offenders and eight years for violent offenders – during which they can be recalled to prison if necessary.

The new sentencing regime is contained in amendments tabled on Wednesday night to the legal aid, sentencing and punishment bill and is likely to increase the already record 87,000 prison population in England and Wales.

It places a serious question mark over Clarke's hopes of stabilising the jail population.

The justice secretary said he expected more dangerous offenders to get life sentences. "The new regime will restore clarity, coherence and common sense to sentencing, rid us of the inconsistent and confusing IPP regime and give victims a clearer understanding of how long offenders will actually serve in prison,

"We have already announced that we are bringing in an automatic prison sentence for any adults who use a knife to threaten and endanger. Clearly any extension of this sentence to children requires very careful consideration. However, we need to send out a clear message about the seriousness of juvenile knife crime, so we are proposing to extend a suitable equivalent sentence to 16-17 year olds, but not to younger children." he said.