A division of Royal Bank of Scotland might be forced to plead guilty to criminal charges in the US for rigging Libor as part of its settlement for manipulating the key interest rate.

RBS is expected to be fined around £500m by regulators on both sides of the Atlantic, but the US is also reported to be trying to pin criminal charges on the bailed-out bank.

It is not yet clear if RBS will agree to any criminality. As part of its £940m total fine for rigging rates, the Swiss bank UBS pleaded guilty to felony wire fraud at one its subsidiaries, UBS Securities Japan. UBS Japan was charged by the US with "one count of engaging in a scheme to defraud counterparties to interest rate derivatives trades by secretly manipulating Libor benchmark interest rates".

The parent company UBS entered into a non-prosecution agreement which required the bank to pay a fine and "admit and accept responsibility for its misconduct".

The US justice department agreed not to prosecute Barclays, the first bank to be fined, because of its "admission of its misconduct, its extraordinary co-operation, its remediation efforts and certain mitigating and other factors".

RBS may be forced to follow a similar route to that taken by UBS and has been preparing for the fallout from its role in rigging the benchmark interest rate – used to set prices on £300tn of financial products – since Barclays was hit with its £290m fine in June. But discussions with regulators over the penalty it should face have been protracted and have not yet been completed.

The Financial Services Authority is expected to fine RBS around £90m – approximately a fifth of the total fine – and is one of a number of regulators around the world which have ongoing investigations into Libor rigging. Some 50 of the 400 staff in the FSA's enforcement division are working on Libor-related cases.

Stephen Hester, the RBS chief executive, said last year that he wanted to have resolved the situation by the time the bank reports its full year results on 28 February – his fourth set of figures since being parachuted in to run the bank after its October 2008 rescue by the taxpayer.

In November Hester said it would be a "miserable day in RBS's history". He said: "We are up for settling with all and everyone as soon as they are ready."

The size of the Libor fine could have an impact on the bonuses handed out in the RBS investment bank, whose boss John Hourican is expected to leave shortly with a bonus, deferred from previous years, of £4m. Union officials at Unite were angered over suggestions that the bonus pool could amount to £250m.

"The RBS division implicated in the Libor scandal is set to reap huge financial rewards, but innocent bank workers in call centres and branches up and down the country are having their jobs cut, pensions slashed and terms and conditions eroded," said Unite national officer Dominic Hook. Cash bonuses are expected to be restricted to £2,000 with the rest paid in shares.

RBS repeated on Thursday that "discussions with various authorities in relation to Libor setting are ongoing".

US regulators have stepped back from prosecuting the parent companies of banks in the past. For instance in December US authorities did not prosecute HSBC for money laundering because of the "collateral consequences" which might have led to the bank losing its crucial banking licence.