State-owned bank could take legal action against US firm fined $550m by SEC for its role in sale of mortgage-backed security

Goldman Sachs faces the threat of legal action from Royal Bank of Scotland, which lost $800m (£523m) on the complex mortgage security at the heart of a record-breaking fine by the US regulators.

Even though Goldman has settled the case brought by the securities and exchange commission over the Abacus collateralised debt obligation (CDO), the Wall Street firm may not yet be able to draw a line under the controversy caused by the way it sold the complex deal to clients.

Goldman is paying a $550m fine in a settlement with the commission, after admitting that the marketing materials it issued to investors buying the Abacus transaction were "incomplete" and did got give a true picture of the product.

Abacus was one of thousands of CDOs created in the run-up to the sub-prime mortgage crisis where banks packaged up lots of home loans and then sold them to on to investors. In this case, though, the SEC argued that Goldman did tell not the investors in Abacus that one of its powerful clients, the hedge fund Paulson, had helped to choose the loans included in the CDO, or that Paulson was also using Abacus to bet that house prices would collapse – and therefore profit at the expense of investors in the financial instrument.

Goldman reached the settlement with the commission as it prepared to publish its results for the second quarter, which are expected to show a slowdown in activity on the same period last year. Analysts have downgraded their forecasts for the group after June proved to be a very slow trading month.

Even so, analysts at Wells Fargo expect the firm to report net revenue of $8.5bn – down more than 35% – and set aside as much as $4.3bn to pay staff their monthly salaries and bonuses at the end of the year.

While the commission is continuing to investigate the Goldman employee Fabrice Tourre, who marketed Abacus, the firm may also face legal action from RBS. The bank inherited an exposure to the transaction when it took over Dutch rival ABN Amro as the banking crisis began.

The City regulator, the Financial Services Authority, is also continuing its investigation into the transaction while Tourre, who had been registered with the FSA to work for Goldman, remains on leave.

About $250m will be paid to clients who lost money in the deal. German bank IKB, the main investor, is being handed all of the $150m it put in, but RBS is receiving only $100m of the $800m that it lost.

RBS, which was bailed out by the taxpayer in October 2008 and is more than 80% owned by the government, is now deciding whether to try to reclaim the rest of its losses. The Edinburgh-based bank said: "RBS has been monitoring the complaint closely. Following the SEC's announcement, RBS will now carefully consider all of its options."

IKB was also "carefully reviewing" the terms of the settlement between the commission and Goldman.

Unlike IKB, RBS was not a direct investor in Abacus but was exposed because ABN Amro had provided credit insurance to Goldman in the event that ACA, a specialist firm that also helped to select the loans included in the deal, went bust. ACA did in fact collapse.

Goldman would not comment on any further cases that might arise after the recent settlement, but lawyers warned that the firm, and others on Wall Street, should be "vigilant".

Ben Bruton, banking litigation partner at Eversheds law firm, said: "Goldman Sachs will no doubt view the settlement of the SEC investigation as drawing a line under the allegations surrounding the marketing of Abacus. However, it and other investment banks should remain vigilant against future investigations by regulators, especially those concerning the way in which other CDOs and structured products were marketed. By undertaking thorough internal investigations into issues of this kind, investment banks will be better prepared to respond effectively."