The world's third richest man, Warren Buffett, today declared that he saw nothing wrong in an allegedly fraudulent $1bn mortgage deal by Goldman Sachs and suggested that losers in the transaction, including Royal Bank of Scotland, had only themselves to blame for exercising "dumb" judgment.

After weeks of silence on the subject, Buffett delivered a valuable endorsement to the embattled Wall Street bank at Saturday's annual meeting of his Berkshire Hathaway business empire in Nebraska. The 79-year-old billionaire said he "loved" his own $5bn investment in Goldman and offered "100%" support to the bank's chief executive, Lloyd Blankfein.

"I do not hold against Goldman at all the fact that an allegation has been made by the Securities and Exchange Commission," Buffett told a gathering of 40,000 of his followers at Omaha's cavernous QWest sports stadium. But if fraud charges were proven, he added, "it's something more serious and we'd look at that at the time".

A long-term customer of Goldman, Buffett has enlisted the bank's help in buying scores of businesses. At the height of the financial crisis in September 2008, he lent $5bn to Goldman – a loan that yields annual income of $500m for Berkshire Hathaway, at a rate of $15 a second.

"We love the investment," said Buffett, who accepted that the SEC's charges had damaged Goldman's reputation. "There's no question the allegation alone causes the company to lose reputation. Obviously, the past few weeks have hurt the company and hurt morale."

The SEC's case against Goldman centres on a 2007 mortgage derivatives deal named Abacus, struck by Fabrice Tourre, a banker now based in London. The SEC contends that Tourre failed to inform participants about the extent of involvement of a hedge fund, Paulson & Co, that had a short position betting on Abacus's failure.

Within nine months, more than 99% of the mortgages referenced by Abacus were in default, leaving Royal Bank of Scotland's Dutch subsidiary, ABN Amro, with an $840m bill as it had insured the derivative against failure.

Buffett was scornful of this loss, pointing out that ABN voluntarily agreed to insure Abacus for a fee of $1.6m. "It's a little hard for me to get terribly sympathetic with the fact that a bank made a dumb credit deal."

The billionaire, who is known as the "sage of Omaha", described Goldman's chief executive as "smart" and "high grade" and brushed aside a question about who could replace him: "If Lloyd had a twin brother, I would vote for him. I've never given it a thought."

Buffett's annual meeting entails a vast influx of people into his modestly proportioned home town. Omaha's Eppley airport said it was expecting arrivals from 100 private jets on Friday, 15 times more than usual.

Buffett and his business partner, Charlie Munger, spend a day each year dispensing pearls of folksy wisdom in response to investors' questions. Front-row spectators include Bill Gates, a close friend and associate of Buffett.

Early in the meeting, Buffett offered thoughts on the impact of Greece's financial crisis on the euro. He described the economic meltdown in Athens as a test case in what happened when a nation made promises to its citizens while it lacked the power to print its own money.

Buffett predicted "high drama" in the eurozone, saying: "I don't know how this movie ends. That doesn't mean I'm forecasting disasters. But I try not go to movies like that."

With a collection of businesses varying from Dairy Queen cafes to Geico insurance and Fruit of the Loom underwear, Buffett's personal fortune is estimated at $47bn. He is ranked by Forbes magazine as the world's third richest man, behind the Mexican tycoon Carlos Slim and Microsoft's Bill Gates.