Bank of New York Mellon Corp. BK 1.45% currency traders used a foreign-exchange system called "Charlie" to create fake trades and overcharge Virginia pension funds by at least $20 million, according to allegations in recently unsealed documents in a Virginia court.

The allegations, made by a whistleblower group, are part of a widening probe by state prosecutors into whether custody banks such as Bank of New York Mellon and State Street Corp. STT 0.43% shortchanged public pension funds in executing currency trades used to complete financial transactions abroad.

Bank of New York Mellon and State Street deny wrongdoing and said they will vigorously fight legal actions against them. In a statement Thursday, Bank of New York Mellon said: "Money managers transact with us at competitive FX prices and we provide reliable, low-risk service and execution."

Separately, Florida Attorney General Pam Bondi on Thursday intervened in a 2009 whistleblower lawsuit in a claim that alleges Bank of New York Mellon overcharged Florida pension funds for foreign-exchange transactions.

California has intervened in a whistleblower claim against State Street; Virginia has intervened in the whistleblower claim against Bank of New York Mellon.

Shares of Bank of New York Mellon and State Street fell 1.2% and 1.5%, respectively, Thursday in 4 p.m. composite trading on the New York Stock Exchange, following a Wall Street Journal report about the probe.

"Custodian banks make a fair amount of money off of [foreign-exchange] transactions as part of their custody operations," said Giri Cherukuri, head trader at OakBrook Investments in Lisle, Ill. She added that "they may have to change their business practices."

Currency trading is a high profit area for some big banks. According to the Office of the Comptroller of the Currency, U.S. banks generated $5.6 billion in foreign-exchange trading revenue in 2009 and $7.2 billion in the first three quarters of last year, making currency trading one of the largest sources of banks' trading profits.

Bank of New York Mellon generated 2009 revenue of $4.26 billion for its asset-servicing unit, including $757 million from foreign-exchange and other trading activities, the latest full-year figures available. The $4.26 billion figure was more than half of the bank's total 2009 revenue, though the 2009 total included an unusual one-time item. Asset servicing typically accounts for about a third of the bank's revenue.

For some banks, foreign-exchange trading revenue is on the rise. Bank of New York's revenue from foreign-exchange trading for clients jumped about 60% in the fourth quarter of 2010 compared with the third quarter of last year, and they rose 19% from 2009's fourth quarter, according to brokerage firm Guggenheim Securities.

The states are investigating claims that the banks didn't charge the pension funds the currency rates that the banks paid but consistently charged them the highest currency-conversion prices of the day, and pocketed the difference. The suits say the banks similarly overcharged when the pension funds exited the trades.

U.S. investors trading in global stock markets must convert dollars into the currencies of the foreign countries in which they invest. Custody banks facilitate the foreign exchange.

Ms. Bondi's motion to intervene was filed in a Leon County circuit court in a move that enables her to control the case. Ms. Bondi, a trustee of Florida's State Board of Administration, a $154.8 billion pension and fund program for Floridian teachers and employees, will file a formal complaint "in the near future," Thursday's filing said.

The 2009 Florida lawsuit was filed by a Delaware shell company called FX Analytics, which also filed the Virginia case against Bank of New York Mellon. Two lawyers, Michael Lesser at Boston firm Thornton & Naumes LLP and Philip Michael in New York, are handling the claims in California, which is suing State Street, and Virginia, which is suing Bank of New York Mellon. Bank of New York Mellon is one of the nation's largest custody banks. The Virginia suit said the bank touts its "24-hour trading capability" and refers to itself as "foreign-exchange specialists."

Bank of New York Mellon and State Street deny wrongdoing and said they will vigorously fight legal actions against them. Associated Press

The court documents filed by FX Analytics allege that currency traders at Bank of New York Mellon at times waited for hours before cherry-picking prices beneficial to them that they would charge the Virginia state pension fund.

Custody clients typically deal in currencies through two ways, known as "direct" or "indirect" trading. In direct trading, clients trade for themselves by negotiating pricing and ensuring a good foreign exchange.

Public pension funds, often ones that don't want to hire people to negotiate, can opt for indirect execution, also known as "standing instruction" or "non-negotiated," according to the Virginia complaint.

At Bank of New York Mellon, the bank would place currency trades for pension funds seeking to make global securities transactions through an electronic pipeline called "Cash Management System," or CMS. CMS then would route the currency request through a bank foreign-exchange system called Charlie.

Instead of pricing the currency trade for the pension fund at the time it was made, the complaint alleges, the bank identified higher prices if the pension fund was buying currencies, and pocketed the difference.

When the pension fund received a monthly custody report, the suit alleges, it was given only the fake, or "falsified" rate. The reports don't contain time stamps.

In one trade cited in the Virginia lawsuit, the whistleblower alleges that in October 2009, Bank of New York Mellon profited after carrying out a foreign exchange of Canadian dollars for a client at the worst U.S. dollar-Canadian dollar rate of the day.

According to the complaint, the fund, which the lawsuit didn't identify, needed to convert $12.5 million into Canadian dollars. Bank of New York sold $12.5 million of the client's money and bought Canadian dollars in the interbank market when the U.S. dollar bought 1.0795 Canadian dollars. Bank of New York Mellon took possession of C$13.5 million.

The fund, however, received C$13.35 million, based on the lowest rate of the day when one U.S. dollar was at C$1.0682. That allegedly enabled Bank of New York to pocket about C$141,250.

If the pension fund had opted to trade the currency itself, rather than allow the bank to do it, the bank's profit would have been about C$6,250, according to the suit.

Write to Carrick Mollenkamp at carrick.mollenkamp@wsj.com and Lingling Wei at lingling.wei@wsj.com