Bank of America Merrill Lynch doctored paperwork on 16 million orders to fool institutional clients into thinking stock trades were taking place in-house when they were not, according to New York's Attorney General.

The bank admitted on Friday to "systematically misleading clients" between 2008 and 2013 about how orders were handled for more than 4 billion shares of stock.

New York Attorney General Eric Schneiderman said Bank of America (BAC) agreed to pay a record $42 million penalty for the "masking" scheme and violations of New York securities law.

The settlement said Bank of America had "undisclosed agreements" with electronic market makers to "secretly" route them trades for five years. Market makers buy and sell securities to help markets function smoothly.

"Bank of America Merrill Lynch went to astonishing lengths to defraud its own institutional clients about who was seeing and filling their orders," Schneiderman said in a statement.

The investigation uncovered an email from a Bank of America executive that explained a desire to increase the volume of trades to electronic market makers "based on other revenue opportunities being discussed with them." It's not clear what those opportunities were.

Wall Street firms "cannot use new technology to exploit their clients in service of their business relationships with large industry players," Schneiderman said.

Authorities said Bank of America carried out the "fraud" by reprogramming its electronic trading system to "automatically doctor" trade confirmation messaging sent back to clients.

Employees referred to this as "masking" — replacing real information about where trades took place with a code to indicate it happened in-house.

Schneiderman's investigation revealed that Bank of America told clients that trades were taking place in-house when they were really going to electronic market makers such as Citadel Securities, Knight Capital and now-defunct Madoff Securities.

Many of these market makers engage in high-frequency trading. Knowledge of looming trade orders can be extremely valuable to these firms, allowing them to jump in front of large trades with ones of their own.

That's why some large institutional investors, such as pension and mutual funds, prefer to conduct trades at in-house "dark pools," alternative exchanges that offer greater discretion. The New York investigation found that Bank of America "inflated its claims" about its own dark pool.

Related: Wells Fargo's CEO gets 36% pay raise

In some instances, Bank of America employees even told institutional clients who asked that trades were not going to electronic market makers, according to the settlement.

In a statement, Bank of America said the settlement "primarily relates to conduct that occurred as long as 10 years ago." The bank said that "at all times we met our obligation to deliver the best prices to clients" and it has since "addressed" the client communication issues.

Bank of America shares fell nearly 5% on Friday, significantly outpacing the declines for the broader markets.

Authorities indicated the "masking" practice only involved institutional clients, not everyday investors.

However, some of those institutional clients were mutual and pension funds that everyday investors own portions of.