By Peter Rudegeair and Tanya Agrawal

(Reuters) - Bank of America Corp said on Monday that regulators had suspended its plan to buy back more shares and raise its dividend after the bank realized it had miscalculated a measure of the capital on its books.

The second-largest U.S. bank said fixing the mistake reduced a capital level by $4 billion, or about three-quarters of the extra money that the Federal Reserve had approved its returning to shareholders over the next year.

News of the gaffe sent the bank's shares down 6.3 percent on Monday to close at $14.95, in the biggest one-day decline in the stock since November 2012. (BREAKINGVIEWS-BofA reclaims banking dunce cap with $4 billion flub.

The announcement illustrates how difficult it is to determine appropriate capital levels for the biggest banks, particularly under hypothetical stress situations that regulators consider. Bank of America now has to submit its request to return more capital to shareholders for a third time, and the Fed itself previously erred in projecting the bank's minimum capital ratios under a stressed scenario.

The previously approved increase in the bank's dividend would have been the first since the financial crisis, and raising it has been a focus of top executives. Banks historically paid out relatively high dividends, spurring retirees and other investors seeing income to buy their shares.

Banks failed to cut their dividends even as their earnings shrank during the financial crisis, burning up valuable capital and leaving them more vulnerable as the housing market deteriorated. In response, lawmakers have given regulators much more control over banks' plans to return funds to shareholders.

The Fed said Bank of America has 30 days to submit a new plan that corrects the errors and ensures no further reporting problems if it would like to return more money to shareholders over the next four quarters. (http://link.reuters.com/zef88v)

The bank said its new plan will likely be more modest than its prior request. In March, the bank received approval to buy back $4 billion of shares and increase its dividend payout by more than $1.5 billion a year, or 5 cents per share per quarter from 1 cent.

Analysts said the bank would likely increase its dividend as previously planned, while not asking to buy back any shares.

The accounting error stemmed from Merrill Lynch & Co debt, which Bank of America assumed after it bought the investment bank and brokerage during the financial crisis.

The acquisition, which Bank of America agreed to in September 2008, boosted earnings in areas like investment banking and wealth management by billions of dollars, but it has also given management some real headaches. Bank of America agreed to pay $2.4 billion in 2012 to settle allegations that it had misled investors about Merrill's health at the time of the purchase. Additionally, the New York Attorney General's office intends to take action against the bank as a result of an investigation of Merrill's mortgage bond practices.

THIRD TIME'S THE CHARM?

The bank already had to scale back its requests to return capital in 2014. The Fed said Bank of America's original request would have left it with too little capital to withstand a hypothetical economic crisis, and it asked the bank to tweak its request. (http://link.reuters.com/byf88v)

Other banks have also stumbled during their efforts to win approval to pay more capital to shareholders. Citigroup Inc's plan was rejected after the Fed took issue with the bank's ability to assess risks and capital needs. Goldman Sachs Group Inc also had to adjust its initial proposed payout to shareholders to win approval from the Fed.

The problems Bank of America announced on Monday related to how it calculated the value of structured notes that investment bank and brokerage Merrill Lynch had issued. Changes in the value of the notes have an impact on the bank's earnings and capital levels under generally accepted accounting rules.

But regulators pay less attention to capital levels under GAAP, and focus instead on a measure known as "regulatory capital," which strips out changes in the value of the notes. In making that adjustment, the bank failed to account for the fact that some of the notes had matured or were redeemed.

Story continues