ZURICH (Reuters) - At least 10 Swiss banks have withdrawn from a U.S. program aimed at settling a tax dispute between them and the United States, Swiss newspaper NZZ am Sonntag said on Sunday, quoting unnamed sources.

Around 100 Swiss banks came forward at the end of last year to work with U.S. authorities in a program brokered by the Swiss government to help the banks make amends for aiding tax evasion.

"At least 10 banks that had decided at the end of 2013 to pay a fine have withdrawn their decision," NZZ am Sonntag said, quoting unnamed lawyers and auditors. It did not name the banks concerned.

The newspaper said the banks were convinced they had not systematically broken U.S. law and lawyers of the U.S. Department of Justice had actually been surprised to see them take part in the program and did not object to the banks leaving the program.

Liechtenstein-based VP Bank came forward last week to say it had concluded that it no longer needed to take part in the program.

About a dozen Swiss banks face a U.S. criminal investigation as part of the tax probe. Credit Suisse has already settled by agreeing to plead guilty and pay a fine of $2.6 billion earlier this year.

(Reporting by Silke Koltrowitz)