John Williams, president of the Federal Reserve Bank of of New York speaks during Politico's Morning Money lunch briefing in Washington, D.C., U.S., on Tuesday, May 17, 2016.

Higher trade tariffs have had a "relatively small" effect on the economy so far but they have hurt confidence and some business investments, which hurts U.S. employment and economic growth, Federal Reserve Bank of New York President John Williams said on Thursday.

"At least so far the tariffs that have been put in place, by the United States and other countries, when you roll that up into a $20-trillion economy it doesn't have a big effect overall on economic growth or inflation," he told a forum when asked about trade wars.

The "much more important and larger" effect is higher uncertainty for businesses, some of whom have told Williams that investments have been put off, he said. "That's a negative for jobs in the short run...and a factor that slows the economy relative to what it could be."