“There’s some nervousness by us that the Iranian government might say, ‘We can’t get visas to go there, so no Americans can come here, either,’” Mr. Fraser said.

While Olympic boycotts have resulted from tense political differences between nations, opposing countries have also long found common ground on playing fields, on the track and in sports arenas. One of the most celebrated examples is the so-called Ping-Pong diplomacy that helped foster the relationship between the United States and China in the 1970s.

In wrestling, the United States, Iran, Cuba and Russia banded together in 2013 to persuade the I.O.C. to keep the sport in the Summer Games. American wrestlers and officials are warmly welcomed in Iran, and Iranian wrestlers compete regularly in the United States. They may be invited to meets in May in New York and in June in Los Angeles, Mr. Fraser said. There is uncertainty now, however, about whether they would be granted P1 visas, commonly known as sports visas, to compete.

In 2014, Christina Kelley, the chief international ambassador for U.S.A. Wrestling, became one of the few women allowed into a wrestling arena in Iran since the 1979 Islamic Revolution. She said on Saturday that she was frustrated by Mr. Trump’s decision.

“I don’t think our current president has any clue what the State Department and what sports diplomats and cultural exchanges do for our country and for the safety of our people around the world,” Ms. Kelley said.

The ban on visitors from the seven nations — Iran, Iraq, Libya, Somalia, Sudan, Syria and Yemen — comes at a delicate time for the U.S.O.C. Los Angeles is seeking to host the 2024 Summer Games, and it will learn in September whether it, Paris or Budapest will get the Games.

(There is some speculation that the I.O.C. will award the 2024 Games to Paris and the 2028 Games to Los Angeles, but the U.S.O.C. remains committed to the bid for the 2024 Games.)