“It was a good opportunity to meet, shake hands,” he told reporters. “The tone was very matter of fact. There was no yelling. We didn’t throw shoes at one another.”

Mr. Tillerson acknowledged that international inspectors have found that Iran “is in technical compliance with the agreement, and no one around the table took exception to that.” But he argued that Iran was violating the larger aspirations of the deal by engaging in destabilizing activities not directly covered by it, like supporting terrorist groups.

He also conceded that prospects of persuading the other powers, much less Iran, to revisit the deal were daunting, even as he said he remained optimistic. When negotiating as chief executive of Exxon Mobil, Mr. Tillerson said he had learned that “it always gets the darkest before you might have a breakthrough.”

No breakthrough was apparent on Wednesday evening. Federica Mogherini, the foreign minister for the European Union who led the 90-minute meeting, rejected scrapping or renegotiating the agreement. “The international community cannot afford dismantling an agreement that is working and delivering,” she told reporters outside the Security Council chamber.

“This is an agreement that prevented a nuclear program and potentially prevented military intervention. Let’s not forget that,” she added. “There is no need to renegotiate parts of the agreement, because the agreement is working.”

Mr. Tillerson outlined the Trump administration approach in a television interview on Tuesday. “The president really wants to redo that deal,” he told Fox News. “We do need the support, I think, of our allies, the European allies and others, to make the case as well to Iran that this deal really has to be revisited.”

Two provisions he focused on involve the expiration of the agreement and its failure to stop Iran from developing ballistic missiles. Under the deal, some provisions expire, or “sunset,” after as few as 10 years while others are in force longer and some are permanent. And although United Nations provisions seek to limit ballistic missile technology, the nuclear agreement does not prohibit Iran from developing such weapons.