President Barack Obama recently dismissed his critics who argue that arming the Syrian moderate opposition long ago would have made a difference in the fight against Bashar al-Assad. But louder than ever those critics – some from within his own administration – are saying, “We told you so.”

That notion, the president said, “has always been a fantasy. This idea that we could provide some light arms or even more sophisticated arms to what was essentially an opposition made up of former doctors, farmers, pharmacists and so forth, and that they were going to be able to battle not only a well-armed state but also a well-armed state backed by Russia, backed by Iran, a battle-hardened Hezbollah, that was never in the cards.”

As the Islamic State grows and gains territory in not only Syria but Iraq as well, some within the administration say they feel “extreme frustration” that their long-standing warnings have come true—warnings about the likely gain of extremist groups on the ground in the face of American inaction in Syria. They also reject the hardening narrative that the rise of the Islamic State was inevitable.

“Two years ago we told them if we did not train the moderates it would be the regime versus al-Qaeda and you would have transnational terrorist networks in Syria,” says one administration official. “The real fear they always had was: arming the opposition means empowering the extremists. And now they say, ‘See? We told you,’ and we say, ‘No, you didn’t arm the moderates and you ended up with the extremists.’”

The back-and-forth within the administration on Syria has played out in public and in private since 2011. Both former Defense Secretary Robert Gates and former Secretary of State Hillary Clinton write in their books about the policy debate on whether and how to arm the moderate opposition in Syria as the Assad regime’s crackdown grew increasingly brutal, eventually with backing from Russia and Iran.

Clinton said that she did believe that “the failure to help build up a credible fighting force of the people who were the originators of the protests against Assad—there were Islamists, there were secularists, there was everything in the middle—the failure to do that left a big vacuum, which the jihadists have now filled. They were often armed in an indiscriminate way by other forces and we had no skin in the game that really enabled us to prevent this indiscriminate arming.”

In June, the president said, “The question has always been, is there the capacity of moderate opposition on the ground to absorb and counteract extremists that might have been pouring in, as well as an Assad regime supported by Iran and Russia that outmanned them and was ruthless. And so we have consistently provided that opposition with support. Oftentimes, the challenge is if you have former farmers or teachers or pharmacists who now are taking up opposition against a battle-hardened regime, with support from external actors that have a lot at stake, how quickly can you get them trained; how effective are you able to mobilize them. And that continues to be a challenge.”

But those who supported and continue to back arming of moderate rebel opposition forces say that it is not entirely accurate to label these forces farmers and pharmacists as the president did again in his recent interview. They note that many of the moderates, especially in 2011, were former regime fighters, and that all Syrians are conscripted to do military service.

And they say that the Islamic State extremists they are now up against were hardly more qualified when they entered the fight.

“Where were these ISIS guys three years ago? They were not part of Jihadi West Point cadres,” said former U.S. Ambassador to Iraq James Jeffrey. “They very quickly have been trained.”

Others who agree say that there were no guarantees that the plan to arm the moderates would have worked. But as Congress faces the president’s June 26 request to provide $500 million to train and arm Syrian moderate rebels, they take issue with the idea that the plan to provide arms two years ago would have made no difference.

“What we do know and what we did know was that the longer we delayed this, the less likely it was going to work,” said the administration official. “Maybe it would have failed, but it had a much higher chance of succeeding then than now.”

Now some inside the administration are arguing for arming the moderate rebels quickly—as well as for striking the Islamic State group within Syria, where they hold territory and have their headquarters. The biggest fear is that the Syrian town of Aleppo will fall, striking a psychological blow and sparking a massive humanitarian catastrophe. The group is said to be closing in on the city, taking a series of surrounding villages in the last week.

“If you think thousands of Yazidis stuck on a mountain is terrible, wait until Aleppo falls,” said the official.

Jeffrey and others who support arming the rebels say that time is of the essence when it comes to fighting the Islamic State, including in Syria. “You don’t want to wait, you have to use judgment, you don’t want to wait until it is incontrovertible that you are facing a huge threat,” Jeffrey said. “We should be arming and providing air support for and air cover for local people.”

Indeed, Joint Chiefs Chairman Gen. Martin Dempsey said on Thursday that the Islamic State could not be defeated by airstrikes in Iraq without also striking them on the Syrian side of the border.

Others who served in the administration say that criticism surrounding its reluctance to act stems from its “very linear perspective, on or off, all or nothing perception of what military action does.”

“It is hard to prove you prevented something bad from happening; it is always hard to prove that,” said Janine Davidson, senior fellow for defense policy at the Council on Foreign Relations who served until 2012 as the Pentagon’s deputy assistant secretary of defense for plans.

“To have influenced the course of events in Syria would have required a steady, strategic effort focused on identifying a certain group of people in Syria and actively helping that group grow,” Davidson said, noting that nothing would have been resolved either easily or quickly and that both risks and costs were significant.

“People just want quicker fixes than that and there was no guarantee.”