By Pam Benson, reporting from Aspen, Colorado

The use of military Special Operations Forces has been a proven success in Iraq, Afghanistan and - with last year's raid on Osama bin Laden's compound - in Pakistan, but that success has some people concerned. Will the forces become the tool of choice for a president?

The former head of the U.S. Special Operations Command told the Aspen Security Forum Thursday he fears there could be a misuse of the highly trained specialists.

"It's a real danger," retired Adm. Eric Olson said. "They come to be thought of as a utility infielder, sometimes a utility infielder with guns, and they may be asked to solve problems that are not necessarily special operations problems."

Olson cited cases of Special Operations Forces being asked to provide security for individuals overseas.

The high demand has strained the elite forces. Last year, Olson warned the units were "beginning to show some fraying around the edges."

With the opposition gaining some ground in Syria in its battle to overthrow the Assad regime, some members of Congress are pressing the Obama administration to do more to help the insurgents, who are still not united or well-organized.

In a quick interview with CNN's Security Clearance on the sidelines of the security forum, Michael Sheehan, the assistant secretary of defense for special operations, said the potential use of Special Operations Forces in Syria is not a focus of his right now.

Sheehan also said there are no U.S. Special Operations Forces in Mali, where internal strife has left the northern part of the country ungoverned and where al Qaeda sympathizers have moved in, a development Sheehan called very troubling.

He said the United States is considering options, but no decisions have been made.

Special Operations Forces are at work around the world, Olson's successor, Adm. William McRaven, told CNN's Wolf Blitzer on Wednesday.

Also from Aspen: U.S. must ‘accelerate’ efforts to help Mali

McRaven said Special Operations Forces are in 79 countries, with the largest contingent in Afghanistan. Sixty-six thousand people are part of Special Operations Command, half of them active forces. Approximately 10,000 to 12,000 are deployed at any given time, with 7,000 currently in Afghanistan and more than 3,000 elsewhere, he said.

Sheehan said the forces can operate anywhere and the numbers can range from just one or two in a country to dozens or several hundreds. The goal is to help the host nations handle their own security operations.