Story highlights Panetta writes that Obama was not forceful enough to try to keep a residual force in Iraq

The White House has said it couldn't convince then-Prime Minister al-Maliki to agree

A U.S. presence in Iraq might have prevented ISIS's growth, Panetta said

Panetta also describes a President of half-steps and hesitation when it came to Syria

A former senior member of President Barack Obama's national security team is panning the administration's decision to rule out the use of ground troops to fight ISIS and questioning Obama's leadership style.

"I take the position that when you're commander in chief that you oughta keep all options on the table ... to be able to have the flexibility to do what is necessary in order to defeat the enemy," former Defense Secretary and CIA Director Leon Panetta told CNN. "We're conducting airstrikes. But to make those airstrikes work, to be able to do what you had to do, you don't -- you don't just send planes in and drop bombs. You've gotta have targets. You've gotta know what you're goin' after. To do that, you do need people on the ground."

Panetta's comments are a stinging rebuke of Obama at a crucial point in his administration as the President battles multiple national security threats, including ISIS, a resurgent Russia and the spread of Ebola.

His memoir, "Worthy Fights," describes a White House that did not use its "leverage" to try to keep a residual force in Iraq.

"Those on our side of the debate viewed the White House as so eager to rid itself of Iraq that it was willing to withdraw rather than lock in arrangements that would preserve our influence and interests," he wrote in the book.

The White House has argued it could not convince then-Prime Minister Nouri al-Maliki to come to terms.

But Panetta told CNN the White House did not use its power to pressure al-Maliki enough.

JUST WATCHED Secret Service's casual response to crash Replay More Videos ... MUST WATCH Secret Service's casual response to crash 01:54

JUST WATCHED Panetta: Obama ignored my Iraq advice Replay More Videos ... MUST WATCH Panetta: Obama ignored my Iraq advice 02:17

JUST WATCHED Gates, Panetta slam Obama on Syria Replay More Videos ... MUST WATCH Gates, Panetta slam Obama on Syria 03:27

JUST WATCHED ISIS blame game: U.S. intel vs. Obama Replay More Videos ... MUST WATCH ISIS blame game: U.S. intel vs. Obama 02:14

"What I'm saying is that Maliki was the kind of leader that you had to constantly put pressure on to direct him in the right direction," he said. "We had, with Iraq -- made a commitment with regards to military assistance, F-16 fighter planes, other types of military aid, that I think if we had said, 'Look, you know, if you're not gonna give us -- the agreement that we need to maintain our forces there, you know, we may not provide this kind of assistance."

The White House, he said, needed to do more "to try to push him."

"You need to threaten guys like that, who won't come along. And everybody knew that," he said. "I think what happened was -- is that because Maliki kept resisting this effort, that there was a sense that, 'Look, why should we want this more than the Iraqis want this?' And if they're -- if they're putting up a fuss about this, then -- we might as well just pick up and leave."

Panetta describes a leadership vacuum on the issue, calling it a lack of "active advocacy" by the President on the issue. In the end, he said, "I think the kind of push and direct involvement that I think could have had an impact simply never developed."

As to whether ISIS would be as much a threat today had a residual force been left in Iraq, he told CNN, "I do think that if we had had a presence there, it might not have created the kind of vacuum that we saw develop in Iraq."

Panetta was aligned with much of the President's national security team on this argument -- and on the question of arming the Syrian rebels in 2012, which the President also resisted.

"I mean, it's understandable," Panetta said, explaining Obama's argument that the weapons could get into the wrong hands. "But at the same time, if we're going to influence the rebel forces, if we're gonna try to establish a moderate element to those forces, so that they can, in fact, provide leadership not only to confront (Syrian President Bashar al-) Assad, but also to help us in terms of a political transition, that it was important to provide this kind of assistance in order to have some leverage over what they -- they -- they were gonna do."

There was honest disagreement -- but then no decision.

"To a large extent it wasn't that the President kind of said -- said, 'No, we shouldn't do it.' The President kind of never really came to a decision as to whether or not it should happen," Panetta said in the interview. "I think it basically sat there for a while and then got to the point where everybody just kind of assumed that it was not gonna happen."

In the book, he described some hesitation and half-steps regarding this issue.

"I mean, it was that kind of just a hesitation to really, you know, do what needed to be done. Now, you know, don't get me wrong," he said. "I think he was very strong in terms of the war on terrorism. And he made some tough decisions. But there were these decisions that basically never were confronted, that I think-- in many ways-- contributed to the problems we're facing today."

Panetta told CNN the President is now taking appropriate, decisive action on ISIS -- albeit a bit late.

JUST WATCHED 'Poor Jim Clapper got thrown under bus' Replay More Videos ... MUST WATCH 'Poor Jim Clapper got thrown under bus' 02:51

JUST WATCHED Christie 'disturbed' by Obama on ISIS Replay More Videos ... MUST WATCH Christie 'disturbed' by Obama on ISIS 00:45

JUST WATCHED Ryan: Ground troops must be on the table Replay More Videos ... MUST WATCH Ryan: Ground troops must be on the table 05:21

"He's made the decision to put troops on the ground in Iraq, to try to help the -- security forces," he said. "He's made the decision to -- arm and train rebel forces in Syria. And he's made the decision to conduct air attacks. So in many ways-- he's made the right decisions now. I think those decisions should've been made two years ago."

The portrait Panetta sketches of Obama sometimes looks more like a professor than a President.

"He relies on the logic of his presentation, with the hope that ultimately people will embrace that logic and then do what's right," he said. "You know what? In 50 years, my experience is logic doesn't work in Washington. You gotta basically go after people and make them understand what they have to do. And that means you create a war room. You go after votes. You have to push people."

Panetta described to CNN a sense the President found that work unrewarding at the very least, even distasteful.

"I think it -- it offended him that people would not really get serious and work on the issues," he said. "I think as a result of that, he just felt, you know, 'How -- how can I deal with people that simply won't -- don't wanna do the right thing for the country?' Well, the reality is, if you wanna govern in this country, you have to deal with people you don't like."

While Panetta had left the administration by 2013, he argues that the President made a mistake by not acting when Syria used chemical weapons against its own people -- violating his own clearly drawn red line. "I think at that point the President very clearly should've said, 'You have crossed that red line. And we're not gonna allow that to happen.' And I think initially my sense was they were gonna do exactly that. But somehow they backed away from it." The airstrikes against Syria did not happen.

It's clearly a struggle for Panetta -- who said he admires and likes the President -- to reconcile the decisive Obama who approved the raid against Osama bin Laden with the President who vacillated over Syrian airstrikes. "A President that made the decision to go after bin Laden, and made a very gutsy decision to do that, in spite of whatever was being recommended to him, but he -- he knew that this had to be done. And I really respected that decision. I just could not have imagined him -- not making the same decision with it came to the credibility of the United States on drawing that red line in Syria."

To no one's surprise, the White House has not warmly welcomed Panetta's version of history. Vice President Joe Biden last week said it was "inappropriate" for former administration officials to write books criticizing the policies of the White House. "At least let the guy get out of office first."

Asked if Biden has a point, Panetta told CNN, "You know, I'm of the view that you don't put a hold on history. History is what it is," he said, adding, "I recommend the President and Vice President Biden take the time to read the book. Because I think -- you know, when you read it, it's a pretty balanced presentation of what happened."