When Vice President Joe Biden told a Harvard audience last week that it was “inappropriate” for former administration officials to write critically about President Obama before he leaves office, Biden didn’t mention that a full accounting of the Obama policy in Iraq might not reflect too well on him, either.

Biden was referring to former defense secretary Leon Panetta’s just-published memoir, Worthy Fights, an IED now exploding inside the Obama White House and the rest of the political world.

Panetta confirms what has been whispered about for years—that Obama shows “a frustrating reticence to engage his opponents and rally support for his cause.” The president often “relies on the logic of a law professor rather than the passion of a leader,” Panetta writes. In reflecting on the president’s habit of bitching privately about obstructionist Republicans without cleverly confronting them, he notes that Obama “avoids the battle, complains, and misses opportunities.”

Panetta is particularly concerned that Obama didn’t fight harder on the sequestration cuts that Panetta believes were harmful to national defense. But the criticism, now hardening into the conventional wisdom among Democrats, also applies to a variety of other issues, where the president had a habit of leading the Democratic troops into battle with a few speeches but not adopting the single piece of advice Oliver Wendell Holmes gave Franklin D. Roosevelt during his first week in office: “Form your battalion and fight.”

For instance, after pushing hard for the American Jobs Act in 2011 and 2012, Obama all but dropped jobs and infrastructure as a cause. If instead he had made the 2014 midterms a referendum on rebuilding America—rather than on his performance as president—the Democrats would be much better off in November.

Panetta isn’t the first Obama Cabinet member to serve and tell. Another former defense secretary, Robert Gates, early this year published an often critical account of Obama. But Gates is a Republican. Panetta started out as a Republican (and served in the Nixon administration) but changed parties in 1971 and became a longtime Democratic congressman from California, then budget director and chief of staff to President Clinton and Obama’s CIA director before moving over to the Pentagon. Even before his long career in government ended, he was an éminence grise in American public life.

So when Panetta takes shots at the president on both Syria and Iraq, it stings.

I have long been suspicious of arguments about American “credibility,” largely because this was used as an excuse for years for why the United States had to keep fighting the Vietnam War. For the last year I’ve been unconvinced by critics claiming Obama committed some terrible sin by drawing a “red line” on the Assad regime’s use of chemical weapons, then not bombing. I figured that, thanks to Vladimir Putin, Bashar al-Assad stopped using chemical weapons and the consequences of Obama not following through were minimal.

But Panetta makes a good point that the reversal of the Syria policy is making it harder to enlist allies in a coalition against the Islamic State. “There’s a little question mark to, is the United States going to stick this out? Is the United States going to be there when we need them?” Panetta said this week in an interview with Susan Page of USA Today.

Panetta is on shakier ground on the question of arming Syrian moderates. Had the U.S. done so, as Panetta and then-Secretary of State Hillary Clinton favored, even more U.S. weapons would have fallen into the hands of ISIS. The widespread assumption that Obama erred in not intervening more directly on the side of Syrian moderates against Assad doesn’t bear scrutiny. “There was no moderate middle, because the moderate middle are made up of shopkeepers, not soldiers,” Biden said. Now, of course, we have no choice but enlist anyone we can in the fight against ISIS.

But if Obama and Biden were sensible about not arming Syrian moderates, the same cannot be said for their policy toward Iraq.

Panetta rightly faults Obama for not pushing the Baghdad government harder to permit a residual U.S. force to remain in Iraq when troops withdrew in 2011. He told USA Today that this “created a vacuum in terms of the ability of that country to better protect itself, and it’s out of that vacuum that ISIS began to breed.”

In his book, he writes: “To this day, I believe that a small U.S. troop presence in Iraq could have effectively advised the Iraqi military on how to deal with al-Qaeda’s resurgence and the sectarian violence that has engulfed the country.”

The botching of the Iraq account is ultimately the president’s responsibility, but it was actually the vice president who mishandled relations with Baghdad. As I report in my 2010 book, The Promise: President Obama, Year One, Obama dumped all relations with Iraqi officials in Biden’s lap in 2009. For years, Obama didn’t even attend meetings on Iraq. That’s how comfortable he was with delegating the whole thing to Biden.

Beyond time management, one of the main reasons Obama delegated Iraq to Biden was that the vice president had a personal relationship with Nouri al-Maliki and the other Iraqi leaders.

I saw this firsthand. In early 2010 I was sitting in Biden's office interviewing him when Maliki called. I got up to leave and Biden, with his usual graciousness, beckoned me to stay. So for 10 or 15 minutes, I heard one end of the conversation. I was impressed at the time to hear Biden schmooze with Maliki as if he were the governor of Delaware. I thought it compensated for Obama’s weakness in nurturing these personal relationships. It also seemed that Biden had a good grasp of the subtleties of Iraqi politics.

In retrospect, of course, it seems that Panetta is right that we didn’t use our leverage to force Maliki to be more inclusive of Sunni Muslims. Biden made many visits to Baghdad and no doubt get fed a load of bull about inclusion on each one. A full account of why the United States didn’t leave a residual force and why we couldn’t get Maliki to stop excluding Sunnis from his government has yet to be written. It will likely be harmful to the reputations of both the president and vice president.

Neither Panetta nor anyone else can blow up Obama’s legacy. The powers of office are such that presidents can often use events to redefine, even reinvent, themselves. And no honest historical assessment of Obama’s accomplishments, especially in his first two years, will fairly conclude that his presidency failed.

But the fallout from the Panetta book will be significant over time. It sets in concrete a critique of the president’s operating style that has been spreading for years. Only a series of unexpected achievements in 2015 and 2016 will dislodge Panetta’s depiction of Obama as an aloof professor-in-chief who didn’t fight hard enough for his ideas and heedlessly placed his faith in logic, always the first casualty in Washington.