Mitt Romney’s flirtation with a third presidential run was brief. But it lasted just long enough to raise a key issue that could become a central part of the 2016 race.

Earlier this week, Romney and President Obama traded barbs over poverty, amid growing calls by Republicans to address inequality.

“Even though their policies haven’t quite caught up yet, their rhetoric is starting to sound pretty Democratic,” Obama said on Thursday, referring to the GOP. He alluded to Romney being “suddenly deeply concerned about poverty,” adding: “That’s great, let’s go. Let’s do something about it.”

Romney shot back at the president late Thursday night. “Mr. Obama, wonder why my concern about poverty? The record number of poor in your term, and your record of failure to remedy,” he tweeted.

Such attacks are likely to pile up as the 2016 race gets going, just as they did during the last presidential election, when Newt Gingrich called Obama the “food stamp president.”

But then, as now, the attacks don’t tell the full truth of the president’s record on poverty.

“Obviously the president came into office at a time of terrible economic conditions,” says Michael Strain of the American Enterprise Institute. “We have mandatory spending programs that are means-tested, and whether it was Bush in office or Obama, food stamp rolls would have come up.”

Obama expanded the safety net through his 2009 stimulus, but the legislation actually prevented even more Americans from falling into poverty through larger tax credits, emergency unemployment benefits, and expanded food assistance. The Center for Budget and Policy Priorities (CBPP) estimates that the recovery act ultimately kept about 7 million people above the poverty line.

That said, it’s still true that a record number of Americans were in poverty during the worst of recession: In 2010, 46.2 million were below the federal poverty line, which is the largest number in more than 50 years, when the government first started publishing the numbers. But the actual proportion of Americans in poverty didn’t hit record levels. It rose slightly above 15% at the peak of the crisis. But as Politifact notes, “it was between 17.3 percent and 22.4 percent in the seven years between 1959 and 1965 under Presidents Eisenhower, Kennedy and Lyndon Johnson.”

What’s more, an increasing number of experts point out that the official poverty rate doesn’t capture the full picture, as it doesn’t account for tax breaks and in-kind benefits; the increase is less dramatic when that assistance is factored in.

The more important question to ask is whether Obama could have done more to alleviate poverty during the worst of the recession. And experts across the political spectrum agree that more could have been done. “Right at the beginning, private economists, government forecasters, and the Obama administration all underestimated how strong the recession was, and they could have asked for a stronger stimulus,” said Arloc Sherman, senior fellow at the CBPP.

Strain similarly believes that the stimulus could have been designed better. And he faults Obama for focusing on passing health-care reform instead of passing a second stimulus or otherwise focusing on job creation. That said, the president was hardly the only one at the helm: It wasn’t the Republicans’ priority either. “Both the president and the Republican party really weren’t focused on jobs,” Strain says.