President Obama says he's all about the middle class.

He's labeled his agenda, Middle Class Economics, and released a raft of plans that he says will help these families get ahead. To pay for it, he'd tax the rich.

But not everyone in the middle class would benefit from Middle Class Economics. It's heavily weighted in favor of married couples and parents of young children.

"Many people in the middle class will get no benefit from the president's proposal," said Roberton Williams, a fellow at the Tax Policy Center. "Among the middle class, it's targeted at people with kids and second earners. Virtually no single middle class people without kids will get anything."

The White House and some liberal policy groups, however, contend that the center's methodology wrongly labels some affluent Americans as middle class.

Related: What you need to know about Obama's proposal for the middle class

It's tough to gauge how the tax plan would actually help the middle class because the averages can be deceiving. Many people on both sides of the aisle pounced on the center's recent analysis that showed middle income people, making between $49,000 and $84,000, would pay an extra $7 in tax, on average.

Drilling down, however, reveals that the center added an average $19 tax bill to everyone in that bracket to account for a tax on banks, which ultimately is borne by all Americans.

A different picture emerges when one looks at just the individual income tax provisions. These include a credit for working couples and expanding the child care tax credit.

That's why taxpayers with children come out the big winners under the president's plan. Nearly 51% of middle class filers with kids would receive a tax cut. These taxpayers would receive a $329 decrease, on average.

Some 45% of married folks who file jointly would get a tax break. This group would see an average cut of $200.

Related: How Obama would close the 'trust fund loophole'

Far fewer middle class single and elderly taxpayers would benefit from Obama's plan.

Only 12.5% of single filers would get a tax cut. Overall, this group would see a $61 increase, because nearly 7% of middle class singles would see their taxes go up and that skews the overall average.

Among the elderly, only 10% would enjoy a dip in their taxes. But because many in this group would be hit with another of the president's provisions -- that would require estates to pay capital gains on appreciated assets -- they would pay an additional $152, on average.

Overall, fewer than one in four middle class taxpayers would benefit from Middle Class Economics, according to the center's analysis.

Still, White House Office of Management and Budget Director Shaun Donovan and some left-leaning groups say the Tax Policy Center overestimates the tax hikes on the middle class.

The center doesn't factor in unrealized capital gains into its income ranking, argues the Center on Budget and Policy Priorities. So people with moderate incomes but considerable assets are considered middle class. A very small number of these taxpayers would be hit hard by the proposed capital gains tax on estates, which skews the overall average.

Donovan, speaking on NPR Monday morning, pointed to the Department of Treasury analysis, which shows that between a quarter and one-third of families in the $50,000 to $75,000 income range would see a tax change. Their taxes would be lowered by between $450 and $570, on average.

Treasury's analysis, which counts the unrealized gains as income, shows that 99% of Americans paying the capital gain tax have incomes above $500,000.

Also, the Tax Policy Center's analysis doesn't take into account the proposal to make the American Opportunity Tax Credit permanent because it wouldn't happen until after 2017, says CBPP. This would also help middle class families.

As for the rich, the president's plan does accomplish its goal of raising their taxes. Just over three-quarters of the Top 1% would see a tax hike, of $34,500, on average. And more than nine out of 10 of Top .01% would see a tax increase, of $167,250, on average.