The Trump administration may be envisioning a 3 percent to 3.5 percent economic growth rate, but that is just "not in the cards," former Labor Secretary Robert Reich told CNBC on Tuesday.

That's because there are two sources of growth — workforce population and productivity, he explained.

With baby boomers retiring and no boom in productivity, "you just can't do it, mathematically," said Reich, who served under President Bill Clinton.

President Donald Trump campaigned on the promise of tax cuts and deregulation, which he said would spur economic growth.

Last week, The Wall Street Journal reported that the Trump transition team ordered Council of Economic Advisers staffers to predict a sustained gross domestic product growth of 3 percent to 3.5 percent.

"Why not be cautious because if your growth estimates are way out of line or overly optimistic, your budget is going to be overly optimistic and you are not going to make a responsible budget," Reich told "Power Lunch."

CNBC contributor Larry Kudlow, on the other hand, thinks Trump deserves a little "common sense credit."

In fact, he thinks the president is being too conservative in his growth projections because the backbone of Trump's plan is to slash taxes for both small and large businesses.

"This is hugely important. It has been the biggest obstacle to growth and productivity," he told "Power Lunch."

Reich argues that corporations already have cash and profitability has never been so high.

"It's absurd to think that if they have corporate tax cuts that they are going to be more productive. They already are swimming in cash and they are not investing more in productivity," he said.