A former Democratic Treasury secretary says President Trump's economic agenda will hurt the economy, not help it.

Robert Rubin, who held the job for four years under President Bill Clinton, tells CNNMoney that imposing tariffs, slashing certain regulations and increasing infrastructure spending without raising federal taxes will stifle long-term economic growth.

"There's the risk that we're going to take some kind of measures that can be very counterproductive," says Rubin, who is on the advisory council for the Hamilton Project, a left-leaning think tank.

Tax cuts for the wealthy, he says, serve "no useful economic purpose."

Rubin, who spoke to CNNMoney from the Brookings Institution in Washington, says Trump's financial stimulus -- infrastructure spending and tax cuts -- may provide a short-term boost to the economy, but at a cost in the future.

Related: Trump's economic boom: 3 red flags

The labor market today is tight, Rubin explains, as evidenced by an unemployment rate of 4.8%. Injecting stimulus into a strong labor market will spark inflation, he says. Add that to tariffs that could raise consumer prices.

"That's a whole set of policies that could be highly detrimental to American workers," Rubin says.

Instead, Rubin advocates career retraining for out-of-work Americans and a path to citizenship for undocumented immigrants so companies don't have to hire as much abroad. All that, he says, would have be to be paid for with higher taxes.

"The reality, no matter who's in office," says Rubin, "we're going to have higher taxes."

Related: Mexico economic minister warns Trump on tariffs

Under Clinton, Rubin was head of the National Economic Council for two years, then served as Treasury secretary from 1995 to 1999. It was a time of strong economic growth, but Rubin was later criticized for deregulation and for failing to identify risks that helped cause the 2008 financial crisis.

After he left government, Rubin joined Citigroup as a top executive. In the decade that followed, Citi ramped up its exposure to risk, notably in the housing market. In November 2008, the bank nearly collapsed before the U.S. government announced a massive rescue package. Rubin stepped down in January 2009.

Today, he supports keeping intact most of the financial reform package known as a Dodd-Frank, another point of disagreement with Trump. Rubin warns that reforms haven't gone far enough to rein in parts of the financial system that are more lightly regulated than traditional banks.

Related: Trump begins dismantling Obama financial regulations

Before joining the Clinton administration, Rubin was a banker at Goldman Sachs. Trump has hired Goldman alums repeatedly for his Cabinet and his circle of close advisers, drawing criticism about a revolving door between Wall Street and the White House.

Rubin says it shouldn't matter, and that the financial background can be an advantage.

"I came from Wall Street to government," said Rubin. "I spent six and half years there and the only focus I ever had and that we ever had was in trying figure out what would be best for the economy." Both he and President Clinton, Rubin noted, were focused on helping middle income and lower income Americans.