Donald Trump's polices will end up hurting his voters, a former U.S. Treasury secretary said on Wednesday.

"The people who will be the victims of populist policies are the lower income and middle class people in whose name the policies are offered," Larry Summers said at the World Economic Forum in Davos.

Even Trump's attempts to bring manufacturing jobs back to the U.S. would be damaging, he said.

Summers, a Harvard professor who was President Clinton's Treasury secretary and director of President Obama's National Economic Council, said pressuring companies to relocate to the U.S. would backfire.

Final tally: Obama created 11.3 million jobs

"Yes, our president-elect has made four or five phone calls to four or five companies, largely suspending the rule of law and extorting them into relocating dozens -- or perhaps even a few hundred jobs -- into plants in the United States," Summers said.

But he warned that the decline in the value of the Mexican peso against the dollar -- which he said was a direct consequence of Trump's rhetoric -- will end up costing the U.S. many more jobs.

"It is a major change in the relative attractiveness of locating production activity in Mexico, versus locating it in the American heartland, and the consequence of that is measured not in the dozens or hundreds, but in the thousands or ten thousands or even hundreds of thousands of jobs," he said.

Davos 2017: Complete coverage

Summers was speaking during a debate on rising inequality and the series of anti-establishment votes in 2016.

The head of the International Monetary Fund, Christine Lagarde, said that in supporting Brexit and Trump, voters have sent a very strong message to leaders that they have to do more to reduce inequality.

"There are things that can be done, fiscal, structural reforms, monetary policies... it probably means more redistribution than we have in place at the moment," Lagarde said.

Much hinges on the specifics of the reforms, however. Nobel Prize-winning economist Joseph Stiglitz said Trump's proposed tax cuts could actually kill jobs.

"The trade deficit will get larger and that means overall, you might save 100 jobs at Carrier, but overall the jobs in manufacturing we lost will get worse," Stiglitz told CNNMoney's Richard Quest.

Trump aide: Stop freaking out about his comments

Summers said the growing gap between rich and poor could only partially explain the success of populism.

"The United States has just elected the world's most visible symbol of conspicuous consumption, that is a bizarre manifestation of a concern about inequality," he said.

He said many people -- "the center of the country" -- felt their concerns were being ignored by governments focused on fighting for people in developing countries, recent immigrants and for minorities.

"They feel that they are not being heard, and they expressed their not being heard in the Brexit vote, in the Italian referendum and of course in the U.S. presidential election," he added.