The $100 bill should be scrapped to make things more difficult for tax cheats and other criminals, according to former Treasury Secretary Larry Summers.

Summers cites a study just published by researchers at Harvard business school arguing that eliminating large denomination bills would have little downside for legitimate commerce, but it would make life far more difficult for those breaking the law.

The report calls for the end of not only the $100 bill, but also the 500 euro note, the UK's 50 pound note and the 1,000 Swiss franc note.

"In certain circles the 500 euro note is known as the 'Bin Laden' [which] confirms the arguments against it," Summers writes in column in the Washington Post on Tuesday. "Illicit activities are facilitated when a million dollars weighs 2.2 pounds as with the 500 euro note rather than more than 50 pounds as would be the case if the $20 bill was the high denomination note."

Summers says recalling all the notes now in circulation probably will not happen. But he argues "a moratorium on printing new high denomination notes would make the world a better place."

Related: Does the $100 bill need to go?

The $100 bill is actually the second most common U.S. bill in circulation, behind the $1 bill. It's 25% more common than the $20 bill, according to the most recent figures from the Federal Reserve. And the number of 100s has increased by 350% in the past 20 years.

Nearly 80% of the $1.4 trillion of U.S. currency in circulation is in $100 bills. Much of it is held outside the United States.

Related: Death of the 500 euro note getting closer

Summers says the greatest objections to dropping the large denomination bills come from European countries.

"Because the Germans were committed to having a high denomination note, the issue was never seriously debated in international forums," he said, recalling when the European currency was being designed in the last 1990s.

But European Central Bank President Mario Draghi said this week that the central bank may drop the note.