Americans who are behind on their taxes could be denied a passport, or even have theirs revoked, unless they enter into a payment agreement -- a radical step that forms part of a crackdown on delinquent tax debt legislated by Congress.

The IRS announced last month that it is beginning implementation of procedures aimed at cracking down on those with “seriously delinquent tax debts.”

It defines that as someone owing more than $51,000 in back taxes, penalties and interest, for which the agency has issued a levy or filed a notice and the period to challenge has expired.

The procedures stem from the 2015 Fixing America’s Surface Transportation (FAST) Act, which requires the IRS to inform the State Department of those who are seriously delinquent. In turn, the State Department is required to deny the application for or the renewal of a passport.

IRS SAYS THIEVES ARE STEALING REFUNDS FROM TAXPAYERS

The IRS states that the State Department can also revoke that passport "in some cases." A State Department spokesman confirmed the guidance and added that the law also allows the department to deny a passport to those who provide incorrect or invalid Social Security numbers.

The procedures raised concerns about whether such a move would infringe on Americans’ freedom to travel.

“One of my civil liberties concerns here is that the right of citizens to travel and emigrate would seem to be a basic right of citizenship. But by denying passport issuance, the government would be denying citizens the right to travel,” Chris Edwards, director of tax policy studies at the libertarian Cato Institute, told Fox News.

The National Review’s Kevin Williamson called on Congress to undo the legislation.

“Suspending passports in the course of a civil dispute — a civil dispute that may well be in litigation or soon to be in litigation — is banana-republic, totalitarian stuff,” he argued.

Edwards also expressed concern about potential risk of privacy leaks from the State Department and whether that data would spread to other agencies.

“Will the State Department become yet another leaky sieve of personal information of Americans that hackers can access?” he asked.

The Taxpayer Advocate Service, in a blog post on the IRS website in June, notes that there is a history of denying passports to incentivize behavior. It cites a 1996 law that requires the State Department to deny passport applications and revoke passports if someone was delinquent on child support.

However, TAS raised concerns that since there are only two forms of notice to affected taxpayers, it could be challenged as unconstitutional.

“Unlike the [notices] in the child support context, currently, the IRS does not plan to provide any additional, direct notice to affected taxpayers beyond the statutory requirements,” the blog states.

“I believe this lack of notice may not satisfy taxpayers' due process rights under the Fifth Amendment of the Constitution because taxpayers do not have a meaningful opportunity to contest the certifications prior to them taking place.”

The IRS says those at risk can avoid the State Department being notified if they pay the debt in full or agree to a payment plan. It also won’t affect those in bankruptcy, serving in a combat zone or those for whom the IRS has determined the debt is not collectible “due to hardship.”

The State Department spokesman noted that passports can also be issued in emergency situations or for humanitarian reasons, while citizens overseas are eligible for a limited validity passport so they can return to the U.S.