One by one, 26 governors came forward Monday to announce that they do not want, and in some cases will refuse to accept, any Syrian refugees in their states as a part of the United States’ refugee resettlement program in 2016.

The governors seized on a Syrian passport found near one of the suicide bombers who attacked Paris on Friday—later revealed to be a fake—as a reason to keep the refugees out.

“Until I can be assured that all potential refugees from Syria have no ties to terrorist organizations, I am requesting that the State Department not resettle any Syrian refugees in South Carolina,” South Carolina Gov. Nikki Haley wrote in a letter to Secretary of State John Kerry that reflected broad security concerns among many governors.

But human-rights attorneys and other legal scholars warned that the governors may not have legal authority to block Syrian refugees from their states, no matter how strongly they feel about the matter. President Obama plans to accept 10,000 Syrian refugees in the coming fiscal year.

“Is it legal? The answer is no,” said J.D. McCrary, executive director of International Rescue Committee, of the governors’ plans to bar Syrian refugees from their states. “Governors can weigh in, but it’s not legally binding.” The IRC resettles refugees in Georgia, whose governor, Nathan Deal, said Monday that he will not accept refugees from Syria based on his worries about potential terrorism.

McCrary described the formal refugee resettlement process as a federally funded, federally run program that begins and ends with the State Department, with input from states but not their approval.

“Ultimately the State Department can make a decision that goes against what governors may say,” McCrary said. “The governors who came out today can be in opposition, but it’s not legally binding.”

Judy Rabinovitz, deputy director of the Immigrants’ Rights Project of the American Civil Liberties Union, came to the same conclusion on the legality of governors saying they’ll keep Syrian refugees out of their states.

“The bottom line is, for the most part, this is political posturing and fear-mongering among the governors,” Rabinovitz said. “It is absolutely clear that if a Syrian refugee is lawfully admitted to the United States and wants to visit one of these states or start living in one of these states, there’s nothing the governors can do to stop them. They’d be violating the Constitution.”

In addition to Georgia and South Carolina, the Republican governors of Texas and Michigan said they will not accept Syrian refugees in the near future. New Hampshire Gov. Maggie Hassan, a Democrat, said the United States should “halt acceptance” of Syrian refugees until the United States can assure Americans the vetting process for incoming Syrians would keep the country safe.

Many of the states whose governors came out Monday, including Texas, Georgia, and Michigan, consistently rank in the top states that accept refugees into their communities. Churches and local nonprofits there often help arriving immigrants secure jobs and apartments, while local schools educate arriving children.

It is at the local level where governors could affect Syrian refugees by withholding state funds for refugee programs. But Muna Jondy of United for a Free Syria said even that approach could be unconstitutional.

“What a state can do is basically refuse to allocate any funding for refugees, thus making the resettlement agencies in a state incapable of supporting the refugees on their own,” Jondy said. “So instead, those agencies choose to resettle in a state that is willing to provide additional support.” But Jondy added that providing funding on the condition it is not for Syrian refugees could potentially open a state to a discrimination lawsuit on the basis of country of origin.

The increasingly contentious issue will move to Capitol Hill this week, where the House Judiciary Committee will hold a subcommittee hearing on Syria’s refugee crisis and its implications for American national security, and members of Congress will have opinions to rival or supersede their governors.

Alison Parker, director the U.S. program for Human Rights Watch, agreed with the emerging legal opinion that it is the federal government, not the governors, that ultimately has the legal authority to decide where refugees are resettled in the country. But she made a larger point that she said she hopes will prevail in the debate.

“The notion of singling out one particular nationality, when the world is besieged by reports of the violence and atrocities these people are fleeing, really is unconscionable,” she said.

—with additional reporting by Tim Mak