The U.S. government recently removed four law-abiding Muslim men from the no-fly list just days before a New York federal district court hears their case, scheduled for Friday.

Tanvir v. Lynch alleges that the FBI used the no-fly list to coerce Muslims into becoming informants on their community and that the government retaliated against them when they lawfully opted not to.

“The fact that the government has confirmed that all four of our clients now can fly really affirms our claims in this lawsuit that the only reason they were ever on a no-fly list is … they were refusing to be informants. There was never any valid reason for their placement,” said Diala Shamas, a senior staff attorney at CLEAR (Creating Law Enforcement Accountability and Responsibility) at the City University of New York School of Law, which brought the lawsuit along with the Center for Constitutional Rights and Debevoise & Plimpton.

According to the suit, Muhammed Tanvir, a resident of the New York City borough of Queens, was not allowed to travel after he refused multiple requests from FBI agents to work on their behalf, saying it would violate his religious beliefs. Furthermore, he said he had no useful information to share.

He was first approached by the FBI in early February 2007, and the suit alleges that the harassment continued for years and included a temporary confiscation of his passport and threats to deport him to Pakistan, where his wife and son lived.

The suit also alleges that the FBI believed him to be a “special,” “honest” and “hardworking” person and offered him incentives, help with his family’s travels to the U.S. and financial assistance for his parents to take religious trips to Saudi Arabia.

Tanvir believes he was placed on the no-fly list sometime before October 2010, after he refused to speak to the agents further.

“Had Mr. Tanvir actually presented a threat to aviation safety, [FBI agent Sanya] Garcia would not and could not have offered to remove Mr. Tanvir from the list merely in exchange for his willingness to become an informant,” the suit states.

The three other men included in the suit — Jameel Algibhah, Naveed Shinwari and Awais Sajjad — believe they too were prevented from traveling after refusing to work on behalf of the FBI, or they found themselves unable to travel and were told by the FBI that they could be removed from the list if they agreed to work as informants.

“I haven’t seen my family in a long time. My youngest daughter doesn’t even know me,” said Algibhah, whose wife and daughters live in Yemen, according to a Center for Constitutional Rights press release issued after the men were notified that they would again be able to fly in the U.S.

Sajjad said he would like to travel to Pakistan to see his ailing grandmother. Shinwari has been unable to visit his wife and family in Afghanistan since 2012.

The men say their First and Fifth Amendment rights to freedom of speech and association as well as freedom of religion and due process, among others, have been violated.

The lawsuit names among the defendants the attorney general, FBI Director James Comey and more than a dozen FBI agents, some whose identities are not known.