(CNN) A 23-year-old New York man remembers being "dramatically" woken up by a flight attendant who "demanded" he exit the aircraft.

Believing the plane was undergoing an emergency evacuation, he complied, only to realize later that he was being singled out.

The man, identified only by the initials "W.H" in a lawsuit filed Monday, along with three of his friends, Shan Anand, Faimul Alam, and another friend identified by the initials M.K., are alleging they were asked to leave an American Airlines flight from Toronto to New York in early December because of their appearance.

Three of the four men ejected from the flight are Muslim; another is Sikh. Three of the four are of South Asian descent and one is of Arab descent. All the men are U.S. citizens in their 20s and all four were sitting near the front of the plane, the lawsuit states.

Removed from plane without explanation

When they asked the flight crew why they were being removed, the flight attendant told them to exit "peacefully" and "demanded" they return to the gate and await further directions, according to the lawsuit.

"It basically made me feel like a criminal," W.H. told CNN in a phone interview. "It was like I was put on a pedestal where everyone is pointing at you. I was frightened that they were frightened."

It was only after the plane took off that an airline agent told the men "they could not board because the crew members, and specifically the captain, felt uneasy and uncomfortable with their presence on the flight and as such, refused to fly unless they were removed from the flight." When the group asked the agent whether their appearance had contributed to their removal, "being that they are dark skinned and had beards," the agent responded that their appearance "did not help," the lawsuit said.

Two of their other friends, one Hispanic and one Pakistani, were assigned seats in the rear of the aircraft and were not asked to leave the original flight. All four men were allowed to board a subsequent flight after that flight's captain allowed them to board, the lawsuit said.

The four men are seeking damages, alleging the airline "disgracefully engaged in the discrimination ... based on their perceived race, color, ethnicity, alienage and/or national origin," the lawsuit reads.

The flight was operated by Republic Airways, a regional partner of American Airlines.

A spokesman for Republic Airways declined to comment on the suit. American Airlines is reviewing the lawsuit, according to spokeswoman Victoria Lupica.

"I hope this suit serves as a reminder that discrimination is not OK and never will be and persons in authoritative positions, such as the defendants in this matter, should act based upon credible information and not ignorance," their attorney, Tahanie Aboushi, wrote in an email to CNN.

Rights group sees rise of anti-Muslim sentiment

Ibrahim Hooper, the National Communications Director for the Council on American-Islamic Relations, expects these types of incidents to continue.

"I think it is symptomatic of the overall rise of anti-Muslim sentiment in our society. We saw a multitude of these incidents after the San Bernardino (California) shootings. I think these incidents will only increase, unfortunately," Hooper said. "It's this whole thing where the flight crew is uncomfortable or the passengers are uncomfortable. Why are they uncomfortable? Because of a perceived faith and ethnicity that leads to them being thrown off planes. It's very troubling."

The men are suing for $1 million in compensatory damages and $5 million in punitive damages, according to the lawsuit.

W.H. was encouraged by the captain of the second flight, the one he was allowed to board with his three friends, to speak up about the incident.

"The captain told me we shouldn't let this pass over. It's not right, it's discriminating. He said, if I was you, I wouldn't let this go," W.H. said.

"You see these kinds of things happen, but you don't really get it until it happens to you. We don't want this to happen to anyone else."

Anand said, "Seeing a mother holding her child closer to her, looking at you in fear...those stares stay in your head."

"When I was coming off the plane, I kept thinking, 'What did we do?' But it was just because we looked a certain way," he said.