Story highlights Muslim family claims it was "humiliated" for actions taken when it paused to pray atop skyscraper

Empire State Building management company says it will respond in court

Family statement: "We weren't doing anything wrong"

Catholic League statement defends family's right to pray anywhere

A New York family's hope for an enjoyable outing high atop Manhattan ended abruptly when the group was escorted out of the Empire State Building for saying Muslim prayers, according to a lawsuit.

Fahad Tirmizi, his wife, Amina, and their two young children were at the skyscraper's observation deck last July 2 shortly after 11 p.m., a time when the family's "religious beliefs require them to recite the evening prayers wherever they may be at the time," according to a lawsuit filed in Manhattan District Court on Tuesday.

The couple silently knelt in an isolated area of the observation deck, with little foot traffic, the court document said.

While she managed to pray without incident, one guard interrupted and "menacingly poked" Fahad Tirmizi "with his hands and feet several times in various parts of his body," the court document said.

The guard informed Tirmizi he was not allowed to pray while at the observatory, and then "forcibly" escorted his family to the ground floor exit of the building, according to the court document.

The suit names the Empire State Building management company, the building, the security firm, and two unnamed security guards as defendants, alleging that they violated the first and 14 Amendments as well as multiple state and city civil rights laws. The building is owned by a publicly traded company called Empire State Realty Trust.

The family is seeking unspecified monetary damages.

In response, an Empire State Realty Trust spokeswoman, Brandy Bergman, told CNN Wednesday, "The claims are totally without merit and we will respond to them in court."

Phillip Hines, a lawyer for the the family, told CNN, "an unwritten rule or policy was being enforced by security when they removed these people."

"They weren't bothering anybody, they were out of the way, and for them to be thrown out of the building is just an ignorant and shameful exercise in discriminatory conduct," Hines said.

The couple was "shamed, humiliated and embarrassed in front of each other, their children, and the general public," according to the lawsuit.

The family released a brief statement through its attorney.

"We weren't doing anything wrong, we just wanted to enjoy the view like everyone else," the statement said.

The alleged response to the incident drew criticism from other religious groups.

Bill Donahue, president of the Catholic League, released a statement Wednesday, which read in part:

"Muslims who pray in public, including in public accommodations, are simply exercising their constitutional rights. In play are two First Amendment rights: freedom of speech and freedom of religion. We hope that Hines investigates the extent to which security guards are expected to censor religious speech. We wish him, and this innocent Muslim couple, well."