(CNN) Concepción and Margarito Silva were hoping to enjoy July 4th with their daughter and son-in-law at Fort Drum, an Army base in upstate New York.

But their Independence Day ended in a federal detention facility after the military base reported the grandparents, who live in Brooklyn and are originally from Mexico, to Immigration and Customs Enforcement for being in the US without proper documentation.

The detention of the Silvas -- who have no apparent criminal record and have significant health problems, their children said -- has caused outrage among immigrant rights groups who see it as another example of how the Trump administration's hardline immigration policies are harming families.

"This is yet another case of rogue ICE and CBP agents, in concert with military personnel, using all of their energy to tear apart immigrant families and put our loved ones at grave risk," said Javier H. Valdés, co-executive director of Make the Road New York , a group that supports immigrant causes.

Perla Silva, one of the couple's children, says the family is devastated.

"We are most worried for my mother and my father, who have serious health issues," she said in a video posted by Make the Road New York. "My father ... is the only father that my daughter has ever known. She misses him dearly. Please ICE, do the right thing for me and my family. We ask for them to just come home safe."

BREAKING: Perla Silva, daughter of Concepción & Margarito Silva, grandparents detained at Fort Drum, calls on ICE to immediately free her parents. "Our family is devastated...Please ICE, do the right thing for me & my family. We ask for them to just come home safe." #FreetheSilvas pic.twitter.com/YDv4WHLjGY — Make the Road NY (@MaketheRoadNY) July 9, 2018

The incident began when Concepción and Margarito Silva went to Fort Drum to visit their daughter and her husband, a sergeant in the 10th Mountain Division who has twice been deployed to Afghanistan, according to CNN affiliate Spectrum News NY1.

The Silvas attempted to enter the military base near Watertown, New York without approved identification, according to Julie Halpin, spokeswoman at Fort Drum. They presented their municipal New York City ID cards, which were not deemed sufficient, Spectrum News NY1 reported

Security personnel found a discrepancy with their passports and then called US Border Patrol, a Customs and Border Protection spokesman said.

The couple admitted to Border Patrol agents that they were in the US illegally, said the CBP spokesman, who added that they were charged with being present in the US without admission or parole.

The CBP spokesman said the husband and wife had medications with them and were allowed to take them when needed and store them in a refrigerator.

The Silvas were taken to the nearby Wellesley Island Border Patrol station, where they received a special meal prepared by their relatives because of their medical conditions, the spokesman said.

The couple were then transported to the Buffalo Federal Detention Center in Batavia, New York, and turned over to ICE Enforcement and Removal Operations to await a hearing with an immigration judge, the CBP spokesman said.

Concepción and Margarito Silva came to the US about 20 years ago, family members say.

Eduardo Silva, one of the couple's children, said that his parents have been in the US for about two decades and obtained a work permit from the Labor Department in 2007.

"All my dad ever does is work, work, work, work," he told NY1. "He doesn't go hang out, he doesn't do this and that, no. He just works for his family. And this is his first time trying to take a vacation, and this happens to him, and it's not fair."

He said he was primarily worried about their health issues and how this will affect them.

"She did not need this stress, and the more stress she gets, the closer for her to dying," he said through tears.

Concepción Silva had heart surgery two weeks ago, according to a GoFundMe page created by her daughter, Silva Perla.

The incident has drawn comparisons to the detention of Pablo Villavicencio, a pizza delivery man who was turned over to immigration officials after he tried to drop off food at a military base in Brooklyn, New York. The undocumented immigrant from Ecuador similarly showed his New York City ID to security and then was detained.

Villavicencio, whose wife and two daughters are US citizens, was set to be deported until a federal court temporarily blocked his deportation.