ICE officers have released 12 of the infants that were being held at a rural Texas detention center, where immigrant advocates claim they dealt with dirty water, limited baby food and a lack of medical care. The release comes just days after immigration advocates called on the Department of Homeland Security to "intervene immediately."

In an email Monday, ICE said there were 16 infants younger than a year old held at the South Texas Family Residential Center in Dilley, Texas as of Friday, March 1. The status of the remaining four babies is unclear. ICE also said there was another infant under the age of one detained at the Texas Karnes detention center. Both facilities are about an hour away from San Antonio, the nearest metropolitan center.

All the mothers and their infants were released to friends and family members who were were "ready to buy them a bus or plane ticket and receive them in their home," said Katy Murdza, the advocacy coordinator at the American Immigration Council's Dilley Pro Bono Project.

"Every mother I spoke to said that her child was sick in some way," Murdza said in a telephone interview Friday with CBS News. Murdza has worked with immigrants at Dilley for almost two years and had rarely seen infants at the detention center until last week. "It's just really hard seeing all of these very small babies in a detention setting."

The infants were the subject of a complaint filed by immigration advocates to DHS's Office of Inspector General last week.

Three advocacy groups — the American Immigration Council, the American Immigration Lawyers Association and the Catholic Legal Immigration Network, Inc. — also sent a letter to the House Judiciary Committee, which plans to "hold long overdue hearings on conditions behind the doors of immigration detention centers," according to Congresswoman Zoe Lofgren, who represents California's 19th district.

"Conditions in immigration detention are not appropriate places for children, period," Lofgren said in an email Monday to CBS News.

It's not clear why the number of detained babies spiked last week. Prior to their arrival, ICE had an "unspoken rule" that mothers with young babies seeking asylum would bypass the usual detainment process, Murdza said. Typically, immigrants are held in custody until officials can determine they aren't a danger to the community or a flight risk, said Katie Shepherd, National Advocacy Counsel for the Immigration Justice Campaign at the American Immigration Council.

"It's very disturbing and ludicrous that ICE is saying we have to make an assessment as to whether a five-month-old is a danger or flight risk," Shepherd told CBS News.

Murdza suspected that the sudden increase of very young infants was connected to the recent arrival of a caravan at the nearby Eagle Pass port of entry, where about 2,000 migrants showed up last month.

In an email to CBS News, an ICE official said that the increase in detained infants was a result of increased border crossings. "As the number of family units crossing the border into the U.S. has increased, so too has the frequency of those with younger children, including infants," the official wrote.

Doctors who visited the babies in Dilley said that a detention center was no place for an infant.

Many infants lost weight after arriving at the detention center because the Dilley facility has only one type of formula available, and it needs to be special requested, which caused delays in accessing it, said Murdza. Mothers weren't given bottled water to mix with the formula, forcing infants to drink the tap water, which she described as potentially unsafe.

"Our staff doesn't even drink the water here," Murdza said. "It smells like chlorine."

The doctors said the mothers recounted significant difficulty in accessing medical attention while at Dilley.

ICE officials told CBS News via email that immigrants are offered "comprehensive medical care," including "registered nurses and licensed practical nurses, licensed mental health providers, mid-level providers that include a physician's assistant and nurse practitioner, a physician, dental care, and access to 24-hour emergency care."

"That's not corroborated by parents who spend time at Dilley," said Colleen Kraft, the immediate past president of the American Academy of Pediatrics. "They'll go to the clinic and be told they need to come back later."

One eight-year-old boy with "boils all over him and a fever" was told to come back later, said Kraft, who just came back from a trip to Dilley. On the visit, Kraft identified 11 children who needed immediate medical attention, but had been told their issue wasn't an emergency and to come back later.

For infants, the immediate medical attention can mean life or death, said Katherine Ratzan Peeler, a member of the Physicians for Human Rights' Asylum Network. She's been working with immigrants at Dilley for nearly two years.

"A normal cold that wouldn't do much to a child can land an infant in the ICU with a breathing tube," Peeler said in a telephone interview with CBS News.