One detained baby remained in ICE custody as of Tuesday evening at the South Texas Family Residential Center, a rural detention facility where immigration advocates say infants are dealing with dirty water, limited baby food and a lack of medical care.

The facility in Dilley, Texas, was holding 16 infants under a year old, including one as young as five months, as of Friday, an ICE official told CBS News via email. By Tuesday, all but one had been released, according to ICE.

One infant was also being held at the Karnes detention center, another facility in Texas, ICE said.

Get Breaking News Delivered to Your Inbox

"The release of these infants is bittersweet," said Katie Shepherd, national advocacy counsel for the Immigration Justice Campaign at the American Immigration Council, an advocacy group. "We remain concerned about the government's documented inability to meet basic standards of care for minor non-citizens in its custody."

The number of babies detained by ICE custody represents a change in the federal agency's policy, Shepherd said in an interview with CBS News on Tuesday. ICE agents previously operated under an "unspoken rule," said Katy Murdza, who has worked with immigrants at Dilley for almost two years and had rarely seen infants at the detention center until last week.

They wouldn't detain particularly vulnerable people, like infants, pregnant woman, and the elderly, she said. Typically, immigrants are held in custody until officials can determine they aren't a danger to the community or a flight risk.

Shepherd and other advocates are concerned that the increase in the number of infants being held may represent a shift in that policy -- that more asylum-seeking babies may be detained by ICE in the future.

"In the past couple of years, we have observed changes in the field -- such as an uptick in the number of pregnant women being held in their 2nd and 3rd trimester, for longer periods of time -- only to see an official policy change by ICE weeks or months later," Shepherd said.

The controversial "zero tolerance" policy, also known as family separation, is another example of a policy that was unofficially in place for months before the Department of Homeland Security officially announced it.

ICE did not answer multiple questions as to whether detaining babies was a new policy. In an email to CBS News, an ICE official said that the increase in detained infants was a result of increased border crossings. "As the number of family units crossing the border into the U.S. has increased, so too has the frequency of those with younger children, including infants," the official wrote.

The recent release of more than a dozen infants from the Dilley facility came just days after three immigration advocate groups called on the Department of Homeland Security's Office of the Inspector General to "intervene immediately." As of Tuesday evening, that office had not responded to their complaint, Shepherd said.

The groups — the American Immigration Council, the American Immigration Lawyers Association and the Catholic Legal Immigration Network, Inc. — also sent their complaint to the House Judiciary Committee, which plans to "hold long overdue hearings on conditions behind the doors of immigration detention centers" in the next few weeks, according to Democratic Rep. Zoe Lofgren, who represents California's 19th district.

"Conditions in immigration detention are not appropriate places for children, period," Lofgren said in an email to CBS News Monday.

The facility has come under fire before for its treatment of especially young children. A Guatemalan woman sued the government last year after her 1-year-old died six weeks after being detained at Dilley. The woman, Yazmin Juarez, alleges her toddler, Mariee, developed a respiratory illness during their detainment and that the facility offered inadequate medical care before releasing her three weeks later.

ICE officials rejected claims that the facility is unfit for babies, pointing to a nearly two-year-old report from the Department of Homeland Security's Office of Inspector General and to a media visit last summer. In an email to CBS News, an ICE spokesperson said "allegations of unsafe and unsanitary conditions at [South Texas Family Residential Center] are utterly false."