At least two undocumented immigrants were detained after leaving a church-run hypothermia shelter in Virginia earlier this month, raising concerns among advocates that similar arrests might cause people to avoid getting much-needed help.

But Immigrations and Customs Enforcement told TIME Thursday that the detainments were a coincidence and that the agency is not targeting churches or other sensitive areas.

At issue is the agency’s “sensitive locations policy,” which bars agents from searching, interviewing or arresting potential undocumented immigrants in places of worship, hospitals, schools, protests, funerals and weddings. Exceptions can be made for a fast-moving investigation or with prior approval from supervisors.

On Feb. 8, immigrations agents arrested two men who had just left a hypothermia shelter at Rising Hope Mission Church in the Alexandria section of Fairfax County. (Advocates say four more men might have been picked up too.) But an Immigrations and Customs Enforcement official told TIME that agents were not staking out the church, that they were in the area looking for a specific person and that the men were arrested after “consensual interviews” on a sidewalk in front of a nearby Aldi supermarket.

“The ICE sensitive locations policy, which remains in effect, provides that enforcement actions at sensitive locations should generally be avoided, and require either prior approval from an appropriate supervisory official or exigent circumstances necessitating immediate action,” said ICE in a statement to TIME. “DHS is committed to ensuring that people seeking to participate in activities or utilize services provided at any sensitive location are free to do so without fear or hesitation.”

One of the men officers arrested was a citizen of Honduras who had previously been removed from the U.S. Immigrations officials says he had a felony drug conviction and a string of misdemeanor convictions dating back to 2000. The other man is from El Salvador, and though an official says he is a legal permanent resident of the United States, he also has multiple misdemeanor convictions including a 2016 charge of driving while intoxicated.

Advocates worry these arrests, as well as others at schools and apartments, will perpetuate fear within anxious immigrant communities. Some are concerned the sensitive-locations policy will be undermined.

“Who knows whether this administration plans on respecting churches, plans on respecting synagogues, and mosques and places of worship,” said immigration attorney David Leopold. “It doesn’t appear that they do.”

“I know from talking to my clients, from talking to universities that I work with and hospitals that I work with—there’s a real fear here that these types of memoranda will not be respected,” he added. “I think it’s all a part of this intentional campaign to instill fear and panic throughout the immigrant community and throughout the larger community by the Trump Administration.”

Rich Morales, an attorney for the PICO National Network, says he is worried about the impact the Virginia arrests could have on churches, where undocumented immigrants sometimes seek refuge. The PICO Network has organized trainings since January with congregations about what providing sanctuary means for churches. The group says over 200 people attended both their first event on Jan. 31 and a follow up event on Feb. 13.

“I would imagine that this is going to have a major impact on our work,” says Morales. “There’s already a lot of fear in the community. We have to make sure that the Administration hears from our clergy that we’re not going to accept this and we’re not going to allow ICE to intimidate our congregations.”

Immigrations officials have skirted the line on sensitive locations before. In 2016, a teen in North Carolina was arrested while waiting for a school bus not far from his home. The teen, Yefri Sorto-Hernandez, said he was arrested as the school bus pulled up. Immigrations and Customs Enforcement, however, said when he was taken into custody they had no idea he was even at a bus stop, according to a report by the Atlantic‘s City Lab. In Illinois, an undocumented Mexican man was reportedly lured out of a church and arrested by agents in a nearby McDonald’s parking lot.

Three dozen protesters affiliated with the Sanctuary Congregations Network gathered at an Immigrations and Customs Enforcement field office in Fairfax on Friday to protest the arrests outside Rising Hope. The group also requested a meeting with Enforcement and Removal Operations to no avail.

“The expansion of the aggressiveness, the escalation of immigration enforcement is happening so fast that you can’t really afford to wait until next week,” said volunteer Mary Smalls. “I know we will be back and I know we will be back with a bigger crowd next time.”

Get The Brief. Sign up to receive the top stories you need to know right now. Please enter a valid email address. Sign Up Now Check the box if you do not wish to receive promotional offers via email from TIME. You can unsubscribe at any time. By signing up you are agreeing to our Terms of Use and Privacy Policy . This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply. Thank you! For your security, we've sent a confirmation email to the address you entered. Click the link to confirm your subscription and begin receiving our newsletters. If you don't get the confirmation within 10 minutes, please check your spam folder.

Contact us at letters@time.com.