Washington (CNN) The border may be the closest place there is in the US to a Constitution-free zone; courts have held that it's an area where people have very limited privacy rights and the government has broad authority.

Privacy advocates are now more and more concerned about how that legal gray zone could be used.

The border has also become the primary battleground for President Donald Trump's hard-line positions on immigration. The President has called for beefing up security on the Southern border, instituted a travel ban and called for more "extreme vetting" of would-be immigrants and visitors.

Reports have spread of travelers -- immigrant and US citizens -- being pulled aside at airports for hours for secondary questioning, as well as having their smartphones and laptops searched.

The courts have held that's mostly allowed, even for citizens. Law enforcement at the border may search an individual and belongings without providing much of a reason.

There are some limits. An appellate court has held that conducting a deeper search of electronics, called a forensic search, requires reasonable suspicion of some wrongdoing -- a lower standard than within the US otherwise. Customs and Border Protection and Immigrations and Customs Enforcement also have a set of internal policies , reviewed by the Department of Homeland Security's privacy office, that govern what agents can do, including in special cases like attorney-client privilege or journalist's devices.

Still, experts say the government has always had the most leeway at the border and its ports of entry.

"It's not a Constitution-free zone at the border, but the government's powers are at their highest and people's privacy interests are considered to be the lowest," said Chris Calabrese, vice president for policy at the Center for Democracy & Technology, a privacy and civil liberties advocacy group.

The numbers have been trending up. CBP said in fiscal year 2016 it conducted 23,877 electronic media searches, up from only 4,764 the year before, out of nearly 400 million arrivals. But CBP also said fewer searches happened in February of this year than the one before.

Privacy hawk Sen. Ron Wyden sent a letter to DHS last month asking for answers about such searches. He has not gotten an answer yet, his office says.

"These unlimited searches threaten Americans' security by distracting our focus from terrorists and criminals, and destroy the rights of law-abiding American citizens at the border," Wyden, an Oregon Democrat said. "If a border agent takes your phone or computer and forces you to unlock it, they are allowed to copy the contents of your device - which can amount to pretty much your whole life."

In response to an NBC News reported Monday that 25 citizens, 23 of whom were Muslim, had their phones and passwords demanded, CBP reiterated that such searches were within its authority.

"All travelers arriving to the US are subject to CBP inspection," the agency said in a statement. "This inspection may include electronic devices such as computers, disks, drives, tapes, mobile phones and other communication devices, cameras, music and other media players and any other electronic or digital devices. ... Keeping America safe and enforcing our nation's laws in an increasingly digital world depends on our ability to lawfully examine all materials entering the US."

US citizens and legal permanent residents could be detained or have their devices searched or confiscated, but they cannot be turned away. Other entrants, however, could be sent home if they don't comply with what's asked of them when they try to enter the US.

Privacy advocates are especially concerned that what's being asked could go even farther. Homeland Security Secretary John Kelly told Congress last month that the administration wants to ask for social media accounts and passwords as part of some visa applications.

"We want to get on their social media with passwords," Kelly testified. "If they don't want to cooperate, then they don't come in."

Calabrese said that's an extension of an Obama administration effort to voluntarily collect social media identifiers from travelers, which he warns could quickly be reciprocated against Americans when they travel abroad.

"The government does a really good job for obvious reasons of identifying you at the border," Calabrese said. "So if they take that opportunity to link that information and link it to your social media information ... that's something that could allow them to surveil you after you leave the border."

Calabrese and ACLU border litigation attorney Mitra Ebadolahi both say they hope the courts will catch up to the issue. The Supreme Court has ruled that cellphones are a special category of possessions when you're arrested, and can't be searched the same way a backpack can.

"There's been bad cases in terms of the scope of privacy rights of searches of personal effects and of people trying to cross into the United States, and that's an ongoing tension and something we continue to have concerns about, particularly in a world where the administration is committed to further militarization and build up at the border and has this real expansive notion of what the border is," Ebadolahi said.