Attorney General Jeff Sessions is an outspoken critic of recreational marijuana, and he has the power to hobble cannabis sales in states where it’s legal. But for now, business owners and advocates say they don’t think he’ll actually do it.

As the head of the Justice Department, Sessions has a few strategies he could use to go after marijuana which, while legalized for recreational use in 8 states and D.C. and legalized medicinally in 29, remains a federal crime.

In a directive issued last week, Sessions said he wants to increase asset forfeiture, which allows the government to seize money and property from people suspected of a crime without ever formally charging them with one, let alone convicting them. Historically, asset forfeiture has been used to disrupt cartels, and Sessions said he would use it “especially for drug traffickers.”

But it also means he could send agents to take cash, properties and supplies from cannabis businesses operating legitimately under state law. Even if those businesses sued for their assets back, the case would be lengthy and expensive, and their shops would be effectively closed in the meantime.

“Does it tie in specifically with our industry? I don’t know for sure,” Bruce Nassau, partner in Tru Cannabis dispensary in Colorado and Oregon, says of Sessions’ push for more asset forfeiture. “But it certainly gets one to speculate, doesn’t it?”

Outside of civil asset forfeiture, which bypasses the criminal court system, Sessions could also choose to prosecute anyone involved in the industry, whether that be the owners of dispensaries or just people who do business with them, like the landlords who rent the property for the stores. Nassau’s concern about asset forfeiture gets to an approach many legal experts and cannabis industry spokespeople think Sessions could employ: target a few high-profile businesses to sow fear.

That would make strategic sense, given Sessions’ relatively limited resources to shut down an industry blooming in nearly 30 states, if you include the ones that have legal medical marijuana.

“I wouldn’t be surprised if what Sessions does is settle for enough prosecutions to terrify people and not try to shut down the system systematically,” says Mark Kleiman, head of the crime and justice program at New York University’s Marron Institute of Urban Management. “Not only can’t they protect themselves from being shut down, they can’t protect themselves from being sent to prison for what they’ve already done… These people are taking insane risks.”

Sessions has already signaled his intent to go after pot. He convened a task force to review drug enforcement, which is expected to release its findings soon. He has rolled back sentencing guidelines put in place under his predecessor Eric Holder which called for granting leeway to drug offenders, now saying instead that prosecutors should go after the most serious offense available. The task force is likely also reviewing the 2013 Cole memo, another Holder-era document, which said that the federal government would largely defer to states on marijuana enforcement. What the Justice Department decides to do about the Cole memo will have huge implications for whether or not Sessions cracks down on the drug.

One marijuana advocate even goes to the Department of Justice’s website to look up the memo.

“I periodically check to make sure it hasn’t disappeared,” says Tom Angell, spokesperson for Marijuana Majority.

Sessions has also asked Congress not to renew the Rohrabacher-Farr amendment, in place since 2014, which prevents the federal government from interfering in medical marijuana at the state level. “I believe it would be unwise for Congress to restrict the discretion of the Department to fund particular prosecutions, particularly in the midst of an historic drug epidemic and potentially long-term uptick in violent crime,” Sessions wrote in a letter to Congress first reported by Angell. (“Congressman Rohrabacher has a clear and strong disagreement with his old friend Jeff Sessions,” Republican Rep. Dana Rohrabacher’s spokesman Ken Grubbs told TIME.)

Sessions has numerous formidable legal tools at his disposal, has indicated that he wants to attack both recreational and medical marijuana, and has previously compared pot to heroin. So why aren’t people in the cannabis industry more concerned?

Because legal pot is hugely popular, even among Republicans.

“I don’t see a mass wave of people feeling panicked or making exit strategies or changing their plans,” says Taylor West, deputy director of the National Cannabis Industry Association. “We are seeing a certain amount of optimism that the support for the industry is such that a move to crack down on it would create a bipartisan outcry.”

A CBS News poll from April found that support for legalizing marijuana is at an all-time high. Sixty-one percent of Americans think it should be legal, 71% think the federal government shouldn’t mess with states that have legalized it on their own and 88% support medical use. This includes majorities even among Sessions’ own party: 63% of Republicans don’t think the federal government should interfere with states on this issue.

“Cannabis right now is a helluva lot more popular than Donald Trump,” says Kleiman. And even Trump himself indicated during the campaign that he’d favor leaving it up to the states. “In terms of marijuana and legalization, I think that should be a state issue, state-by-state,” Trump said in a 2015 interview with the Washington Post. “I think medical should happen, right? Don’t we agree? I think so. And then I really believe we should leave it up to the states.”

Along with potentially running afoul of the president, with whom he has already recently fallen out of favor, Sessions would also cross congressmen from states with legal pot.

“This is not a fight this Administration wants to take,” Democratic Rep. Earl Blumenauer from Oregon warned in a statement to TIME. “The legalization train has left the station.”

Blumenauer has introduced multiple marijuana reform bills with the Democratic senator from his state Ron Wyden, who also told TIME in a statement, “Jeff Sessions can’t cherry-pick on a whim which states’ rights he likes and which ones he doesn’t. Voters in Oregon and a growing number of states who have chosen to legalize marijuana should not have their votes casually thrown in the trash by this administration.”

It doesn’t seem that Sessions or other members of Trump’s Administration are cowed by politics. Still, pot advocates feel protected by the swelling public support for their industry. And although Sessions’ task force on marijuana was directed to look into links between the drug and violent crime, many are hoping he will realize that regulated cannabis businesses can actually help him fight the crime rates he’s eager to lower.

“We are the wall between the black market and the cartels and our society,” says Nassau of Tru Cannabis. “The president talks about building a wall, and we are a virtual wall. You want this? We are it.”

Get our Politics Newsletter. The headlines out of Washington never seem to slow. Subscribe to The D.C. Brief to make sense of what matters most. Please enter a valid email address. Sign Up Now Check the box if you do not wish to receive promotional offers via email from TIME. You can unsubscribe at any time. By signing up you are agreeing to our Terms of Use and Privacy Policy . This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply. Thank you! For your security, we've sent a confirmation email to the address you entered. Click the link to confirm your subscription and begin receiving our newsletters. If you don't get the confirmation within 10 minutes, please check your spam folder.

Write to Tessa Berenson at tessa.berenson@time.com.