Sen. Cory Gardner (R-Colo.) threatened on Thursday to start holding up the confirmation process for White House Justice Department nominees unless Attorney General Jeff Sessions reverses a decision to roll back a policy allowing legalized recreational use of marijuana in some states.

Gardner said in a series of tweets that Sessions had told him before he was confirmed by the Senate that he would not change an Obama-era policy that discouraged federal prosecutors from pursuing marijuana-related offenses in states where the substance had been legalized. Colorado is one of those states.

Sessions moved on Thursday to roll back the so-called Cole memo, written by former U.S. Attorney General James Cole, which effectively gave states that chose to legalize marijuana the leeway to do so. So far, six states have legalized recreational use of the substance, though it remains federally prohibited.

Gardner's home state, Colorado, was among the first states to legalize recreational marijuana use in 2012.

Sessions has been a longtime opponent of marijuana legalization. But in a 2016 interview with 9News in Denver, then-candidate Donald Trump said that he would not support using federal power to crack down on marijuana legalization, adding that the issue should be left to the states.

The Justice Department's reversal of the Cole memo on Thursday came three days after California's new law allowing recreational marijuana use went into effect.