Good news for marijuana users in Colorado and Washington: President Obama says the feds are not likely to bust you for breaking out your bong.

“We’ve got bigger fish to fry,” Obama said in an interview with ABC’s Barbara Walters.

Some White House and Justice Department officials have indicated the administration was pursuing a strong legal stance against states like Washington and Colorado, where citizen-passed ballot initiatives legalizing marijuana have been enacted. But in the exclusive interview with Walters, Obama suggested that federal enforcement was not at the top of the administration’s to-do list.

“It would not make sense for us to see a top priority as going after recreational users in states that have determined that it’s legal,” he said.

Politically, waging this particular war on drugs is not in Obama’s favor. A new Gallup poll out this week shows that a clear majority of Americans are against federal intervention on state marijuana laws. In the poll, 64% said they were against having the federal government enforce federal anti-marijuana laws in states where the drug is legal. Only 34% said the feds should. Another poll conducted by CBS News/Washington Post found that 77% of respondents in November felt doctors should be allowed to prescribe marijuana to patients with serious illnesses.

And in Colorado, pot was more popular than Obama on Election Day—Amendment 64 out-performed the president by more than 50,000 votes.

Obama pointed to the shift in public opinion on marijuana legalization to his own evolution on softening the administration’s tone, but said he did not “at this point” support widespread legalization. Obama, in his 1995 memoir “Dreams from My Father,” was extremely candid about his frequent marijuana usage as a teenager—habits his biographer says earned Obama a spot in what was known as the “Choom Gang.” The president has since downplayed his past, and continues to oppose legalization. He and his administration say they are weighing the impacts of Colorado and Washington’s laws on under-aged people.

“There are a number of issues that have to be considered, among them the impact that drug usage has on young people,” U.S. Attorney General Eric Holder said Wednesday in review of the state laws.

After quietly approving Colorado’s voter-backed measure legalizing marijuana on Monday, Gov. John Hickenlooper, a Democrat, drew up a 24-member task force of lawmakers, marijuana advocates, civic leaders and law enforcement dedicated to the law. Hickenlooper says he expects the task force to ”work to reconcile Colorado and federal laws such that the new laws and regulations do not subject Colorado state and local governments and state and local government employees to prosecution by the federal government.”

The Justice Department has yet to release its official response to how it plans to navigate the murky gray areas that overlap enforcement jurisdictions. And it looks as though Congress is taking up the controversy.

Sen. Patrick Leahy, D-Vt., who chairs the Senate Judiciary Committee, is planning to hold a hearing at the beginning of next year to address the disparity between federal and state drug laws. Leahy brought the issue to the Obama administration’s drug czar and former Seattle police chief, R. Gil Kerlikowske, to map out definitive lines in the legal grey areas.

“One option would be to amend the Federal Controlled Substances Act to allow possession of up to one ounce of marijuana, at least in jurisdictions where it is legal under state law,” Leahy wrote to Kerlikowske.