UNITED STATES - MAY 14: Rep. John Fleming, R-La., speaks during a news conference at the House Triangle on immigration reform on Tuesday, May 14, 2013 . (Photo By Bill Clark/CQ Roll Call)

WASHINGTON -- Conservatives are losing ground on the fight to keep marijuana illegal, a Republican congressman warned in dire terms at the Values Voter Summit.

During a conference mostly focused on religious liberty, abolishing the IRS and promoting small government, Rep. John Fleming (R-La.) delivered a full speech on the ills of decriminalizing and legalizing marijuana.

Fleming, a medical doctor, warned of a "growing acceptance" of marijuana -- a fact hard to deny, considering public opinion and state laws. Colorado and Washington state have legalized recreational marijuana, and 23 states and the District of Columbia have allowed the substance for medical use.

"The time for us to speak up on this issue is now," Fleming told the crowd of conservatives.

Fleming linked marijuana to deaths and domestic violence and said legalization supporters spread lies that marijuana is not addictive. Most supporters don't actually say it can't be addictive.

Of course, those ills apply even more so to alcohol. Asked about the comparison after his speech, Fleming acknowledged that there are problems with alcohol, but said it has been accepted by the culture for thousands of years, making prohibition "obviously problematic." The same social acceptance doesn't exist for marijuana, he said, echoing an argument he has made before.

"If you and I accept the fact that alcohol is a problem and a danger, is it logical to say, 'Well, instead of having one problem, that we should have two problems?'" he asked reporters after his speech. "Why add a second one if one is already causing problems?"

Fleming also dismissed the argument by advocates of legalization that it would allow states to collect significant amounts of tax revenue. A study released this month by personal finance site NerdWalletestimated that states would bring in a total $3.1 billion each year if they legalized marijuana.

The congressman said that more revenue may come in, but the cost of dealing with health issues, larger homeless populations and other social ills would make legal marijuana a net loss. (Whether Colorado's pot laws have actually drawn more homeless people is unclear.)

Fleming said he thinks efforts to keep marijuana illegal seem to be working -- at least to some degree. A recent study from the Public Religion Research Institute found that support for legalizing marijuana nationwide had dropped from 51 percent in 2013 to 44 percent this year.

But Fleming lamented that marijuana supporters remain unconvinced that the drug is dangerous.