Colorado was the first state to legalize recreational marijuana, and in Denver, most merchants are based in low-income neighborhoods where residents have complained about lingering and strong odors, according to local news media accounts.

It is not the first time a social justice issue has clashed with Mr. Rice’s strong stance on drugs. More than a decade ago he helped block the passage of a statewide needle-exchange program aimed at curbing the spread of H.I.V. from contaminated hypodermic needles, pitting himself against his own party.

His position on cannabis legalization not only puts him at odds with the governor and members of his party, but also with many African-Americans.

In New Jersey, African-Americans are three times more likely to be charged with marijuana possession than whites, even though both populations use the drug at similar rates. That has galvanized civil rights groups like the N.A.A.C.P. and the American Civil Liberties Union of New Jersey to support legalization.

“All the collateral consequences that come with an arrest — jail time, losing your job, losing your housing — are disproportionately falling on communities of color,” said Dianna Houenou, a lawyer with the A.C.L.U. of New Jersey. “Through legalization we can begin to address the harms that have been inflicted.”

A statewide coalition of black pastors, the N.A.A.C.P. and the New Jersey chapter of the Drug Policy Alliance is pushing for legalization as a social justice issue, but only if it is linked to some type of compensation for the harm they say was done to black and brown families whose sons were incarcerated. The pastors said they wanted to make sure members of their communities were able to participate in the billion-dollar cannabis industry as growers and sellers, not just workers. They are frustrated that the wealth being generated in the other states where marijuana is legal is not reaching people of color.