Three out of four Americans support legalizing marijuana — but almost half of Americans don’t want dispensaries in their neighborhood, according to a study from Pew Research.

With medical marijuana now legal in most states, municipalities are frequently left to decide where dispensaries can set up shop. For many towns, sharp criticism from neighbors and pushback from local businesses can leave them in a tight spot as they walk a fine line between following the law and keeping neighbors happy.

Laws governing cannabis dispensaries vary from state to state, but most follow a similar set of guidelines such as, they can’t be a certain distance from a school, daycare or another cannabis dispensary. While many towns follow these guidelines, others have grown creative in staving off cannabis dispensaries with blanket moratoriums or specific fees outside of the normal course of business.

In Massachusetts, for example, cannabis is legal for recreational use; however, cannabis dispensaries have struggled to open retail stores due to “host community agreements.” These contracts, set up by the state’s Cannabis Control Commission, must be established between the retailers and the city, resulting in an agreed-upon yearly fee called a “community impact fee.”

Ryan Smith, chief operating officer of cannabis dispensary Cure Holdings, worked closely with a town in Massachusetts to try to come up with an agreement to open a dispensary but in the end couldn’t negotiate the terms. Aside from Massachusetts, he has also experienced resistance in other communities.

“We’ve faced some fierce opposition, with some zoning officials and some city councils not wanting us there, and if we were going to be there, they want influence in what the design was of our building, they want to tell us what we could and couldn’t do, and tried to get us to pay for things that are outside the scope of what our responsibility was,” Smith told NBC News.

Opposition tends to be concentrated in wealthier communities, as they have less of a need to create new jobs and to fill commercial real estate vacancies. The wealthier a community, the more likely it is to push back on dispensaries moving in, resulting in a disproportionate amount opening in lower-income neighborhoods, according to a study by Marijuana Business Daily. The study looked at Denver and Seattle and found that due to fewer regulatory hurdles, low-income neighborhoods in those cities housed a larger number of retail marijuana stores than wealthier ones.

The same situation is playing out in Columbus, Ohio. Luis M. Alcalde, an Ohio-based cannabis lawyer, has had difficulty helping his clients open stores in the suburbs of Columbus, forcing all of the five dispensaries that were approved to open in one general spot in the city.

“Every single suburb of Columbus put a moratorium in place,” said Alcalde. “As a result, all five dispensaries for central Ohio are in the city of Columbus. There’s not a single dispensary in any of the suburbs of Columbus because of that.”

Alcalde was also unable to lease space in a Columbus shopping center that was 50 percent vacant because the landlord couldn’t get comfortable with the fact that a cannabis dispensary was moving in.

“He would not lease to my client because he’s old school and he’s very biased against cannabis,” said Alcalde. “And here we were offering to build a really nice retail establishment that was going to generate a lot of foot traffic, but he wouldn’t budge.”

Smith explained that part of the reason wealthier communities don’t want dispensaries in their neighborhood is because they fear a cannabis dispensary will lead to crime or draw unsavory characters to the area.

“Someone asked me once, ‘Will there be people in tie-dyed shirts walking out the front door?’ And I said, ‘well, there might be, but that’s probably the exception rather than the rule,'” said Smith. “This is a medical facility, we have medical professionals on staff.”

In fact, residents are more likely to see an adult over 50 exiting a medical cannabis dispensary than one with a tie-dyed shirt. Among adults 65 and older, cannabis usage is up 250 percent from 2006 to 2013, while adults 50-65 have increased usage by almost 58 percent, according to a national survey on drug use. That rise is attributed to an increase in older adults searching for a safer alternative to opioids, which are highly addictive and have driven a spike in opioid-related deaths.

Despite trepidation from many neighbors about dispensaries moving in, statistics show that they are less dangerous than bars or even convenience stores. Cannabis dispensaries are also exceedingly regulated and most have upwards of $100,000 in security installed. Medical professionals are on staff and many dispensaries won’t let you in the front door without a medical marijuana card.

Still, dispensaries are treated the same as a bar, strip club, or tobacco shop, since they are considered an “undesirable business.” And with more dispensaries opening in lower-income neighborhoods, it further drives the stigma around who and what that business will attract.

“For most people, change is challenging, and when you don’t have anything to compare a dispensary to, you don’t know what it’s going to bring and what the impact is," said Philadelphia City Councilman at Large Derek Green, who recently introduced legislation to legalize recreational cannabis.

"People have concerns about having a medical cannabis dispensary in their neighborhood or community, when in fact the pharmacy that’s right next to your house or down the street has much harsher types of drugs, and that’s what has led to this opioid addiction,” said Green.