One in 10 Americans believe small business owners should be free to refuse to serve or do business with African-Americans on religious grounds, according to a new poll.

A survey released this week by the Public Religion Research Institute finds that strong majorities of Americans reject the idea that businesses should be legally allowed to refuse to serve either African-American, Jewish, gays and lesbians, or atheists, but a small portion of the country still believes you should be able to.

Among the four groups, the survey found the least support for refusing service to African-Americans, but the most for discriminating against gay or lesbian individuals.

Asked if it should be legal to refuse to do business with members of the LGBT community on religious grounds, 16% said yes and 80% said no. Similarly, 15% said it should be legal to refuse service to atheists, with 81% saying it shouldn’t be. The polling found slightly less support for religious beliefs being sufficient to allow a small business owner to refuse to business with Jewish people, 12% said yes and 85% said no. And when it came to African-Americans, 10% said they supported the legal right to refuse service and 87% said they did not. PRRI conducted the survey of more than 1,000 adults via telephone, including cell phone users, and results have a margin of error of 3.1%.

The poll also found that a slight majority of Americans, 54%, believe “the right of religious liberty is being threatened” while 41% say it is not.

The survey found a relatively even three-way split when asking respondents what they thought the most serious problem regarding the role of religion in American society. Three in 10 said “Religion is being removed from public places.” Another 25% said it’s that “government is interfering with the ability of people to freely practice.” And 24% picked “Religious groups are trying to pass laws to force their beliefs on others.” Another 9% picked “the rights of smaller religious groups are not being protected” as their number one concern.

While less common today, supporters of racial segregationists often used religious arguments to advocate their viewpoint before the civil rights movement of the 1960s. Language similar to that often used by marriage equality opponents was regularly used by those opposing interracial marriage and others who sometimes argued different racial groups had been placed on separate continents by God and that served as proof they should not mix.

The issue ended up before the Supreme Court when religious school Bob Jones University argued that its First Amendment protections should allow it to prohibit interracial dating without losing tax benefits. The Supreme Court ultimately ruled that “Government has a fundamental, overriding interest in eradicating racial discrimination in education … which substantially outweighs whatever burden denial of tax benefits places on petitioners’ exercise of their religious beliefs.”