California Rep. Dana Rohrabacher, a Republican in one of the tightest races of his career, lost the support of realtors in his district after he told members of the Orange County Association of Realtors that homeowners should have the right to refuse to sell to LGBTQ buyers.

“Every homeowner should be able to make a decision not to sell their home to someone (if) they don’t agree with their lifestyle,” Rohrbacher said, according to the Orange County Register.

When the newspaper asked him to clarify his remarks, Rohrabacher added, “We’ve drawn a line on racism, but I don’t think we should extend that line,” and that “a homeowner should not be required to be in business with someone they think is doing something that is immoral.”

California is one of 20 states and the District of Columbia that bans state-level housing discrimination based on sexual orientation and gender identity.

But in the majority of states across the country, homeowners can (and do) refuse to sell homes to LGBTQ homebuyers. That’s because while current federal law prohibits housing discrimination based on based on race, color, religion, sex, disability, familial status, and national origin, the Fair Housing Act (FHA), established under the Civil Rights Act of 1968, does not include sexual orientation or gender identity.

The remarks came after Orange County realtors engaged in the National Association of Realtors’ annual lobbying event in Washington, DC. Members of the group met with Rohrabacher and urged him to support HR 1447, or the Fair and Equal Housing Act, which would add sexual orientation and gender identity to the list of classes protected from housing discrimination at the federal level.

The National Association of Realtors has withdrawn its endorsement of Rohrabacher.