Republican Rep. Dana Rohrabacher lost the support of a national Realtors group after he said they should be allowed to refuse to sell houses to lesbian and gay people.

Rohrabacher made the comments to an Orange County Association of Realtors delegation in Washington, the Orange County Register reported.

“Every homeowner should be able to make a decision not to sell their home to someone [if] they don’t agree with their lifestyle,” he said, according to Wayne Woodyard, a former Orange County Realtor president who attended the event.

Woodyard posted the comments on Facebook, which prompted outrage from a real estate gay rights group. Rohrabacher had been designated as a “Realtor Champion,” eligible for support from top Realtor donors, the newspaper reported.

“When a supposed champion of the Realtor Party outright states that housing discrimination should be lawful, I hope you agree there should be cause for concern,” wrote Jeff Berger, founder of the National Association of Gay & Lesbian Real Estate Professionals, in a letter to National Association of Realtors President Elizabeth Mendenhall.