By Ismael Estrada, CNN

Maiden, North Carolina (CNN) – Just about everyone here is talking about the local pastor who made national headlines this week after a video that features him telling congregants how to “get rid of” gays went viral.

Neighbors of the Providence Road Baptist Church – where a sign advertises “old time religion” - say Pastor Charles Worley is known for being over the top, with one neighbor describing him as “fire and brimstone” whose views are out of sync with much of the surrounding community.

"I figured a way out, a way to get rid of all the lesbians and queers, but I couldn't get it past the Congress,” Worley told his church on May 13, in a video that has been seen half a million times on YouTube.

“Build a great big, large fence – 50 or 100 miles long – and put all the lesbians in there,” Worley said. “Fly over and drop some food. Do the same thing with the queers and the homosexuals, and have that fence electrified so they can't get out. Feed them. And you know in a few years, they'll die out. You know why? They can't reproduce."

My Take: The Christian case for gay marriage

Some church members, who declined to give their names, defended their pastor, saying his words had been taken out of context. "He said he would feed them!" some church members told CNN, referring to the Worley’s idea for rounding up gays.

Worley “takes a real firm stand on the Bible and what it says about different things,” said church member Joe Heffner. “Whether I like it or not or whether anybody else likes it."

Another church member, who declined to give his name, said that “Being gay and lesbian or homosexual is wrong according to the Bible… it’s Adam and Eve, not Adam and Steve."

My Take: The Bible condemns a lot, but here's why we focus on homosexuality

"We love the people, hate the sin, OK? Point blank,” he continued. “You need to lay off my pastor."

The pastor's sermon also attacked President Obama, with Worley saying that "I ain’t going to vote for a baby killer and a homosexual lover."

On Wednesday, the group Americans United for Separation of Church and State filed a complaint with the IRS alleging that Worley violated the rules of his church's tax-exempt status by engaging in political speech against a candidate.

“Pastor Worley’s vicious and mean-spirited assault on gays and lesbians is bad enough,” said the Rev. Barry W. Lynn, executive director of Americans United. “His pulpit command that people not vote for President Obama is a violation of federal tax law. I urge the IRS to act swiftly to investigate this matter.”

Video of the sermon had initially been posted on Providence Road’s website but was recently taken down, according to CNN affiliate WBTV-TV in Charlotte. The phone line at Worley’s church has been busy since Monday night, and Worley’s home number has been busy since Tuesday.

The church’s website has also been down, but it had described the house of worship as fundamentalist, meaning it represents a Baptist tradition that's more conservative than the Southern Baptists.

CNN’s Belief Blog: The faith angles behind the biggest stories

Worley’s sermon was posted on YouTube by a group called Catawba Valley Citizens Against Hate, which is organizing a protest at the Providence Road Baptist Church on Sunday.

Addressing his congregation last Sunday, Worley referred to his earlier controversial sermon.

"I talked a little bit, I believe it was last Sunday, on the homosexual lifestyle, and there was a whole lot of people who didn't like what I said," Worley told his congregation Sunday, according to WBTV. "I want to read it out of the Bible, and then we'll go from there."

“Listen, all of the Sodomites, the lesbians, and all of the ... what's that word? Gays – I didn't wanna say 'queers' – that say we don't love you, I love you more than you love yourself,” Worley said, according to WBTV. “I'm praying for you to be saved."

Worley’s initial sermon was partly framed as a response to Obama’s endorsement of same-sex marriage, which he made in a TV interview a day after North Carolina voters passed a state constitutional amendment banning legal recognition of such marriages and other types of gay unions.

The Gay and Lesbian Alliance Against Defamation, or GLAAD, was working Tuesday to gather criticism of Worley’s comments from other North Carolina pastors.

“I am angry and sick at heart over Pastor Worley's comments,” said the Rev. Dennis Teall-Fleming, pastor at Open Hearts Gathering in Gastonia, North Carolina, in a statement distributed by GLAAD.

“Nothing he says has anything to do with the Gospel of Jesus Christ,” said Teall-Fleming, who leads a Disciples of Christ congregation. “I call on all Christian and Baptist organizations that have any connection with him to condemn his comments as strongly as I do, including Providence Road Baptist Church of Maiden.”

–CNN's Dan Gilgoff contributed reporting.