More than 500 miles from home, North Carolinian Mac Sullivan slowed down as he walked past a small rally on the streets of New York City. A group of women wielding signs were protesting one of Sullivan's favorite TV personalities: recently ousted Fox News host Bill O'Reilly.

For Sullivan and other fans, O'Reilly's departure on Wednesday from his show "The O'Reilly Factor" spurred a mix of emotions— from confusion to disappointment. The show was hosted by the 67-year-old author and political commentator for more than two decades.

Sullivan and his wife, Gray Sullivan, see O'Reilly's exit as a loss for conservative fans who admired his "tough" interviews and philanthropy. O'Reilly often encouraged viewers to support the Wounded Warriors project along with a number of other charities.

O'Reilly, the channel's biggest on-air talent, was let go from parent company 21st Century Fox on Wednesday amid multiple sexual harassment allegations. Earlier in April, the New York Times reported that O'Reilly and his bosses dished out about $13 million in settlements to five women over the years for the claims.

In a statement released Wednesday, O'Reilly denied the allegations as "completely unfounded."

"It is tremendously disheartening that we part ways due to completely unfounded claims. But that is the unfortunate reality many of us in the public eye must live with today," he said.

Let our news meet your inbox. The news and stories that matters, delivered weekday mornings. This site is protected by recaptcha

The Sullivans plan on continuing their Fox News viewing and described other news outlets as biased. With Tucker Carlson taking O'Reilly's 8 p.m. prime time slot, Sullivan said he doesn't expect the channel to take a hit.

"O’Reilly spoke for the common person. He did not speak for those in power, but for those not in power. [He spoke] from a conservative viewpoint that many working class, common folk people have," said Mac Sullivan, adding that he has been watching Fox News for decades.

Men place boards over a poster of former cable news host Bill O'Reilly outside of the Fox News offices in New York on April 20, 2017. Rashid Umar Abbasi / Reuters

"O’Reilly is number one in cable news, but somebody else will step in. Maybe it won’t be the same following, but I tend to think it’ll still be strong just because there needs to be another voice in the marketplace," the 60-year-old said.

The channel has already begun scrubbing O'Reilly from its brand and programming. The show aired Wednesday night with a revised title of "The Factor." Posters of the former host plastered outside Fox News' Midtown Manhattan headquarters were removed Thursday morning.

On Wednesday night's show, temporary host Dana Perino noted that O'Reilly's absence would lead to "a lot of feelings" from his loyal viewers.

"Bill has been the undisputed King of Cable News — and for good reason... And you, his audience, responded in record numbers, making “The Factor” the No. 1 cable news show for more than 16 years. You have also been loyal, and we can't tell you how much that means to everyone on 'The Factor," the former White House press secretary said.

Some fans have been left confused by the host's ousting, which came after dozens of advertisers fled the show. Automaker BMW and family history service Ancestry.com were among the companies that pulled ads.

O'Reilly was on preplanned vacation and expected to return to his show on April 24 when the news of his leaving broke. He will receive a net payout of $25 million after leaving, NBC News reported.

A worker cleans a sign outside the Fox television studios, where a poster of Bill O'Reilly has been removed, in New York on April 20, 2017. Richard Drew / AP

Rich Gibbons, 63 of New Jersey, said he is skeptical of the allegations against O'Reilly and puzzled by his departure, as well of Fox News' handling of it.

“For him to get dismissed like that, I would think there would have to be more detailed evidence," Gibbons said. "Who knows though, maybe there’s something behind the scenes that you don’t know or I don’t know...I would think the organization would want to back him until something is proven."

The latest accusation against O'Reilly came Tuesday when attorney Lisa Bloom announced she is representing another accuser. Bloom told the Hollywood Reporter that O'Reilly called her client "hot chocolate" and feared termination if she told him to stop.

"What's only fair to his audience is that [Fox News] explain why they let him go, without giving all the sordid details," Gibbons said. "At least let him come on [the show] and give his side of it. I still feel empty because I don't know why he is gone. You always want to know."