The family of KFC’s Colonel Harland Sanders has come to his defense after Papa John’s founder John Schnatter suggested the colonel had used the N-word but hadn’t faced criticism for doing so.

Sanders’ grandson, Trigg Adams, told the Louisville, Kentucky, Courier Journal on Wednesday that it is “an absolute lie” that Sanders used the racial slur or any other racially offensive words.

“He’s a weasel,” Adams said of Schnatter, according to the report. “Because he's prejudiced, he’s trying to say somebody else was, too. (Sanders) had absolutely no prejudice against anybody.”

Sanders, who died in 1980, donated more than $20 million to charities, including to black churches, which he sometimes attended, Adams added.

Schnatter resigned as chairman of Papa John's last week after admitting to using the racial slur during a conference call, and apologizing for it.

Schnatter was on a call with marketing agency Laundry Service in May when he tried to downplay comments he made about the National Football League and allegedly said, “Colonel Sanders called blacks n-----s," and complained that the KFC founder never faced public backlash.

The call was a role-playing exercise for Schnatter to prevent future public relations fumbles and came to light as Forbes magazine was conducting a larger investigation of the culture at Papa John's, which was published Thursday.

"Based on interviews with 37 current and former Papa John’s employees—including numerous executives and board members—Schnatter’s alleged behavior ranges from spying on his workers to sexually inappropriate conduct, which has resulted in at least two confidential settlements," Forbes wrote.

Schnatter's attorney said the Forbes story "contains numerous inaccuracies and misrepresentations." Papa John's said it has hired an outside firm to oversee an audit and investigation of the culture at the company and to make recommendations for whatever changes may be necessary.

Last Friday, Schnatter told Louisville radio station WHAS that he was “just talking the way that the colonel talked.”

In a statement, Sanders' great-granddaughter Cindy Wurster Sjorgen, said that Sanders "was known to throw around a few cuss words but never a racial slur. For Mr. Schnatter to use the colonel as a scapegoat for his own horrible, disgusting mouth and racist beliefs is inexcusable."

Since May 2015, KFC has shifted its advertising focus to the colonel as a symbol of the company with a series of quirky advertisements featuring a rotating cast of celebrity Colonel Sanders including, Ray Liotta, Reba McEntire and George Hamilton.

Both Papa John’s and KFC owner Yum Brands are headquartered in Louisville.

Fast-food chain Wendy’s had preliminary merger talks with Papa John’s before Schnatter stepped down, a source familiar with the situation told CNBC. The deal talks highlight the opportunity potential acquirers see in improving Papa John's technology and scaling up in a still fragmented pizza chain industry. Still, the public relations fall out that followed Schnatter's resignation is likely to complicate any potential deal.

Since his resignation, Papa John's has worked to distance itself from Schnatter, prohibiting him from talking to the press, removing him from the pizza chain's advertising materials and revoking his office space at the company's headquarters.

The company has more than 5,000 locations and around 23,000 employees worldwide.

— CNBC's Lauren Hirsch contributed to this report.