However, KCIR said, as of May 2017, he had personally spent $12.5 million for 42 years of name recognition, or about $300,000 a year.

Schnatter entered into a multimillion dollar agreement with the university back in 1996 to assist in constructing and equipping its football stadium. The naming rights were a $5 million arrangement — a $4 million contribution from Schnatter and a $1 million sponsorship from Papa John's, according to a report by the Kentucky Center for Investigative Reporting .

"In moments of crisis, the best communities find a way to come together," Neeli Bendapudi, president of the university, said in a statement. "Over the last 24 hours our community has been fractured by the comments made by former UofL trustee John Schnatter."

The University of Louisville announced Friday that it will change the name of its football stadium from Papa John's Cardinal Stadium to just Cardinal Stadium after the company's founder admitted he used the N-word during a conference call with marketing executives in May.

According to the agreement, Schnatter could donate either cash or shares of publicly traded companies and change the name of the stadium at any point as long as he covered the costs of changing the logos and signage. Papa John's is also named as the exclusive pizza provider at the stadium and during university events.

The incident during the May conference call came to light after Forbes magazine detailed it in an article Wednesday. Schnatter later confirmed he was on a call with marketing agency Laundry Service when he tried to downplay comments he had made about the National Football League last fall. He said, “Colonel Sanders called blacks n-----s" and never faced any public backlash at KFC.

In the wake of this report, Schnatter resigned as chairman from Papa John's board and stepped down from the University of Louisville board of trustees. His name name was also removed from a signpost of a gymnasium in his hometown of Jeffersonville, according to the Louisville Courier-Journal.

Schnatter, who owns a 24 percent stake in Papa John's, remains on the company's board.

Late Friday, Papa John's CEO Steve Ritchie issued a statement about the steps the company is taking to regain customer trust. These steps include an outside audit of its business to gauge its diversity efforts and removing Schnatter's image from its marketing.

"Papa John’s is not an individual," Ritchie said. "Papa John’s is a pizza company with 120,000 corporate and franchise team members around the world.

Papa John's has suffered backlash after Schnatter's comment became public. Major League Baseball indefinitely suspended its Papa Slam promotion — a campaign that both sides have collaborated on since 2016. The Miami Marlins, New York Yankees, New York Rangers, Atlanta Falcons and Atlanta United have all suspended their relationships with the brand.

In addition, Olson Engage, the public relations agency that was hired by the company in February, dropped them as a client.

"While Schnatter’s resignation as chairman of the board was a first step forward, Papa John’s needs to do more to show that that its corporate values are not aligned with its founder’s behavior," Michael Gordon, chief executive of Group Gordon, a strategic communications firm, told CNBC via email. "With Schnatter remaining on the board, he also will remain a liability for Papa John’s. It doesn’t matter how much of a stake Schnatter has in the company or that the company is named after him – he needs to be extricated from the brand for Papa John’s to truly move on from this and other ugly episodes."

Papa John's shares closed Friday at $53.55, recouping the losses it logged Wednesday, and then some. In the aftermath of the report, the stock hit a 52-week low of $47.80, but is now up about 4 percent for the week as a whole.

"This reaction and backlash shows that customers play a much larger role than ever before, and people are only going to continue to cite how they feel publicly and become vocal about it," said Nat Sutton, a partner and head of Buffkin/Baker, an executive recruiting agency. "This reaction shows that the community will come together and that this has more power than a large corporation."

Christopher Gilbert, a business ethics consultant at NobleEdge Consulting, told CNBC he doesn't expect this event to be "a brand or business killer."

"The Papa John brand image is strong," Gilbert said.