SI has learned that at least four former Panthers employees have received ‘significant’ monetary settlements due to inappropriate workplace comments and conduct by owner Jerry Richardson, including sexually suggestive language and behavior, and on at least one occasion directing a racial slur at an African-American Panthers scout

Friday was Jeans Day, when most staffers at the Carolina Panthers team offices would wear denim to work. The female employees knew what that meant. As the team’s owner, Jerry Richardson, made his rounds on the way to his spacious office, he would ask women to turn around so he could admire their backsides. Then, in his rolling Southern drawl, he’d offer comment, drawing from a store of one-liners he’d recycle each week. Among those in heaviest rotation: Show me how you wiggle to get those jeans up. I bet you had to lay down on your bed to fit into those jeans. Did you step into those jeans or did you have to jump into them?

Richardson’s conduct was treated as something of a running office joke, according to multiple former Panthers team employees, all of whom spoke on the condition of anonymity out of fear of retaliation. Male employees would knowingly ask the women whether the Carolina owner had noticed them that day. Women subjected to Richardson’s comments would often dismiss them with a sheepish wave of a hand. “No one ever said anything, at least not that I heard,” says one former Panthers employee. “He was the boss. It was [viewed] more of a creepy-old-man thing than a threat.”

But to other Panthers employees, Richardson’s behavior on Jeans Day was consistent with a broader pattern of disturbing—and potentially actionable—office behavior. On Friday evening, the Panthers announced that they had commenced an internal investigation into allegations of workplace misconduct against Richardson, to be led by the outside law firm Quinn Emanuel Urquhart and Sullivan LLP and overseen by former White House Chief of Staff and Panthers minority owner Erskine Bowles. On Sunday morning the NFL announced it was taking over the investigation.

During its own investigation in the weeks prior, SI learned that on multiple occasions when Richardson’s conduct has triggered complaints—for sexual harassment against female employees and for directing a racial slur at an African American employee—he has taken a leaf from a playbook he’s deployed in the past: Confidential settlements were reached and payments were made to complainants, accompanied by non-disclosure and non-disparagement clauses designed to shield the owner and the organization from further liability and damaging publicity.

SI has been made aware of at least four former Panthers employees who have received significant settlements from Richardson or from the team in exchange for what amounted to a vow of silence. One of the deals was confirmed by a recipient’s significant other, who had contemporaneous knowledge of Richardson’s conduct. On the condition that no potentially identifying details (such as dates or dollar figures) be revealed in this story, SI viewed the physical legal document—which included what appears to be Richardson’s signature—for one such settlement. No public documents or EEOC complaints have been found linking Richardson to workplace abuses, but a former Panthers employee tells SI that, while working for the team, she personally saw documents detailing sexual harassment claims against Richardson that were being investigated by the Panthers.

SI asked the Panthers for response to a lengthy and detailed list of questions and the allegations in this story. The team responded on Sunday morning, declining to address specifics.

“The Carolina Panthers recently commenced an internal investigation into allegations of workplace misconduct against the team’s owner and founder, Jerry Richardson,” team spokesman Steve Drummond said. “We welcome the involvement of the NFL.

“The Carolina Panthers and Mr. Richardson take these allegations very seriously and are fully committed to a full investigation and taking appropriate steps to address and remediate any misconduct. The entire organization is fully committed to ensuring a safe, comfortable and diverse work environment where all individuals, regardless of sex, race, color, religion, gender, or sexual identity or orientation, are treated fairly and equally.

“Because this matter is under an ongoing legal review, the Carolina Panthers cannot comment publicly on the specifics of the allegations.”

Richardson is 81, and has been slowed by a 2009 heart transplant. But he still cuts an imposing figure as one of the most prominent owners in America’s most prominent professional sports league. His aura, mannerisms and values ring through the Panthers’ headquarters. Though the antebellum echoes trouble some African-American employees, Richardson is referred to by all simply as Mister, no surname required. When Mister swings by and asks you to lunch, you accept the invitation even if you’ve already eaten. When he confuses your name, you don’t correct him. When the Panthers hold staff meetings, you know to arrive early—punctuality is a core Richardson virtue—and sit as close as possible to Mister’s position, a bar stool and high-top in the middle of the room. Even before you park your car in the lot, you feel Mister’s towering presence: On the occasion of Richardson’s 80th birthday last year, the Panthers unveiled a 13-foot statue of him outside Bank of America Stadium.

For a time, at least, the billionaire was perhaps the most powerful NFL owner behind Jerry Jones and Robert Kraft. Richardson has headed various NFL committees—he co-chaired the search committee that selected Roger Goodell as commissioner—but is perhaps best known for leadership in labor disputes with the players. During the 2011 lockout, Richardson was either intensely miserly or intensely cost-conscious, depending on your view. He reportedly implored his peers, “We signed a [expletive] deal last time, and we're going to stick together and take back our league and [expletive] do something about it.” He also memorably insulted Peyton Manning during a negotiation, reportedly belittling the quarterback’s knowledge of player safety and asking, “Do I need to help you read a revenue chart, son?”

Richardson is a living folk hero in Charlotte, a self-made success who built a fortune, became a local celebrity and kingmaker and brought the mighty NFL to the region. Before the Hornets returned, the Panthers were the only major pro team in town; indisputably, Richardson helped make Charlotte a big-league city. “It was a power culture. You did what Mister said, when he said it,” says one former employee. “He thinks he’s really great. You’re supposed to reinforce that… Even when he does things that make you feel like half a person, that you know are wrong.”

Richardson’s various accusers, speaking independently, describe a strikingly similar pattern of behavior that they say created a hostile work environment. Rather than making sudden bursts of lewd comments or committing isolated lapses in decorum, Richardson worked gradually, starting with kind gestures and pleasant interactions before pivoting to inappropriately intimate behavior. “Looking back,” says one former employee, “he was gaining our trust before doing things he shouldn’t be doing.”

A proud traditionalist who long resisted email and waited until later years to acquire a mobile phone—a flip phone at that—Richardson often sent employees handwritten notes. It was a gesture perceived as chivalrous and quaint. Recipients often felt obligated to send handwritten notes in return. Multiple female employees recall that their notes eventually came accompanied by small cash payments and encouragement to use the money to treat themselves to massages or dresses. The women would thank Richardson; when he responded with lines on the order of You won't find another man to treat you the way I treat you, it was still viewed as flattery, if clumsily rendered, by an older man from an older era.

But multiple former female employees recount that Richardson's behavior began to feel like a violation when he spoke of their bodies. He had a special interest in female grooming, they say. He would notice when their nails were not up to his standards, and pay for them to get manicures. Multiple female employees recalled to SI that Richardson asked them if he could personally shave their legs.

Former employees allege that in addition to verbal harassment, Richardson engaged in improper acts. According to sources, on multiple occasions Richardson requested female employees to visit him during a workday in his suite inside Bank of America Stadium. The women would be escorted by Richardson’s assistant, who would then depart, leaving the owner alone with a junior employee. One former female employee recalls Richardson, who stands 6'3", arriving barefoot and asking for a foot massage. Says one such invitee: “The first time, you thought it was an important meeting with the owner. You [then] realized it was never anything that couldn’t be discussed over the phone.” Others talk of Richardson giving back rubs that lingered too long or went too low down the spine.

Richardson was also known for what multiple women call the “seatbelt maneuver.” He would invite female employees out to lunch, and in keeping with his reputation as a self-styled gentleman, he would open the car door for his guests. Once they were seated, however, he would insist on fastening their seatbelt for them, reaching across their lap and brushing his hand across their breasts before putting the belt in the clasp.

“You look back and it’s wackadoo,” says one former Carolina employee. “You felt preyed upon. You felt fear. You felt self-doubt. But when you’re in [that environment], everywhere you go, every family gathering, it’s, ‘Oh, you work in the NFL? That’s so cool.’ And you don’t want to lose your job.”

At least in Charlotte, Richardson’s backstory carries the ring of legend: Much of the citizenry is aware, at least in broad strokes, of how the man became a billionaire. In the 1950s, Richardson walked onto the football team at Wofford College in Spartanburg, S.C. He played wide receiver, occasionally kicked field goals and was good enough to get drafted in the 13th round by the Baltimore Colts in 1958. He not only made the team but caught a 12-yard touchdown pass from Johnny Unitas in the 1959 NFL Championship Game.

Before his third pro season, Richardson demanded a five-figure contract—that is, a $10,000 salary. The counteroffer was $9,750, and Richardson left camp over a $250 disagreement. Out of football, he returned to South Carolina and used his bonus check from the championship game to open a Hardee’s restaurant.

Richardson timed the growth of the fast food industry perfectly. As Richardson, summarizing his fortuitous timing, once recalled to SI, “[I was] as lucky as a dog with two.... Shut that tape recorder off for a minute.”

You could make the case that Richardson was the South’s answer to Ray Kroc, the McDonald’s founder. Richardson parlayed his one restaurant into many and, along with a former college teammate, formed Spartan Food Systems, a collection of fast food franchises. By the early 1990s, Richardson, nothing if not a survivor, had become CEO of Flagstar Inc., based in Spartanburg, and worth well into eight figures.

Around the same time, Richardson became consumed by the idea of bringing a pro football team to Charlotte. In 1991, Richardson Sports submitted a formal application to the NFL for an expansion club. He decided it would not just represent a city or a state; the Carolina Panthers, he’d call it, would celebrate an entire region. Two years later, the owners unanimously voted to select Carolina over four other finalists to become the 29th NFL franchise. Jerry Richardson would join George Halas as the only NFL owner to have played in the league.

As Richardson was applying for ownership, his company faced legal trouble. Flagstar was the subject of multiple racial bias lawsuits brought by the Justice Department. On the same day in April of 1993 that the Denny’s chain, owned by Flagstar, settled one suit for discriminating against African-Americans—segregating black patrons and requiring them to pre-pay for meals and make various other payments not required of white patrons—six black Secret Service agents assigned to President Bill Clinton's detail were refused a table at a Denny's in Annapolis, Md., while their white colleagues were seated and served. Fortune called the company “a shameful example of entrenched prejudice.” In 1994 Flagstar agreed to pay more than $54 million to settle lawsuits filed by thousands of black customers. At the time it was the largest public accommodations settlement on record in the U.S

Another bias suit brought against Flagstar in 1994, referencing Richardson personally as an actor, did not get picked up by the media. Brenda Reed, an African-American woman, was a general manager at a Hardee’s in Sumiton, Ala., until she was terminated in November 1993. She alleged that she was criticized by a superior who commented that Reed’s franchise “was filled with monkeys,” referring to the number of African-Americans Reed hired. According to public filings, Reed attended a Flagstar corporate event at which Richardson sought to assure managers that the Denny’s racism had been addressed. From the audience, Reed spoke out and told Richardson that Flagstar needed to address Hardee’s because “there is some [racism] in this company.”

Shortly thereafter, Reed was fired. She filed a federal lawsuit against Flagstar, claiming unlawful employment discrimination and retaliation based on race and specifically citing her public exchange with Richardson at the corporate event. Among the allegations were that she was fired for that confrontation of Richardson. Leslie Proll, then a Birmingham civil rights lawyer, represented Reed. Shortly before trial, the case settled for an undisclosed amount. Proll recalls that the settlement figure “was not insubstantial.”

During his tenure as Panthers owner, Richardson has, outwardly anyway, been a champion of diversity. Cam Newton, the Panthers’ African-American quarterback, is the spindle on which the entire team is wound. The Panthers have a Hispanic head coach, Ron Rivera, who took over for John Fox in 2011. A decade ago, Tina Becker was the team’s manager of cheerleader and mascot programs. Steadily promoted, she is now the de facto team president, a member of Richardson’s inner circle who represented the Panthers at October’s NFL owners’ meetings.

But multiple former employees told SI that there were also occasional racial overtones in Richardson’s interactions with his workers. Pressed for specifics, they cite everything from Richardson’s expressed preference that black players not wear dreadlocks to an alleged request that an African-American employee apply sunblock to Richardson’s face. After drafting Newton in 2011, Richardson memorably asked the player, “Did you get crazy after the draft and go out and get any tattoos or piercings? Do I have to check you for anything?” Earlier this year, some Panthers expressed frustration when Richardson indicated that players addressing social issues could be subject to punishment.

Perceptions of casual racism hardened recently when, multiple sources told SI, Richardson directed a racial slur at an African-American scout for the Panthers. The scout left the team this year—but not, according to sources, before he sought the counsel of a Charlotte attorney who negotiated a confidential settlement on his behalf. Contacted by SI and asked if he wished to comment, the scout responded, “I’m not in a position to talk.”

Last week, the Panthers learned that Sports Illustrated was investigating Richardson, the team’s office culture and allegations of the existence of confidential settlement agreements. Sources tell SI that Drummond, the team spokesman, and Richard Thigpen, Carolina’s longtime general counsel, reached out to various present and past employees encouraging them not to speak. In some cases, SI was told, Thigpen made specific reference to confidential settlements some former employees had signed.

One former Panthers worker told SI that this experience rekindled the feeling of harassment she says that she had endured in the office. “Once again, they should be concerned about [the employees], but instead, they’re covering for Mister.” She added that it reinforced her decision to leave her job and share her story, choices she did not reach lightly.

Reflecting on her time with the team, she says that one of the great shames was that she loved the work itself, as well as most of her colleagues. “But,” she says, “it became exhausting. That’s something I don’t know if people realize. Being [harassed] is horrible in so many ways, but it’s also just exhausting. You think about it all the time. Did I do something or say something to give the wrong idea? Do other people know what’s going on? How do I avoid this guy today?” She then issued a refrain uttered by many women throughout the country who have been breaking the silence about sexual harassment these past few months: “I don’t want anyone else to have to go through what I went through.”

Thinking back, she’s struck by one of those handwritten notes from Richardson, the man, she says, whose conduct she feels gave her little choice but to leave. This note was not personal, but one that he gave to every Panthers employee, players included. Scrawled in his distinct handwriting, it contained a list of his five core principles:

Hard work

Harmony

Teamwork

Listen

Respect.