(CNN) Brett Kavanaugh's time at Georgetown Prep, a private high school in the Washington suburbs, and Yale, the Ivy League college where Kavanaugh attended both undergraduate and law school, have cast the shadow over his nomination to the Supreme Court.

Over the weekend Deborah Ramirez alleged in The New Yorker that Kavanaugh exposed himself to her at a dorm room party during their freshman year in college, an allegation Kavanaugh has vehemently denied. But, with just days until he could be confirmed to the Supreme Court, Kavanaugh's time at Yale, the groups he joined there and the people he knew have all come under renewed scrutiny.

In recent days, supporters of both Kavanaugh and Ramirez have come forward to tell stories of two very different people who came from disparate worlds: Kavanaugh from the world of privilege and private schools. Ramirez from a more humble upbringing that required her to work at the Yale dining halls while in school, according to former colleague and 1988 Yale graduate Jo Miller.

More than a dozen interviews with people who attended Yale during the mid-1980s, when Kavanaugh and Ramirez were students there, paint a complex -- even contradictory -- picture of who Kavanaugh was at the time: a young man from an elite prep school in Maryland who was considered a jock, belonged to one of the school's secret senior societies, Truth and Courage, as well as a fraternity -- Delta Kappa Epsilon -- known at times for antics on campus including one it became famous for: marching across campus with a flag made of women's undergarments. In a photo the Yale Daily News republished last week, two men can be seen hoisting the underwear flag. The caption at the time according to the paper was "DKE at play."

"DKE had a hard-drinking reputation," recalled John Hechinger, a 1988 Yale graduate and author of a book about fraternities, "True Gentlemen." "It was more attractive to conservatives with traditional views on masculinity and femininity."

Hechinger didn't know Kavanaugh personally, but said that in the 1980s at Yale, DKE was one of the few fraternities and was considered an outlier on the school's campus, a vestige of the "old establishment" culture known for heavy drinking and the flag underwear incident.

Another woman who graduated Yale in 1987 who also did not know Kavanaugh personally said "DKE was regarded as disreputable," "widely regarded as drunks and jocks. Joining a fraternity like that was at the time, a way of saying no to Yale's overall culture."

But others charge the criticisms of DKE at that time have been too harsh.

Christopher Munnelly who was a few years behind Kavanaugh and graduated in 1990, challenged the notion that DKE was a magnet for out-of-control partying. At the time, Munnelly said that DKE did not even have a fraternity house and that their official meetings would occur in the common rooms of residential buildings where alcohol wasn't permitted.

"You can't have animal house without a house," he said. "The notion that [Kavanaugh] was part of a party fraternity is not the case."

Munnelly got to know Kavanaugh when he was a freshman pledge. He recalled Kavanaugh being kind and outgoing to the younger pledges despite the divides that sometimes exist between lower and upperclassmen in fraternities.

"He was a listener, introspective," Munnelly said, adding Kavanaugh, "didn't wear his ambition on his sleeve."

Munnelly recalled pledge week or "hell week," as he referred to it, as a collection of silly pranks, but nothing dubious, recounting how one pledge had to carry around a pole vault for a week or how older brothers would have pledges measure campus buildings with pencils. But when it came to Kavanaugh, Munnelly said he never remembered anything remotely controversial.

James Roche, Kavanaugh's roommate in the Fall 1983, also issued a statement saying that Kavanaugh was a "notably heavy drinker, even by the standards of the time."

"(H)e became aggressive and belligerent when he was very drunk," Roche said.

One classmate who attended many of the same parties as Brett Kavanaugh but did not want to be identified, says he was "aggressive, obnoxious drunk, part of the crowd he hung out with."

Roche added that he became close friends with Ramirez in the early days at Yale and while he "did not observe the specific incident in question," he did remember "Brett frequently drinking excessively and becoming incoherently drunk."

"Based on my time with Debbie, I believe her to be unusually honest and straightforward and I cannot imagine her making this up," Roche said. "Based on my time with Brett, I believe that he and his social circle were capable of the actions that Debbie described."

Ramirez was known as friendly, but as Roche recalled, someone who was "very worried about fitting in."

"She felt that everyone at Yale was very rich, very smart and very sophisticated and that as a Puerto Rican woman from a less privileged background, she was an outsider."

Others also described Ramirez as an honest, humble person, someone who was outgoing and friendly.

"There is no reason in the world why she would come forward if this wasn't true," said Julie Heller, who graduated from Yale in 1988 and sometimes attended parties with Ramirez. "She was social, but not particularly rowdy. I wouldn't say she was a huge partier."

At the time, female alumna of Yale's 1987 class have said, it would have been hard for Ramirez to even know who to talk to about that kind of allegation. They pointed out that the institutional attitudes towards the kind of assault that Ramirez alleges have changed dramatically since their time at the university.

"I don't know how Yale would have handled it all," said an acquaintance of Ramirez from Pierson College.

Minouche Kandel, a former resident of Ezra Stiles College in college when Kavanaugh was there, said that "there was just not the same infrastructure" for reporting sexual assault that exists today.

"Very likely if something like that happened you wouldn't report it to the university, you may tell a friend," she said. "It wouldn't have even occurred to me to say something."

Friends of Kavanaugh's meanwhile have struggled in recent days to square the person they knew with the allegations against him, allegations they say they never heard of at the time.

Dwayne Oxley, who lived on the same floor as Kavanaugh for a few years in Ezra Stiles College, said he's "sympathetic" to Ramirez, but "it's just not who I knew as Brett Kavanaugh." He said like a lot of students at the time, Kavanaugh might have drank, but "I never saw him in a state where he wasn't in control."

JUST WATCHED Third woman accuses Kavanaugh of misconduct Replay More Videos ... MUST WATCH Third woman accuses Kavanaugh of misconduct 02:58

The varying classmate recollections of Kavanaugh's time at Yale speak to what has become a larger struggle for the Senate's Judiciary Committee as it prepares to hear testimony from Christine Blasey Ford who alleged Kavanaugh sexually assaulted her at a house party in the Washington, DC, suburbs when they were teenagers in high school -- another accusation Kavanaugh strongly denies.

For some senators, Kavanaugh's own freshman roommate saying he was "belligerent" when drunk was disqualifying.

"The fact that his college roommate said he came home drunk and belligerent, I find credible and I find it is corroborating evidence of the claims by Ms. Ramirez and Dr. Ford," said New York Democratic Sen. Kirsten Gillibrand.

Sen. Bob Corker, a Republican who has been vocal about his desire to see Ford testify, said that he has no way of sifting through memories of classmates from Yale or otherwise. For him, the story of who Brett Kavanaugh was will have to come from two people testifying before the Senate Judiciary Committee Thursday: Ford and Kavanaugh himself.

"I have cleared my schedule tomorrow. I plan on watching the hearing from beginning to finish and that will be the testimony we will have to make decisions. I haven't talked to any classmates. I don't really have any ability to do so," Corker said, adding that he believes the majority staff is following up on any potential leads about Kavanaugh's background.

Chris Dudley, a fraternity brother who graduated in 1987 with Kavanaugh and eventually ran for governor of Oregon as the Republican nominee, said he got to know Kavanaugh on the basketball court where Kavanaugh played JV. Dudley has stayed close to Kavanaugh, sending him a text message in recent days to hang in there as Kavanaugh fought against allegations of sexual assault and misconduct.

He was "just a great guy to be around," Dudley said, adding that he never saw Kavanaugh get aggressive when he consumed alcohol.

"He wasn't that type of person," Dudley remembered.

One woman who was a resident in Kavanaugh's college at Ezra Stiles and who said she knew Kavanaugh very well and often shared meals with him, said the allegation from Ramirez that had surfaced against Kavanaugh was "shocking" and "I don't believe it for a minute."

"I never saw that behavior from Brett anytime in the entire period I knew him," she said. "There was definitely silliness by some people around us at that time, but not from Brett Kavanaugh. I never saw that , never saw him doing anything like that."

Programming note: Watch CNN's full coverage of Thursday's Kavanaugh-Ford hearing beginning at 8:30 AM ET at CNN.com/go on your desktop, smartphone, and tablet, and via CNNgo apps for Apple TV, Roku, Amazon Fire and Android TV -- no log in required.