It’s summer 2010, and Jeffrey Epstein has just returned to New York City after serving out an 18-month sentence in Palm Beach, Florida, including parole, for soliciting prostitution from a minor. He’s hosting dinner at his townhouse on the Upper East Side of Manhattan. To his left is John Brockman, the literary superagent who seems to represent every scientist who’s ever written a bestselling book (Daniel Dennett, Richard Dawkins, Jared Diamond, Daniel Kahneman, and so forth). Brockman has brought along a client—a young professor whose line of research interests Epstein. Across the table, and to Epstein’s right, is an aspiring fashion model and her companion.

There’s no cross-talk or conversation between these pairs of guests; it’s more like Epstein has convened two separate interactions for his private entertainment, and these just happen to be coinciding both in time and space. “He would alternate between us,” recalled the professor, who asked that his name not be included in this story. “Sometimes he’d turn to his left and ask some science-y questions. Then he’d turn to his right and ask the model to show him her portfolio.” At one point, a young female staffer stepped into the room to give Epstein a massage, rubbing his neck as he talked and listened.

When Epstein convened 21 physicists on his private island for a 2006 meeting on gravity, for example, he “was always followed by a group of something like three or four young women.”

No one seems to know that much about Epstein’s occupation, but there’s little doubt about the ways he liked to spend his time. “I only have two interests,” he once told a longtime friend and former academic. “Science and pussy.”

It seems those interests overlapped. As the New York Times reported on Wednesday, Epstein’s “passion for cutting-edge science” at times verged into eugenics. Multiple sources told the Times that Epstein had described a plan to inseminate women at his ranch outside of Santa Fe, New Mexico. According to a shakier, secondhand account, also relayed to the Times, Epstein said he wanted to use his ranch to impregnate 20 women at a time, as a means of strengthening the gene pool.

The Times also says that Epstein had an interest in cryogenics and that he told one “adherent of transhumanism” that upon his death he’d like to have both his head and penis put on ice. This can’t have been an earnest scheme (though the Times implies it was); it would make no sense for anyone to reanimate a disembodied penis. Still, Epstein’s “joke” plays off the nature and extent of his dual obsessions.

His pursuits of sex and science weren’t merely carried out in parallel but all at once and often in full view. When Epstein convened 21 physicists on his private island for a 2006 meeting on gravity, for example, he “was always followed by a group of something like three or four young women,” one participant told the New York Times. That meeting’s organizer, the cosmologist Lawrence Krauss, remained friends with Epstein during and after his prison sentence. He told the Daily Beast in 2011 that, speaking “as a scientist,” there was no reason to believe that Epstein was committing crimes. “I always judge things on empirical evidence,” he said, “and he always has women ages 19 to 23 around him, but I’ve never seen anything else.”

Epstein’s former neighbor, the psychologist and computer scientist Roger Schank, describes another such event that he attended: a meeting of artificial-intelligence experts, organized by Marvin Minsky and held on Epstein’s island in April 2002. “Epstein walks into the conference with two girls on his arm,” said Schank. The scientists were holding their discussions in a small room, and as they talked, “[Epstein] was in the back, on a couch, hugging and kissing these girls.”

Like Krauss, Schank says that Epstein always had young women in his company. The first time they ever met, over lunch at Epstein’s house, “It was me, him, and six girls,” Schank said. Like Krauss, he insists that none of Epstein’s “girls” were underage. “I never actually believed this underage thing,” he told me. “They might have been in their early 20s or late teens, but when I talked to them … they were always in college or had just graduated college or something like that. They were not high school girls.”

I asked him how he knew that any or all the girls he saw were actually adults. “How would I know? I’m a professor. You don’t think I know what a college girl looks like?” (Schank was a professor at Northwestern, Yale, and Stanford, but is now retired.) “I’m not saying that [underage girls] weren’t there,” he added. “What would I know? But we never saw them.”

The scientists were, in their own way, members of Epstein’s entourage. “Beautiful women are only a part of it,” wrote the journalist Landon Thomas Jr. in a 2002 profile of Epstein for New York. “Because here’s the thing about Epstein: As some collect butterflies, he collects beautiful minds.” That phrase comes up in other places, too: “Jeffrey’s [hobby] was scientists. He liked to collect them,” an anonymous Epstein associate told Connie Bruck for her recent piece on Alan Dershowitz in the New Yorker. Left unmentioned, though naturally implied, is the fact that Epstein’s butterflies were almost always men. Twenty scientists were flown in for that island meeting on A.I. in 2002, of which 18 were men. Among the famous physicists who were present for the 2006 meeting, a dozen have been publicly identified—and 10 were men. When Epstein arranged to host another science conference at his island home in 2010, not long after his first stint in prison, his guest list included 11 men and two women.

Recent rundowns of Epstein’s science funding, his science fandom, and his science friends show the extent of this division. Almost every science scholar whom he’s said to have courted or supported is a man: Lawrence Krauss, Marvin Minsky, and Roger Schank; also Gregory Benford, George Church, Murray Gell-Mann, Stephen Jay Gould, David Gross, Stephen Hawking, Danny Hillis, Gerard ’t Hooft, Stephen Kosslyn, Jaron Lanier, Seth Lloyd, Martin Nowak, Oliver Sacks, Lee Smolin, Robert Trivers, Frank Wilczek, and more. Truly, the list of men goes on and on. (Among the only women I can find in this group is Harvard’s Anne Harrington, who took a grant from Epstein around 1998.) Lanier told the New York Times that when well-credentialed women did show up at Epstein’s genius gatherings, he wondered if it might be so they could be screened as potential breeding partners. It was one of these women, he said, who described Epstein’s 20-at-a-time breeding project in New Mexico. Lanier could not recall her name (and did not mention Harrington).

News reports and legal filings suggest that Epstein’s women and his girls were treated as commodities: allegedly shipped from place to place on private aircraft, allegedly photographed and turned into porn collectibles, allegedly passed around for sex with fellow VIPs. Women were even swapped from one lech’s employ to another’s: New York magazine reports that Epstein referred five women to Charlie Rose as possible assistants. One, he said, “used to work for Harvey Weinstein he’s lucky if he can get her.”

There is a crude likeness in the ways that Epstein solicited the men and women (and girls) that aroused his interests—in how he reached for what he wanted. Among Epstein’s circles of the superrich, genius men could be swapped and traded, too. Another recent profile of Dershowitz, this one in New York magazine, describes a late 1990s birthday party for Epstein’s loyal friend, the billionaire Leslie Wexner. In lieu of gifts, the story says, Wexner asked each of his guests to bring the smartest person that he or she had met that year. Epstein’s offering that time was Dershowitz—a “beautiful mind” flown out to Ohio on Epstein’s private jet and shared with a wealthy friend.

Over time, Epstein would build a network for procuring brilliant men. Chief among his fixers was the superagent, John Brockman. (Brockman declined to comment for this story.) When Richard Dawkins, Daniel Dennett, and Steven Pinker flew out to a TED conference on Epstein’s private jet in 2002, their agent Brockman was on board as well. (Pinker’s scanty ties to Epstein have been singled out in recent weeks; he says he only boarded Epstein’s jet because of Brockman.) Roger Schank says it was Brockman who introduced him to Epstein, too. “Everybody goes through John Brockman,” he told me.

It’s Brockman’s job to help his clients make connections. Epstein, in turn, helped pay for Brockman’s salon for science intellectuals, the Edge Foundation. According to the Miami Herald’s analysis of public filings from three of Epstein’s charitable organizations, Edge received at least $455,000 between 1998 and 2005. That total doesn’t include grants from Epstein’s other, more mysterious charities. The Jeffrey Epstein Virgin Islands Foundation once claimed, in a press release, to have provided “substantial backing” for Edge, which is there described as a “treasure trove of over 660 virtuosi, geniuses, and masterminds.” (Disclosure: I responded to one of Brockman’s “Annual Questions” in 2008, and my answer is still on Edge’s website, along with a defunct bio page. I also attended an Edge event on the science of morality in 2010.)

As a self-described “member of the Edge community,” Epstein often delved into that treasure trove at the group’s “Billionaires’ Dinner,” Brockman’s yearly “gathering of outstanding minds” at which “the richest people in the world come together with the most intelligent people in history.” Also present at those Billionaires’ Dinners on at least two occasions: Epstein’s “potential co-conspirator” Sarah Kellen, who, according to court filings, was a key figure in the administration of his network of masseuses.

Epstein wanted to have a baby, or multiple babies, but he wasn’t planning to get married or spend much time with any of the families that he created.

Another Brockman client, Cornell University psychologist David Pizarro, remembers seeing Epstein at a meeting held at Brockman’s Eastover Farm estate in Connecticut in 2013. At one point during that meeting, says Pizarro, a helicopter touched down and the financier popped out with a young, Slavic-looking woman. “The scientists were all just in awe that a billionaire with a private helicopter had come to listen to what they had to say,” he recalled.

One young, Slavic-looking woman who spent lots of time with Epstein was his pilot and alleged sex slave, Nadia Marcinko (also known as the Gulfstream Girl). According to police reports, Epstein told at least one of his alleged victims that Marcinko had been “purchased from her family in Yugoslavia.” On this topic, and many others, Schank offered a defense. “What does that mean, a ‘sex slave,’ ” he said. “This is a thing I don’t like the media for, because they come up with these ideas. ‘Sex slave.’ He bought her from her parents, and yes, he was having sex with her, her and 17 other girls.” He added: “I didn’t see her as being underage. I did occasionally talk to her because she was around a lot. She didn’t seem like a child.”

As for the eugenics stuff, Schank says it’s “nonsense” to suggest that Epstein meant to seed the human race with his DNA. He did want to be a father, though, and by many different women. About 15 years ago, Schank said, when Epstein had just turned 50, the two had a long conversation about parenting. Epstein wanted to have a baby, or multiple babies, but he wasn’t planning to get married or spend much time with any of the families that he created. He wanted to know what would happen if he impregnated “lots of different girls” and then helped them out financially. Would the kids come out the way he wanted?

Schank wrote about this conversation for his 2005 book, Lessons in Learning, e-Learning and Training, though in that text he refers to Epstein only as a “friend” who “lives the kind of life that many men fantasize about, and, shall we say, moral rectitude doesn’t come up much.” Schank offers his advice as an expert in educational psychology: Epstein would be wise to keep the mothers of his offspring from shacking up with other men; otherwise, he might lose control of the children’s development. “Little by little, my friend’s children would be his only genetically if he wasn’t careful,” Schank writes.

Based on his reading of this week’s story in the Times, Schank now wonders if maybe Epstein changed his mind and decided that the mothers of his children should all live with him (and one another) in New Mexico. “The fact that he came up with a new plan is not surprising to me,” Schank said.

He was intent on clarifying that he always felt Epstein’s intentions, even with the women, were benign. “This guy was actually not a bad guy,” Schank told me at the end of our conversation. “I mean, put the 14-year-olds out of the picture. Those even make me think he was a bad guy. But to my knowledge he was not a bad guy. He was a good guy.”