Billionaire buy-and-hold investor Ron Baron told CNBC on Friday that he passed on investing with Hollywood producer Harvey Weinstein years ago due to "very disturbing" stories.

Baron said he was told by people he knew in the entertainment industry about reports of Weinstein's alleged sexual misconduct when vetting an opportunity to put money with Harvey and Bob Weinstein in August of 2009.

"An investment banker brought them to visit us, Harvey and his brother Robert, Bob. We spent a day and a half speaking with them," the chairman and CEO of Baron Capital said on "Squawk Box" at his annual investor conference in New York.

After that meeting, Baron said: "We were convinced that Harvey was this fantastic producer and would make great movies and great television shows. So we were interested in investing a small amount even though it was a private company."

But when it came time to do his homework before making a decision on whether or not to move forward with investing in The Weinstein Company, Baron said his sources at the time made allegations that were very specific about what they said women had to do to get jobs working on projects with Harvey Weinstein.

"Uniformly, we had the same story of the events that are being written about today. We knew that eight ... years ago," Baron said. "The people we spoke to knew that as well."

"We heard a lot about women. So it was very disturbing," he added. "You don't want to do business with people who you don't care for."

Baron said that when he called to decline, Harvey Weinstein was very upset, saying Baron wasted his time. Baron said he can't remember who hung up on whom.

"It's amazing to me that this is eight ... years later, and you're seeing it for the first time in the paper," Baron said. "How did this happen? How was this allowed to happen?"

Representatives for Harvey Weinstein and The Weinstein Company were not immediately available to respond to CNBC's request for comment on the allegations from Baron, who first wrote about this situation in his quarterly letter to shareholders.

Since the avalanche of allegations about sexual improprieties, following last month's bombshell reports in The New York Times and the New Yorker, Harvey Weinstein has apologized for behavior that he said "has caused a lot of pain." He has also said he sought treatment, but said through a spokesperson, "Any allegations of non-consensual sex are unequivocally denied."

Weinstein was fired from The Weinstein Company on Oct. 8, three days after the Times broke the story. The Weinstein Company, and before that Miramax, produced some of the most critically acclaimed movies of a generation. On Oct. 14, the organization behind the Oscars expelled Weinstein, who won an Academy Award in 1999 for Best Picture for "Shakespeare in Love."

Baron has made his fortune by generally buying stocks of publicly traded companies that he feels are undervalued, and then holding those positions for a long time. Baron Capital manages nearly $26 billion on behalf of clients.