NEW YORK (Fortune) -- As the number of victims of Bernard Madoff, the criminally charged founder of the investment firm that bears his name, seems to multiply with the speed and force of a hurricane, certain types of investors seem to be absent -- so far, anyway -- from the casualty list.

That's no accident, argues James Hedges IV of LJH Global Investments, a boutique firm that invests in hedge funds and private equity for high-net-worth families. In other words, score one for the big institutions that stick to standard rules rather than allowing their managers to invest on personal connections or hunches.

"There's no Duke Endowment [among the list of Madoff investors]," Hedges says. "There's no Harvard management, there's no Yale, there's no Penn, there's no Weyerhauser, no State of Texas or Virginia Retirement system."

The reason is simple, in Hedges' view. Letting Madoff manage your money "wouldn't pass an institutional-quality due diligence process," he says. "Because when you get to page two of your 30-page due diligence questionnaire, you've already tripped eight alarms and said 'I'm out of here.' "

In short, in Hedges' opinion, any sophisticated entity that actually did its homework would have seen the warning signs.

Hedges got the chance to see those signs up close: In 1997, when he was advising the Bessemer Trust, the giant wealth manager, he visited Bernard Madoff to discuss investing with Madoff's firm.

"I found him stylistically like a lot of traders: fast-talking, distractable, not remarkable," Hedges says of Madoff. But during their two-hour meeting, Hedges says, "there was one red flag after another."

For starters, he couldn't grasp Madoff's investing strategy. "I kept saying, 'you've got to explain it to me like I'm in first grade,' " he says. To no avail.

Then there was the fact that Madoff was charging no fees other than trading commissions: "The notion that something is fee-less -- which is what they largely proferred -- is too good to be true."

The fact that Madoff's operation was audited by a microscopic accounting firm also worried him. "He was also so secretive about his asset base -- that was another red flag."

In the end, Hedges was uncomfortable and Bessemer decided not to let Madoff manage any of its money.

In Hedges' view, those that went with Madoff chose faith over evidence. "You've got people who were disintermediated [i.e., didn't have a professional representative], or unsophisticated, or went in through a personal relationship. That's what a con man is -- a confidence man is somebody that engenders a relationship and then subsequently lures somebody into doing something that they shouldn't do." (According to the federal criminal complaint against him, Madoff has confessed that he ran a "giant Ponzi scheme." His lawyer, Ira Sorkin, declined to comment.)

Certainly many of the institutions that turned to Madoff will challenge Hedges' views, as many will face litigation from their own clients. So far, two of the large fund-of-funds with the largest sums under Madoff's control, Tremont and Fairfield Greenwich, have already asserted that they conducted extensive due diligence before investing. Many others will take the same position.

Should Hedges' opinion be borne out and corporate and state pension funds remain absent from the roster of Madoff victims -- of course, there will be many more names added to the list -- it will only heighten the Madoff tragedy. Because, in the end, it would show that this was one investing disaster that could easily have been avoided.