It’s a story Malcolm Gladwell has told many times before, including in his book Outliers: how his Jamaican mother earned one of only two scholarships to attend public high school on the island in the 1940s. To attend college, she would also need a series of fortunate events, from a generous loan from a neighborhood grocer to an actual act of Parliament. It just goes to show, says Gladwell, the bestselling author of books like The Tipping Point, Blink and David and Goliath, the “extraordinary role serendipity and luck play in success stories.”

In retelling this story to the thousands gathered at OZY Fusion Fest in Central Park this summer, Gladwell was making a different point in his keynote address than the one he makes in Outliers. The success stories he wanted to talk about were not those that had been realized, like his mother’s. He wanted to talk about all of the successes that never came to be — and still don’t today — because countless talented students are left behind by a society and educational institutions that assume opportunity is adequate and talent is scarce.

Hear more from Malcolm Gladwell by catching OZY Fusion Fest on the Fusion Network on Sunday, August 21, at 8 p.m. or 11 p.m. Eastern. You can also watch the special on Apple TV, Hulu or Roku.

“The biggest problem is money,” Gladwell tells OZY about the causes of educational inequality today. “College has gotten absurdly expensive, and lower-income Americans can’t afford it.” It is an issue that the New Yorker staff writer has revisited in several episodes of his hit new podcast, Revisionist History (the No. 1 podcast on iTunes). Gladwell’s focus there, and in his Central Park address, was something called a “capitalization rate.” It may sound like a banking term, but it’s essentially a measure of “what percentage of people in any society get to fulfill their potential,” says Gladwell — an indicator of how good a society is at ensuring its members get to make the most of their abilities. And, much like Jamaica in the 1940s, modern-day America does not do a particularly good job of capitalization, much as we might like to delude ourselves otherwise.

Why is our capitalization rate so low? First, says Gladwell, “we suffer from the illusion that talent is scarce,” an assumption he calls “spectacularly wrong.” The Commonwealth offered only one scholarship to Jamaica, a nation of 1.5 million people, not because they were cheap or unserious, but because they thought there was only one deserving student. The fallacy continues: Today, college admissions officers at America’s most elite schools make it their job to identify smart, low-income students to attend their institutions free of charge. According to research from Harvard economists Caroline M. Hoxby and Christopher Avery in 2013, there were up to 35,000 such poor, high-achieving students in the U.S., but around 20,000 of them didn’t apply to any selective colleges.

We ought to be very angry that our colleges are more interested in spending money on football stadiums and luxury dorms.

The other major reason behind our poor capitalization rates is that we believe so wholeheartedly that the current system works effectively to identify talent and to provide opportunity. In fact, as Gladwell puts it, “the system is insanely inefficient.” He points to how, despite a collective endowment of close to $140 billion — a “mound of cash that could fill Central Park” — the Ivy League colleges plus Stanford serve just slightly more than 11,000 low-income students at any given time.

We ought to be very angry, says Gladwell, that our colleges are more interested in spending money on things like new football stadiums and luxury dorms than in fulfilling their core purpose as educational institutions. Gesturing to the museums and skyscrapers that surround Central Park, Gladwell argued that such monuments to our accomplishments would not matter in the end. Rather, what future generations will want to know is: “How good a job did you do at helping people in your society reach their potential?”