"Pale, male and stale."

That's how the CIO of California State Teachers’ Retirement System Christopher Ailman typically describes the demographic that he finds at business conferences. On Monday at the Milken Institute Global Conference in Los Angeles, Ailman sat on a panel with four female leaders in high-growth investment to discuss how to make that dynamic change.

This year at Milken, roughly 30% of the participants are women. Yet when you look into the business world itself, the number of women in leadership pale in comparison to those seen walking around the Beverly Hilton right now. Roughly 5% of CEOs and around 18% of board directors at Fortune 500 companies are women. The numbers are even starker in Silicon Valley, where just 4.2% of decision-makers at venture capital firms are female.

The panel discussion on Monday honed in on Silicon Valley as one of the main epicenters of the diversity debate right now. Shanna Tellerman, the founder and CEO of Pencil & Pixel and a former Partner at Google Ventures, said she long avoided taking on the "women's issue" as both a female VC and entrepreneur. She wanted to make sure that her peers saw her as a valuable colleague and thought-leader as opposed to just the token female in the room.

Yet after some time in the industry, Tellerman thinks there is a simple and effective way that we can change the lopsided gender dynamic plaguing the valley.

"The power in Silicon Valley, in an odd way, is in the entrepreneurs hands. All the investors want to get in on the same deals," said Tellerman. "Something I fantasize about is that the small number of CEOs [of those sought after startups] put out a statement to everyone who is interested in investing in them and say they will only take money from those teams that make diversity a priority."

Silicon Valley "lives and dies on deal flow," says Tellerman. She believes the minute that you see company founders with highly successful ideas bring up the diversity issue, we will see more female VCs in leadership roles.

Mellody Hobson, the president of Ariel Investments as well as an outspoken advocate for diversity in business, said Tellerman's "fantasy" hits on a key component of the issue: accountability. Until existing leaders make it a priority to change the dynamic and hold each other accountable to do so, we will continue to see the numbers move at their current sloth-like pace.

"We need to stop admiring the problem," said Hobson. "It is fun to just sit around and kick it around about how unjust it is and how nothing has moved... But we say to ourselves what individually are we all going to do."

Hobson's team assesses the diversity of a company's workforce and leadership team as a part of any due diligence into a potential investment. She said she once challenged the CEO of a very successful (unnamed) firm in Silicon Valley to make it a policy that no company they invest in will go public without a diverse board.

His response?

"We can't do that... Our own diversity isn't very good. He said our job is to help entrepreneurs and the entrepreneurs need to make the decision of how to run their company," said Hobson. "The reason they won't [interfere] is because it is not a priority.

So why aren't entrepreneurs in Silicon Valley demanding diverse partners today? Tellerman said it because it is jus taken for granted right now. As an entrepreneur trying to get your business off the ground, you have so many questions about the potential partnership that have nothing to do with diversity.

"It is not on the standard list of questions today, but it very well could be and it matters a lot," she said. "For every entrepreneur if that was a standard question that they asked... we could see some of that unconscious bias go away."

This post is part of LinkedIn's full coverage of the Milken Institute Global Conference.