It is only a matter of time before driverless cars take us to work and our children to school, according to James Peng, CEO and co-founder of Pony.ai, a California-based self-driving car start-up.

"If I have to give a number, I'll say probably in five years," Peng told CNBC's Deirdre Bosa at the East Tech West conference in the Nansha district of Guangzhou, China. "We'll definitely see a wide adoption of autonomous driving vehicles — fully autonomous driving vehicles — on the open roads."

That could happen in any part of the world, but Pony.ai has been focusing on the U.S. and China, where the start-up has been testing autonomous vehicles.

It recently partnered with Hyundai to introduce an on-demand vehicle service for residents in Irvine, California, where passengers can share autonomous cabs using an app. The self-driving taxi uses Pony.ai's technology to navigate the roads.

Peng, who was previously the chief architect of Baidu's autonomous driving division, said he believes that the U.S. still leads, but China is catching up very fast. While he hasn't witnessed any direct effect of trade tensions, Peng said it's an element of uncertainty for people considering working for the company.

Even as regulations in both countries are favorable, Peng said the industry is progressing "slowly and steadily."

While major automakers, including GM, Toyota, Audi, Ford and Volkswagen dabble in driverless cars, Peng said the biggest player remains Alphabet's Waymo. Peng, who started his career as a software engineer at Google, said smaller players are falling off because of high costs and the economics of scale.

Peng said that for the industry, "the biggest challenge is all of us."

"It's the other vehicles, it's the pedestrians and bicyclists on the road that behave, a lot of times, irrational," he said, explaining that those factors are difficult for AI to predict.