Autonomous vehicles are already a common sight on the streets of Silicon Valley, an international hub for self-driving technology. But this month, California set the stage for the next phase of innovation that could dramatically alter transportation and mobility across the globe. The state has proposed regulations to allow fully autonomous vehicles to drive on public roads – meaning empty cars with no steering wheels and no backup driver inside.

The new rules are a game-changer for the nascent industry, opening the doors to a host of complex questions about legality, ethics and safety. The regulations, which could go into effect this year, pave the road for a deployment that could revolutionize modern society.

“This is like the smartphone transition times 10 as far as the potential to change our existence on the planet,” said Karl Brauer, senior analyst with Kelley Blue Book, an automotive research company. “There’s a sense of almost panic and certainly a frantic pace that all these industries are going through to try to position themselves in this new world.”

The race to dominate the market is rapidly accelerating in California, where major technology corporations, traditional automakers and artificial intelligence startups are engaged in aggressive competition. In an industry that could be worth $26bn by 2025, with potentially millions of vehicles on the road in just a few years, there’s a lot at stake.

‘It will be life-changing’

California recently overtook the UK to become the fifth largest economy in the world, and there are a total of 27 companies that now have permits to test autonomous cars on the road, though current rules require a human behind the wheel. With a total of 180 vehicles approved for operation, there are already six times as many vehicles permitted on public streets here compared with 2014 – and probably more than the rest of the US combined.

“The technology itself will perform a lot better than we perform now as humans,” said Bernard Soriano, deputy director of the Department of Motor Vehicles. “We needed to provide a clear path to completely driverless vehicles, because of the safety benefits.”

Jaime Waydo, head of systems engineering at Waymo. Photograph: Sam Levin/The Guardian

Waymo is Google’s recently renamed self-driving car operation. Seated in the passenger seat of a Waymo on a sunny afternoon in Mountain View, Jaime Waydo, head of systems engineering, recited a string of alarming statistics – 1.2 million people die on the road each year, equivalent to a 737 plane falling out of the sky every hour. In 94% of the cases, the cause is human error.

“It’ll be life-changing once we get this technology,” she said, after the push of a button in the car prompted a voice inside to declare “auto driving”, leading the car to maneuver itself out of the Google parking lot.

This is like the smartphone transition times 10 as far as the potential to change our existence on the planet Karl Brauer, automotive analyst

The Waymo cars depend on a Lidar system, which are the “eyes” of the cars that allow them to detect a small object two football fields away as well as pedestrians, vehicles and other objects all around them.

In a move that has dramatically intensified the contest to corner the market, Waymo recently sued Uber, accusing the San Francisco-based taxi startup of a “calculated theft” of its trade secrets, and alleging that former Google employees brought the proprietary Lidar tech to Uber. The case could be disastrous for Uber, which has stated that autonomous vehicles are critical to the company’s future and now faces an injunction request from Google to block its self-driving program.

The lawsuit is moving forward just as Uber has received a permit to test its autonomous vehicles in San Francisco. The company rolled out a pilot program without obtaining permits last year, later canceling it under threat of legal action and after its vehicles were caught running red lights and making dangerous moves in bike lanes.

Uber declined to provide a demo for the Guardian and did not respond to requests for comment.

In yet another indication of how fierce the competition has become, chip-maker Intel also recently announced the $15.3bn purchase of Mobileye, a firm that manufactures cameras and sensors for autonomous cars.

‘A very big leap’

The new change in rules in California would allow companies to self-certify that their vehicles are safe to operate without a human. It’s a “magic sentence” said Brad Templeton, a self-driving car expert who consulted with Google. “Fortunately, it doesn’t say how they have to go about that.”

But for others self-certification is “a very big leap”, said Ryan Calo, a law professor at University of Washington, who is arguing for independent audits. “I’m worried by the idea of a company saying, ‘We’re good.’”

Inside a Tesla equipped with autopilot in Palo Alto, California. Photograph: Bloomberg/Bloomberg via Getty Images

The first known fatal crash involving a partially self-driving vehicle happened in a Tesla car in Florida last year.

Traditional car manufacturers are likely to be more cautious in testing and deployment, but some warn that the Silicon Valley ethos of disruption and flouting regulations could lead startups to quickly take advantage of California’s new rules before they’re ready.

“Some smaller firms are a little more gung-ho and a little more willing to take risks and push limits,” said Erick Guerra, assistant professor in city and regional planning at the University of Pennsylvania.

The race to test

Drive.ai, a Mountain View startup founded by former members of Stanford’s Artificial Intelligence Lab, is focused on deep learning, meaning creating a machine that operates like a human brain in decision-making on the road.

“We view this as a race, and the technology and the company and the approach that can really push this adaptability as far as possible is going to make the most impact here,” said CEO Sameep Tandon at his garage office, located a few miles away from Google.

“The cars are thinking like a human,” said Tory Smith, technical program manager, who was seated in the back of a Drive.ai car as it rode on its own around Mountain View.

Sameep Tandon, right, and Tory Smith at startup Drive.ai in Mountain View, California. Photograph: Sam Levin/The Guardian

Testing in the real world is critical, Smith said, noting that Drive.ai encounters all kinds of unexpected challenges on the street – including people who purposefully jump in front of the car to see if it will stop.

Waydo concurred, noting that Google’s more than 2.5m miles on public roads haveproven the car’s capabilities in dangerous and bizarre situations that would never occur in a lab or closed setting.

Once, a woman in a wheelchair entered the middle of the road in front of the car and moved in circles, chasing what appeared to be a duck or a turkey. Video captured the car waiting patiently.

Testing without drivers is also critical because studies have shown that in partial automation, where a human is still behind the wheel, it can be difficult for a driver to stay engaged.

“When the human driver is expected to take over at any time … we know that’s a real unsafe situation,” said Corinne Kisner, director of policy and special projects at the National Association of City Transportation Officials.

Waydo said the deployment of fully autonomous cars is personal for her: “My mom’s getting older and having those hard conversations about you’re not able to safely drive any more. We need to take your keys away. A technology like this can really give a lot of people mobility for a very long time.”