Recently, when a Tesla driver was caught appearing to doze with his car on autopilot, the incident sparked a full-on freakout. The conversation about the safety of self-driving cars turned even more serious with the recent news that a man died in a crash while using his car’s autopilot.

Future generations, however, may not share such concerns: Over time, self-driving cars and trucks are expected to be much safer than those with people behind the wheel.

That presages huge changes for the massive auto insurance industry, which must prepare for a world in which using your “inferior” human driving skills could become more indulgence than necessity.

Self-driving cars could be on public roads within five years, market watchers say, as auto makers and technology companies continue to invest in the technology. That’s expected to gut demand for personal auto liability insurance, and while business models and laws related to computer-piloted vehicles will take years to evolve, experts already see hints of what’s to come.

Robots will soon put your human driving skills to shame. Everett Collection

Tepid responses from established insurers could lead to the rise of new insurance companies that are quicker to respond to changing market conditions, some experts say. Others anticipate lower premiums for people who continue to drive, though perhaps still much higher than for those with self-driving cars.

“The technology is getting better and better and software can react so much faster than a human,” said Joe Schneider, managing director of KPMG Corporate Finance. “When you go without accidents, you’re funding other, lesser, drivers. It’s tough to compete against a robot.”

Robocars may rise sooner than you think

Ford Motor Co. F, -0.68% , Tesla Motors Co. TSLA, +4.42% and others are developing self-driving vehicles with advanced safety features. Google parent Alphabet Inc. GOOG, -2.37% GOOGL, -2.41% agreed to buy 100 minivans from Fiat Chrysler to equip them with self-driving technology for testing. General Motors Co. GM, -1.31% and ride-sharing company Lyft Inc. plan to test a fleet of self-driving electric Chevy Bolt taxis within a year.

And in July, Mobileye US:MBLY and Intel Corp. INTC, -0.85% announced a deal with BMW Group to bring driverless cars to streets by 2021.

Startups are in the game too. In late May, for example, a company called Zoox said it raised a round of funding valuing it at above $1 billion.

Read: Auto suppliers are a way to invest in a driverless future

GM’s forecasting unit expects the technology to hit roads by 2020 or even sooner, faster than the company thought just a few years ago. Questions and concerns about the technology about, notably following reports of a deadly Florida crash that happened while a man was using his Tesla’s autopilot feature. That crash has sparked investigations by federal authorities.

But experts generally expect that unless the machines rise up in some “Terminator”-like scenario, self-driving cars are likely to continue the historical trend of safety technology lowering the number of serious accidents.

When self-driving cars will take over the roads is still a matter of speculation, with adoption subject to regulatory hurdles and how well the technology performs in testing and on the road. It may be 30 years before self-driving cars make up the majority of vehicles on U.S. roads, according to Moody’s Investors Service; they could be “nearly universal” in 40.

Meanwhile, questions about how insurance products will evolve with fewer full-time drivers on the road abound. Several insurers contacted by MarketWatch declined to discuss their outlook in detail.

Read: Why GM’s president says you won’t be driving its cars

Experts, meanwhile, generally expect premiums for safer self-driving vehicles to be lower than those for today’s cars and trucks. But what if you own a self-driving car with the option of manual control, which you regularly choose to use?

“A human driver could be a higher risk,” said Jerry Allbright, principal in KPMG’s actuarial and risk practice. “Even if a person decides to drive — they hit a button — the car will be shadowing the driver. The car might still intervene, and drivers may be prone to a higher premium if their car doesn’t have that capacity.”

In such cases, self-driving cars could store a treasure trove of data for insurers and courts to mine to determine responsibility.

In a recent report, Moody’s said licensed drivers will likely need to maintain some sort of personal auto insurance for the first few generations of self-driving vehicles. But the burden of insurance may shift to auto makers in the form of product liability as the technology becomes more widely adopted, the report said.

“It will be a question of driver negligence versus product liability,” said Jim Whittle, chief claims counsel at the American Insurance Association.

Read: Car software glitches on the rise, report finds

At least one auto maker, Volvo, has already said it would accept liability for accidents caused by its vehicles when they are in full self-driving mode.

The Mercedes-Benz F 015 self-driving concept car in Amsterdam's Dam Square in March 2016. AFP/Getty Images

Humans who on insist on driving their own cars will likely be judged as “lesser” drivers from a risk management standpoint, and pay a higher insurance premium than those with self-driving vehicles.

“It’ll be more expensive, but it could be less than what [drivers] are paying today,” said KPMG’s Allbright. “On a relative basis, all cars will be safer, reducing premiums, but more than likely those [people] driving will pay more.”

Robocars threaten largest chunk of insurance industry’s premiums

Some insurers seem to be taking a wait-and-see approach as they predict the effect of self-driving cars on their industry. The longer the timeline, however, the more change they anticipate.

Advisory firm KPMG recently firm surveyed senior executives at insurance companies representing about $85 billion in auto premiums, finding that 58% see self-driving cars having little to no impact on their industry over the next six to 10 years. Only 16%, however, felt the same way more than a decade out.

Auto insurance, including commercial vehicles, is a $220 billion-a-year industry, according to KPMG; Moody’s says it makes up 40% of all property and casualty premiums, by far the largest single category.

Five insurers — State Farm, Geico, Allstate, Progressive, and USAA — hold about 53% of the market share for personal auto insurance and account for more than $101 billion in premiums a year, according to SNL Financial LC.

If self-driving technologies radically reduce the number and severity of auto accidents on the road, analysts say, demand for traditional auto insurance will plummet — and rising product liability insurance revenue won’t cover the difference.

The core of the auto liability insurance business model is threatened by self-driving cars, said Cathy Seifert, S&P Global Market Intelligence analyst, in a recent note. Seifert also sees the liability insurance burden shifting to the car’s manufacturer from the driver.

“This shift from driver-dependent vehicles to driverless vehicles will likely bifurcate the insurance industry, as those insurers who have shifted their business and underwriting models are equipped to deal with this change, to the detriment of those who have not embraced it or have discounted its impact,” Seifert said.

KPMG’s Schneider foresees a “chaotic” effect on the insurance landscape as more self-driving vehicles hit the roads, and worries that many insurers won’t be flexible enough to adapt.

Among large insurers, Seifert sees Progressive Corp. PGR, +1.49% as best-positioned for this sea change, noting that the company has for years used plug-in dongles to align premiums with a customer’s driving patterns and collect detailed automobile use data.

Berkshire Hathaway Inc.’s BRK.A, +0.36% BRK.B, +0.07% Warren Buffett — whose company owns Geico — has admitted that self-driving cars, should they significantly reduce accidents, would be “bad for the auto insurance industry” as it stands now.

Warren Buffett, Chairman and CEO of Geico Owner Berkshire Hathaway, in November 2013. Getty Images

More than 40% of respondents to the KPMG survey believe niche underwriters and new providers will rise up to take advantage of the shifting landscape created by the adoption of self-driving cars. Some companies have already sought to adjust to today’s changing driving trends: San Francisco-based Metromile, for example, offers pay-per-mile auto insurance for low-mileage drivers and personal coverage to Uber drivers.

Who’s ready to ride in a robocar?

Meanwhile, the American driving public is still getting comfortable with the idea of sitting in a car that drives itself.

According to AAA, three out of four U.S. drivers would be “afraid” to ride in a self-driving car—a sentiment that recalls times more than a century ago, when a vast majority was leery of riding in a carriage that lacked a horse.

Case in point: When one of Google’s self-driving cars crashed into a bus, the incident was widely reported — but, for the most part, not the number of human-driven cars that crashed that day or, for that matter, any other given day. Police reported over 6 million traffic accidents in the U.S. in 2014, nearly 17,000 a day, according to the National Highway Traffic Safety Administration.

The table for safe self-driving cars, meanwhile, has long been set by many of the safety features available in today’s cars, according to John Nielsen, AAA managing director of automobile engineering and repair.

Current car models already feature automatic braking, automatic cruise control, lane departure alarms, blind spot warnings systems, parallel parking assistance monitors — and arrays of sensors that provide input to those systems.

“The building blocks toward self-driving cars are already in today’s vehicles,” Nielsen said in a statement earlier this year. “The technology is constantly improving and well-trusted by those who have experienced it.”

This story was first published on June 1, 2016. It has been updated to reflect the news of the deadly Florida crash and subsequent investigation, as well as the BMW partnership.