The first semi-autonomous cars can handle highway driving surprisingly well. But as they lull us into distraction, we risk an increase in the number of avoidable fatalities on our roads.

According to statistics recently published by the National Highway Traffic Safety Administration, 35,092 people died in crashes on U.S. roads during 2015, up from 32,744 in 2014. That’s a 7.2 percent increase, and the largest year-on-year percentage jump in nearly 50 years. And according to Debbie Hersman of the National Safety Council, speaking to NPR, that trend has been observed to continue into 2016.

The increase in the absolute number of fatalities can partly be explained by the fact that there are more cars on the road—the number of overall miles driven rose by 3.5 percent. But NHTSA figures also show that the number of fatalities in accidents said to be influenced by distracted driving—the result of, say, texting on a cell phone or fiddling with the car stereo—have risen by 8.8 percent year-on-year, from 3,197 to 3,477.

Many of the companies developing driverless cars cite safety as a primary motivation, arguing that autonomous vehicles will remove human error from our roads. Even President Obama, writing about new government guidelines for autonomous vehicles in the Pittsburgh Post-Gazette, said that self-driving cars could provide “safer, more accessible driving” with “the potential to save tens of thousands of lives each year.”

For now, though, self-driving cars are only partially autonomous. If the vehicle is unable to make a confident decision, it alerts the driver to take control of the wheel. If the driver is distracted and the autonomous system does not work properly, or if the human places too much confidence in the abilities of the driver aid and ignores the warnings and the road, things can go wrong.

Earlier this year, a Tesla owner died while his car was driving autonomously. That incident saw the car’s semi-autonomous driver aid, known as Autopilot, fail to detect a semi trailer across the road ahead. The final results of the NHTSA investigation into the accident are yet to be published, but data from the vehicle did reveal that the driver did not apply the brakes before impact. That suggests that it’s likely that he wasn’t watching the road.

With a car handling the basics of driving, it’s obviously easy for attention to wander. Brandon Schoettle, from the Transportation Research Institute University of Michigan, warns that the current systems in place to alert drivers to take over the wheel may not be satisfactory. In fact, he suggests, the rate of fatalities as a result of distracted driving could even increase if the warnings in such semi-autonomous systems don’t improve.

“It is critical that the hand-off between vehicle and driver be done properly and safely or the potential safety gains of such systems may be canceled out by the increased risk during this transition,” he says. “It is difficult to regain situational awareness when engaged in other activities in a vehicle, so if drivers are not given ample time to respond, or the system simply fails, then the fatality rate may not improve or may even worsen until these systems are perfected.”

Tesla is currently planning an update to its Autopilot software which will demand that drivers don’t take their hands from the wheel for extended periods. If a driver is warned three times, the feature will be disabled until the car is stopped and the engine restarted. That’s something that Nissan’s recently announced ProPilot system will also do. GM, meanwhile, is developing a system that will use eye-tracking software to ensure that a driver’s attention doesn’t wander from the road.

In the longer term, many aim for cars to become fully autonomous, at which point the need for handing over control will become irrelevant. Those kinds of vehicles “are viewed by many as the most reasonable way of dealing with the added, and increasing, distractions in the vehicle,” says Schoettle. “Simply take the driver out of the loop altogether.”