In late 2018, Krzysztof Czarnecki, a professor at Canada’s University of Waterloo, built a self-driving car and trained it to navigate surrounding neighborhoods with an annotated driving data set from researchers in Germany.

The vehicle worked well enough to begin with, recognizing Canadian cars and pedestrians just as well as German ones. But then Czarnecki took the autonomous car for a spin in heavy Ontarian snow. It quickly became a calamity on wheels, with the safety driver forced to grab the wheel repeatedly to avert disaster.

The incident highlights a gap in the development of self-driving cars: maneuvering in bad weather. To address the problem, Czarnecki and Steven Waslander, a professor at the University of Toronto, compiled a data set of images from snowy and rainy Canadian roads. It includes footage of foggy camera views, blizzard conditions, and cars sliding around, captured over two winters. The individual frames are annotated so that a machine can interpret what the scene conveys. Autonomous driving systems typically use annotated images to inform algorithms that track a car's position and plan its route.

The Canadian data should help researchers develop and test algorithms against challenging conditions. But the team also hopes it will prompt carmakers and startups to think more about bad-weather driving. “It’s in the interest of everyone to think about this,” Czarnecki says.

The Canadian driving data set includes lane markings and vehicles that are covered with snow. Courtesy of Jeff Hilnbrand

A few companies, including Alphabet’s Waymo and Argo, backed by Ford and Volkswagen, are testing self-driving cars in winter conditions. But as a whole, the industry is far more focused on demonstrating and deploying vehicles in fair-weather locations such as California, Arizona, Texas, and Florida.

As even optimists about self-driving cars temper forecasts on their arrival, testing in sunnier climates is seen as a way to move the technology out of first gear. But the warm-weather bias could limit where autonomous vehicles can be deployed, or cause problems if it is rolled out in colder climates too quickly.

“It’s a very noticeable blind spot,” says Alexandr Wang, CEO of Scale AI, which annotated Czarnecki’s data and works with other autonomous-driving companies. “Deploying autonomous vehicles in bad conditions is not really tackled, or really talked about.”