Snow and Ice Pose a Vexing Obstacle for Self-Driving Cars

In late 2018, Krzysztof Czarnecki, a professor at Canada’s University of Waterloo, built a self-driving car and trained it to navigate surrounding neighborhoods with an annotated driving data set from researchers in Germany.

The vehicle worked well enough to begin with, recognizing Canadian cars and pedestrians just as well as German ones. But then Czarnecki took the autonomous car for a spin in heavy Ontarian snow. It quickly became a calamity on wheels, with the safety driver forced to grab the wheel repeatedly to avert disaster.

The incident highlights a gap in the development of self-driving cars: maneuvering in bad weather. To address the problem, Czarnecki and Steven Waslander, a professor at the University of Toronto, compiled a data set of images from snowy and rainy Canadian roads. It includes footage of foggy camera views, blizzard conditions, and cars sliding around, captured over two winters. The individual frames are annotated so that a machine can interpret what the scene conveys. Autonomous driving systems typically use annotated images to inform algorithms that track a car's position and plan its route.

The Canadian data should help researchers develop and test algorithms against challenging conditions. But the team also hopes it will prompt carmakers and startups to think more about bad-weather driving. “It’s in the interest of everyone to think about this,” Czarnecki says.

A few companies, including Alphabet’s Waymo and Argo, backed by Ford and Volkswagen, are testing self-driving cars in winter conditions. But as a whole, the industry is far more focused on demonstrating and deploying vehicles in fair-weather locations such as California, Arizona, Texas, and Florida.

As even optimists about self-driving cars temper forecasts on their arrival, testing in sunnier climates is seen as a way to move the technology out of first gear. But the warm-weather bias could limit where autonomous vehicles can be deployed, or cause problems if it is rolled out in colder climates too quickly.

“It’s a very noticeable blind spot,” says Alexandr Wang, CEO of Scale AI, which annotated Czarnecki’s data and works with other autonomous-driving companies. “Deploying autonomous vehicles in bad conditions is not really tackled, or really talked about.”

Inclement conditions are challenging for autonomous vehicles for several reasons. Snow and rain can obscure and confuse sensors, hide markings on the road, and make a car perform differently. Beyond this, bad weather represents a difficult test for artificial intelligence algorithms. Programs trained to pick out cars and pedestrians in bright sunshine will struggle to make sense of vehicles topped with piles of snow and people bundled up under layers of clothing.

“Your AI will be erratic,” Czarnecki says of the typical self-driving car faced with snow. “It’s going to see things that aren’t there and also miss things.”

Matthew Johnson-Roberson, a professor at the University of Michigan who is developing a delivery robot optimized for difficult weather conditions, believes tackling bad weather may offer a way to gain a competitive edge. Troubling conditions are a major source of accidents, he says, so they arguably should be a priority.

“The really big players are not focused on this,” says Johnson-Roberson. “There’s still a lot of work to be done on self-driving cars in general, but [driving in bad weather] is going to be a big differentiator, and also important to scaling this.”

Waymo declined to comment, but a spokesperson pointed out that the company’s newest sensors and software are better suited to challenging weather. A spokesperson for Argo said initial deployments of the company’s technology should be able to handle light rain, but “for heavier rains and snow, there still needs to be advancements in both hardware and software.”

When industry players decide to tackle bad weather, they’ll gather lots of training data of their own. But in the meantime, Czarnecki’s data should help the field advance.

“Adverse weather presents tremendous challenges for automated driving technology, and I applaud these researchers for releasing a challenging-weather data set,” says John Leonard, a professor at MIT who is also affiliated with the Toyota Research Institute. “Publicly available data sets can have a huge positive impact on research in the field.”

“The complexity of winter weather is going to take an incredible amount of work for automation technology to tackle,” says Bryan Reimer, a research scientist at MIT specializing in autonomous driving. “Ice-weather conditions are incredibly difficult.”

As for tackling the worst conditions on, say, Interstate 70 through the Rocky Mountains, where 24-hour snow crews are required, Reimer thinks self-driving cars won’t be there for a while. “The only way you’re going to drive that autonomously is to heat the road,” he says.