Heat-Seeking Cameras Could Help Keep Self-Driving Cars Safe

After Uber’s fatal self-driving crash last month in Tempe, Arizona, most observers had two basic question: Why did the car not see Elaine Herzberg crossing the street and stop before hitting her? And how can we stop this happening again, to someone else?

The ride-hailing company has indefinitely suspended its testing program, and is cooperating with the National Transportation Safety Board’s investigation of the crash. The NTSB hasn’t revealed any findings yet, but the lidar—the laser-shooting sensor that should have spotted Herzberg, even in the dark—is an obvious focus. Maybe it had a blind spot, or lacked the resolution to identify Herzberg as a pedestrian. Maybe the car’s software failed to translate those data points into a decision to slam on the brakes or swerve around her.

Whatever happened, the crash makes a life-and-death argument for anything that could make self-driving systems safer. Which is why a company called Flir has spotted a new market opportunity: equipping self-driving cars with heat-seeking cameras.


The WIRED Guide to Self-Driving Cars

“Thermal cameras are really good at seeing the things you most don’t want to hit,” says Mike Walters, the head of product for the Oregon-based company. “Humans, of course.” Where conventional cameras look at visible light, Flir’s sensor focuses on the infrared bit of the spectrum. It can detect tiny differences in temperature—as little as 0.1 degrees Fahrenheit—so something like the cold metal of a bicycle stands out, even on a chilly night. It can do it from up to 240 meters away, a range that matches some of the most capable lidar sensors on the market. And, unlike lidar systems, these sensors don’t sweat fog or direct sunlight.

The temperature-sensitive sensors have popped up in all sorts of places: putting the heat-seeking in heat-seeking missiles, detecting corroded fuses in electronics, spotting people in burning buildings. Flir’s not the only company working to make thermal imaging cameras more robust. “We made it to have the durability of automotive, the battery consumption so I can clip it onto a phone, the shock resistance calibration so it can go on a gun scope, the heat so it can deal with things like fire,” says Tim LeBeau, who oversees corporate strategy for Seek Thermal, a Santa Barbara-based Flir competitor. So putting the things in cars, even with their bumpy outdoor lifestyle, doesn’t seem too tough.

Flir has been applying machine learning techniques to infrared readouts, helping the computer learn to identify things like pedestrians and cyclists.

Indeed, Flir has already put its cameras into half a million cars in the past decade, on models like the BMW 7 Series. In the dark, when animals or people might be hard to spot, drivers would see a bright white human- or deer- or whatever-shaped blog appear on the dashboard screen.

The next logical step—the one Flir has already started to take—is teaching the car’s computer to pick out those obstacles. The company has been applying machine learning techniques to infrared readouts, helping the computer learn to identify things like pedestrians and cyclists, just as others do with conventional camera data. Flir hopes to produce a system that can use thermal imaging to automatically spot problems up ahead, alerting the driver or even applying the brakes as needed.

From that kind of driver assistance feature, it’s easy to imagine a thermal image sensor for a fully self-driving vehicle. It wouldn’t replace today’s radars, cameras, and lidars, but complement them—one more way to spot things that need spotting.

“A key benefit of thermal imagers are their complementary nature,” says Karl Iagnemma, CEO of Nutonomy, a Boston-based self-driving car company acquired by automotive supplier Delphi last year. “They can perform in dark environments and during harsh weather conditions.”

The heat-sensitive cameras are not, of course, perfect. First, they’re expensive. Seek Thermal’s cameras, for example, cost around $2,500 a piece, though it’s working to drop the price to about $1,000. That’s a ton of money compared to regular cameras (that’s why you’ll find this kind of thing in a BMW 7 Series and not a Nissan Sentra), though it’s still a chunk cheaper than lidar.

Plus, Iagnemma notes that their resolution isn’t as good as that of a conventional camera. And the infrared waves they look for don’t penetrate glass, so if the pope rolls by in his bullet-proof enclosure, no thermal camera would be able to spot him. But hey, that’s what regular cameras (and the Swiss Guards) are for.

Eye Openers

Read more: https://www.wired.com/story/self-driving-cars-thermal-image-cameras/

Related posts