Smile, You’re on an In-Car Camera!
Interior cameras began by combating driver distraction and drowsiness and now can capture everything from occupant health to a baby left in the back seat.
Do you know what you look like when you’re behind the wheel? If you drive a late-model car, chances are a camera does. Those cameras can help determine whether you’re distracted or drowsy, and soon, they’ll be able to track not only how you and your passengers behave but also your size, weight—even your vital signs. They’ll also be able to detect everything from a baby left in a back seat to whether someone got sick in a car.
0:00 / 0:00
In-car cameras first started appearing in luxury vehicles close to two decades ago, and in recent years they’ve become more commonplace—so much so that they’ll soon be mandatory in Europe. They were initially developed in large part to help drivers keep their eyes on the road and have since evolved to support hands-free automated driving. Now, cameras are increasingly being incorporated into what are known as driver monitoring systems (DMS). And they’re evolving to do everything from recognizing different drivers and adjusting the vehicle settings for each to having a simple gaze activate various vehicle features to sensing the actions of occupants to accessing their health and emotional states.
But will all this advanced monitoring technology designed to make the time we spend in cars safer and more comfortable come at a cost? As is the case with every emerging technology designed to track humans and human behavior, there are trade-offs in the form of emerging privacy and security concerns.
Cameras Watching Outside and Inside the Car
Drivers should, of course, always keep their eyes firmly affixed to the road ahead while constantly being aware of what’s going on behind and around them. But whether it’s glancing away to check navigation directions, changing a radio station, or simply reading a traffic sign, no one keeps their eyes locked on the road 100 percent of the time. And with a pinging smartphone riding shotgun or even just a chatty passenger, it’s easier than ever to be distracted while behind the wheel.
Taking your eyes off the road for just four seconds—less time than it takes to send or read a text message—at 55 mph is equivalent to driving the length of a football field with your eyes closed. The National Highway Traffic Safety Administration said that in 2021, more than 3,500 people were killed in motor vehicle crashes directly attributed to distracted driving.
Cameras and the technologies they help power have been working to lower those numbers. Since 2018, backup cameras have been mandated in all new U.S. market passenger vehicles. And most of today’s new cars sold here now come with an advanced driver assistance system (ADAS) package that usually includes adaptive cruise control, forward collision warning with automatic emergency braking, and lane keep assist to help avoid or mitigate damage from crashes caused by driver distraction. These systems and others, such as blind-spot warning and pedestrian detection, use exterior cameras in conjunction with other sensors that act as a second set of eyes for drivers.
Now, the focus is turning to in-car cameras and the DMS tech they enable, both here and elsewhere. The European Union has implemented mandates that all new vehicles sold—including passenger cars and trucks as well as commercial vehicles such as big rigs and buses—be equipped with multiple ADAS systems including DMS tech as part of the greater rollout. In addition, the European New Car Assessment Programme, or Euro NCAP, (the E.U.’s equivalent of NHTSA’s 5-Star Safety Ratings system for new vehicles), favors the inclusion of a DMS to achieve a top five-star rating.
“For most carmakers it’s extremely important to maximize their star ratings,” said Brian Brackenbury, director of product development at Gentex, a supplier of automotive mirrors, vision, and connected technologies that it combines into DMS systems. “By 2026 they’ll be adding DMS in everything from a Toyota Yaris to Mercedes-Benz S-Class.”
Eye Control of Vehicle Features
Toyota was the first automaker to introduce an in-car camera system for Lexus vehicles back in 2006. It used an infrared camera mounted on the top of the steering column, and it would issue audible and visual alerts if the driver looked away for too long.
Similar systems have spread to other luxury brands, including BMW, Cadillac, Mercedes-Benz, Tesla, and Volvo, and are primarily used to support automated hands-free driving applications, according to Nick DiFiore, senior VP and general manager of automotive at DMS supplier Seeing Machines. “GM started that with its Super Cruise system,” DiFiore said. “That was our first production launch.”
The systems use various methods, such as steering inputs, to detect whether the driver is engaged, Brackenbury said. “But they're also using cameras to confirm where a driver is looking. If a car is in semi-autonomous mode driving down the freeway, automakers want to ensure that the driver’s capable of taking control.”
DMS technology is increasingly trickling down to non-luxury brands. Advanced facial-recognition systems are being used to track a driver’s head and eye movements and to identify individual drivers. Subaru’s DriverFocus feature, for example, can recognize up to five drivers and tailor vehicle settings such as seat and side-mirror settings to each.
A driver’s eyes are also now able to control vehicle features, such as BMW’s new Highway Assist with automatic lane changing, which will make its debut in the 2024 5 Series. It deploys gaze control technology developed by Seeing Machines. “The way you tell it to change lanes is looking at the sideview mirror for two or three seconds, and the car will take that as a cue to change lanes without the driver pressing any buttons or controls,” DiFiore said.
Capturing the Entire Interior
Gentex’s latest DMS technology integrates into a regular rearview mirror housing, using cameras and infrared emitters designed to monitor the car's interior and occupants. It also uses structured-light technology such as lidar to detect passengers and objects by deploying up to 14,000 beams of infrared light. It can even sense a child’s respiration in a rearward-facing car seat.
DMS technology can also help automakers more finely tune airbag deployment. “We know how a person is seated, their body pose, and what are they doing with their hands,” Brackenbury said. “A lot of advanced airbag systems have multistage deployment, and knowing a car occupant’s position, mass, and size can ensure an airbag deploys with the right amount of energy.”
Combining DMS with other sensing technologies also allows for the monitoring of driver behavior and health. “By increasing [DMS] capability for vital health monitoring, emotional state detection, and other things, we can add more value to what’s already going to be in the vehicle,” said Eric Miciuda, head of display solutions business area UX for Tier 1 supplier Continental North America. Continental is developing DMS technology to capture biometric data such as blood pressure and pulse and to use galvanic skin response to better authenticate drivers. “It can tell not just that your pupils are dilated but also detect if your face is flush, if you’re sweating, or stressed out.”
Brackenbury said Gentex is using AI machine vision to learn more about the health and emotions of vehicle occupants. “Once we can train these systems, it’s not just about identifying where drivers are looking or if they're sleepy,” he said. “We’re involved in early research to identify health parameters such as breath rate and heart rate, and many of these systems can detect various emotions.”
This technology is still in the research phase, Brackenbury noted. “But emotions and health are a key part of where we're focused in the future,” he said. “You're going to definitely see that start coming into cars as there are more sensing capabilities in the cabin,” he said.
Caught on Camera
Whenever a camera is present—whether in a car or a public place—it brings up privacy concerns. And a recent report by the Mozilla Foundation *Privacy Not Included project titled “It’s Official: Cars Are the Worst Product Category We Have Ever Reviewed for Privacy” probably won’t make car owners comfortable about being on camera. While the report primarily focused on connected car data, *Privacy Not Included program director Jen Caltrider said DMS technology opens a new avenue of concern. “To think that now cars have these completely privacy-violating cameras on us all the time studying our moods and emotions creeps me out,” Caltrider said.
Reuters reported that between 2019 and 2022, Tesla employees shared among themselves “sometimes highly invasive videos and images” recorded by in-car cameras. One of the images showed a nude man approaching a vehicle; another depicted a Tesla driving in a residential area at high speed and hitting a child riding a bike.
While Tesla’s online “Customer Privacy Notice” states that its “camera recordings remain anonymous and are not linked to you or your vehicle,” several former employees told Reuters they could determine the location of recordings, potentially revealing where a Tesla owner lived. Partially in response to the Tesla reports, California Governor Gavin Newsom signed legislation in late 2023 that mandates car dealers inform consumers if a potential purchase includes one or more in-vehicle cameras and prevents profiting off driver data without consent.
Brackenbury said Gentex’s in-car cameras don’t store video. “We’re analyzing a frame of video, doing what we need with it, and then deleting the frame,” he said. “OEMs are being very cautious from what we can see. I don't know where this will go long term with regulators and whether they'll want to store driver information for a crash like a black box. We won't get to decide that.”
In-Car Selfies and Driver Monitoring When There’s No Driver
Automakers are just starting to fully explore the capabilities of DMS technology, DiFiore said. One of the uses cases: Apple CarPlay and Android Auto include apps for Zoom and Teams for audio calls, and there's interest in adding features such as video conferencing to vehicles with automated driving systems. “They're using our technology to track the driver and blur the rest of the vehicle in case there are passengers, similar to what you do with conference call at home,” he said.
Brackenbury added that a DMS also allows for “having some fun and convenience and not just regulatory stuff.” For example, he said in China the in-car selfie is a thing. “They're crazy about posting what's going on inside the vehicles in their social media feed,” he said. “Many Chinese cars have a button on the steering wheel that basically takes a selfie of what’s going on in the cabin and sends it to a WeChat feed.”
DMS technology can also watch—and even smell—for things even when there isn’t a driver in a vehicle. Brackenbury said Gentex is working on chemical sensing for robotaxis. “In a completely autonomous age, you’re going to need to be able to detect various chemicals,” he said. “Whether that’s an incendiary device, VOCs, or a biohazard. One of the things we're just beginning to bring to market is digital olfaction using different ways to detect particulates and chemicals.”
He added that Gentex is working with autonomous-taxi company Zoox on a particulate sensor that looks for smoking and vaping in a vehicle or even for when someone vomits. “It tells Zoox it’s been detected, and then they could determine if they want to charge that passenger a cleaning fee or something like that,” he said. “If someone got sick in a car, you don’t want it to be going to the next passenger.”
So even if it can’t catch it on camera, your car or the driverless car you hail in the future will potentially be able to know who you are and what you’re doing on multiple levels. It’s a future that despite its stated benefits for the driver and passengers is one where you’re increasingly being monitored—whether you want to be or not.
