- OEM interest in driver monitoring systems (DMS) is regulatory driven, currently based on drowsiness distraction and Child Presence Detection (CPD).
- Consumers want safety and wellbeing experiences in their car, seeking for it to become part of their digital-health ecosystem.
- Supporting subscriptions is part of Harman’s envisaged business model.
The idea of a car watching the driver is not new. Systems that scrutinize the driver’s gaze for signs of drowsiness have been around for years, and they are gaining importance as cars offer more advanced driver assistance features. Although the term “in-cabin monitoring” sounds intrusive, the technologies currently under development in this area will shape and define tomorrow’s cockpit, helping make all occupants feel safer, more comfortable and healthier by monitoring their every move. Whether or not we warm to the thought of being under surveillance while driving or sharing a car, DMS should help prevent accidents.
Harman has developed a suite of automotive features focused on enhancing the health and safety of drivers and passengers, including an advanced DMS that can measure a driver’s heart and breathing rate. To learn more, we spoke to Robert Kempf, vice president of Strategy and Product Management, Strategic Business Unit (SBU) Advanced Driver Assistance Systems (ADAS) at Harman Automotive.
The following is an edited transcript of the interview.
S&P Global Mobility: How does Harman Ready Care work?
Robert Kempf: Harman Ready Care is the automotive industry’s first closed-loop interior sensing product that measures a driver’s eye activity, cognitive load and vital signs to determine their level of focus and attention on the road and delivers tailored interventions to increase safety as well as wellbeing. From sensor data, Ready Care uses neuroscience, artificial intelligence (AI) and machine learning to classify if a driver’s behavior is “focused” or “distracted,” based on levels of stress, anxiety, distraction and drowsiness. If classified as “distracted,” it initiates a personalized in-cabin response to help mitigate the dangerous driving situation. Originally launched in September 2022, Harman upgraded its Ready Care features in January 2023 and announced them at the Consumer Electronics Show (CES).
Ready Care uses the same devices, such as cameras, that have been mandated for driver distraction and attention warnings. These devices became a requirement in new M-category and N-category vehicles in Europe, in 2022. An infrared (IR) global shutter camera surveys the driver’s face, with active illumination enabling this to be done accurately in all lighting conditions. This provides real-time data monitoring facial expressions, eye gaze, eyelid state and more. The information is fed into Harman’s predictive-AI solution that assesses it based on insights from neuroscience to determine cognitive distraction, stress or drowsiness. To enhance the predictive model and measurement accuracy, we collaborated with a team of 18 top neuroscientists from Samsung Health, who have years of experience in personalized health and wellbeing monitoring. If determined as “distracted,” the system can then generate customized in-vehicle cabin prompts to trigger a range of behavioral responses with the aim of increasing awareness and keeping eyes and mind on the driving task.
In addition to cognitive load and distraction measurement via IR cameras, through the acquisition of CAARESYS in September 2022, Ready Care now incorporates contactless measurement of human vital signs, such as heart rate, breathing rate and interbeat levels, to determine a driver’s state of wellbeing based on in-cabin radar sensor technology. In addition, Ready Care sensors also detect if a child is left unattended in the vehicle and support general life presence detection in the vehicle.
A driver-facing IR camera or in-cabin radar is used to monitor important vital signs via remote photoplethysmography (RPPG) without the need to wear or use any external devices. In addition, a radar sensor is used for CPD and general life presence detection. This will enable OEMs to provide an alert or notification to the driver to inform them after they have exited the vehicle.
We have seen driver-alert systems being offered by several OEMs using different guises. What is your Unique Selling Proposition (USP)?
Harman Ready Care’s first USP is its ability to decipher the in-cabin camera and sensor data and use predictive AI to deliver robust real-time visual and mental state measurements; in other words, to determine cognitive load. It is incredibly sensitive, so it is able to distinguish driving-related cognitive load (for example, approaching a complex junction) from distraction-related cognitive load (such as daydreaming), as well as identify visual and manual distraction, which could be caused by complex screen data or by other occupants within the vehicle.
Secondly, the system makes closed-loop, in-cabin interventions possible via the Ready Care Software Development Kit (SDK) with supporting application programming interfaces (APIs), allowing OEMs and other third-party suppliers to integrate vehicle features or functions that increase awareness, reduce stress and ultimately bring the driver’s focus back onto the driving task in hand.
Obviously, all drivers and journeys are unique, so Ready Care utilizes advanced machine learning to provide personalized intervention strategies tailored to the individual. For example, adjusting the in-cabin temperature might make driver A. less sleepy but might create a different impact on driver B. As the system learns about the effects of different interventions on the individual, it can adjust its strategy and perhaps instead opt to change the music volume settings for driver B. to help them refocus. One example of a customized intervention is stress-free routing. The customized intervention can detect real-time stress factors such as traffic jams or weather situations and adapt an alternative route to the suggested route to help mitigate the driver’s state of stress while on the road.
Can your solution detect when drivers wearing face masks become distracted or drowsy?
It is a key requirement of the system to work robustly in all relevant scenarios. This includes wearing masks or polarized sunglasses, to give two real life examples. Regarding the scenario where the driver wears a face mask, eye blink frequency, fixation time and gaze direction all contribute to the detection of driver distraction and drowsiness. When drivers wear a face mask, these attributes can still be measured. However, the system performs at its most accurate and covers all “distracted” parameters when the full face can be assessed.
How did CAARESYS make a difference?
The acquisition of CAARESYS has significantly enhanced Harman’s capabilities in the in-cabin safety and wellbeing space. Through its technology and the integration of its experienced and talented team, based in Israel, it has been possible to integrate contactless radar-sensing into Harman products. Ready Care now benefits from the best-in-class vital sign sensing and CPD features. Moving forward, combining the technology, talent, scale, resource and reach of both companies will result in a comprehensive portfolio of in-cabin safety and wellbeing products that are available to the automotive sector in the near term, not just years from now.
What are the key features of the upgrade?
CAARESYS’ acquisition has led to two new features in the upgraded Ready Care product: Vital Sign Sensing and Child Presence Detection (CPD). As mentioned, important driver vital signs such as heart rate, interbeat intervals and breathing rate via RPPG can be monitored using a driver-facing IR camera or in-cabin radar without the need for wearables or external devices.
Meanwhile, CAARESYS’ radar technology enables CPD and general life presence detection after occupants have exited a vehicle, and alerts the driver. CPD will become a future New Car Assessment Program (NCAP) safety requirement, with Euro NCAP rewarding vehicles that offer such a solution as standard from 2023.
What has been the feedback from OEMs to date?
Addressing not just the regulatory aspect of driver monitoring but also looking at how the technology can be used to increase the wellbeing of drivers, and introducing tailored interventions, has resonated extremely well with the OEM community. The concept of harnessing data from multiple sensor inputs across camera and radar is seen as a big plus towards performance and robustness. In addition, the ability to remote sense vital-sign parameters as providing additional insight to the driver state was seen as a key differentiator.
Of course, much interest from OEMs is regulatory driven, currently based on drowsiness distraction and CPD. For the former, OEMs need to add a camera, while the latter requires the integration of radar. The addition of CPD requirements has expanded the original focus on DMS into a full in-cabin monitoring space. Here, Harman has the opportunity to help OEMs go beyond just fulfilling the necessary regulation-imposed requirements and accepting those associated costs. Harman can help OEMs turn this into an opportunity to provide consumers with better experiences and value.
Consumers want safety and wellbeing experiences in their car, seeking for it to become part of their digital-health ecosystem. Ready Care provides OEMs with customer understanding and how they can fulfill consumer needs. Whether that is optimization of heating, ventilating and air conditioning (HVAC) control, lighting or audio settings — all these things have an impact on how the occupants feel when they travel from point A to point B in the vehicle. It requires some different thinking and guidance from Harman on the potential benefits, but through an integrated system, OEMs can support new health and wellbeing applications in the vehicle in the medium-to-long term.
Can OEMs select which features to use or buy?
Absolutely. Ready Care is a robust, affordable and scalable product with features and technology that OEMs can configure into specific bundles of software, sensors and the compute platform to suit requirements, with the possibility of customization to align with model and trim levels. It can comprise anywhere from the full system (electric control units [ECUs] plus sensor plus software) to feature only (sensor plus software to software), or managed-service application. Through this flexibility, it can reinforce or differentiate a carmaker’s brand.
Can you see it being part of a subscription package?
Supporting subscriptions is part of our envisaged business model. With the transformation going on in the automotive industry, we cannot just stick to a one-off license business model. We need to deliver a continuously updatable experience, providing our customers with the latest version of our product; therefore, subscription models are particularly suited to enable that.
How does it work with Ready Vision?
Ready Care is compatible with Harman Ready Vision, which comprises Augmented Reality (AR) heads-up display (HUD) hardware and AR-software products that support greater driver awareness and safety. The result is that drivers receive enhanced audio and/or visual alerts when it has been detected that they are distracted. The alerts improve situational awareness and bring attention back to the task of driving. The key to Ready Vision is that it delivers such alerts in a timely, accurate and nonintrusive manner, ensuring the driver is only given critical information at such key times instead of having to view and process data from multiple screens.
Who are you partnering with to further develop it?
Harman Ready Care has been built as an open platform, so it can easily accommodate new features and functions from Harman, OEMs or third-party suppliers for more customized and personalized intervention options. Such interventions could range from audio or visual prompts to adjusting HVAC, audio or lighting settings, or others linked to advanced driver assistance system (ADAS) features.
As well as the discussed evolution of the product via the CAARESYS acquisition, to date Harman has worked with Gentherm, which provides thermal management technologies. Through this collaboration, Ready Care is fully functional and compatible with Gentherm’s ClimateSense solution, which uses localized convective and conductive heating and cooling, plus radiative heating, to make occupants comfortable. Via Ready Care, this solution provides support by cooling the cabin in cases of drowsiness, because it has been found that lower temperatures can help reduce this.
Driver monitoring for fatigue and distraction has become a major focus of automotive safety regulators and governments worldwide. This trend will likely continue in the Society of Automotive Engineers’ (SAE) Level 2 (partial) and Level 3 (conditional) semi-autonomous vehicles. To what extent are regulations driving increasing demand for DMS (Euro NCAP, European Union General Safety Regulation, Stay Aware For Everyone Act of 2020)?
Regulations are absolutely making this a priority for OEMs, but it brings with it further opportunities for consumer experiences. OEMs can take advantage of this for brand differentiation. Harman’s value proposition is “Consumer Experiences: Automotive Grade” and in-cabin monitoring opens the door for significantly improved health and wellbeing experiences, beyond the safety case that regulation is responding to.
