Visual interaction is moving to the forefront of HMI design

AutotechInsight Analysis Monthly

This analyst insight is authored by Rightware and S&P Mobility jointly and is aimed at automotive supply chain vendors to summarise the evolving nature of the HMI visual interaction market landscape.

Source: Bosch

Automotive display HMI is growing in demand owning to convenience features and enhancing customer experience. Moreover, in the coming years development of multi-display will not only provide greater freedom of information usage and control between driver and passenger, but it will also reduce product development complexities and costs for automakers. Although, in real-terms automakers will invest more during the initial years to gain longer-term benefits. Automakers are also engaged in the development of ‘Augmented Reality’ (AR) content, either fusing GPS navigation data or Heads up Display data together with overlayed display HMI content.

Growing HMI demand is positively influencing vehicle visual UI (graphics) experience

No matter what vehicle features and functions are designed, it is important to allow the drivers and passengers of a vehicle to take advantage of all that a system has to offer. The visual user interface (UI) or sometimes known as display HMI (used in this insight throughout) is an interface that allows the users to achieve this. Display HMIs are transforming the way users interact with vehicles. Users are expecting a lot from vehicle display HMI, in fairness they even evaluate vehicle experience based on the experience they have with HMI. In this regard, the visual display HMI, companion apps, virtual assistant, navigation, and display-based fully digital instrument cluster – have become differentiating factors for vehicle users and influence buying decisions. Most buyers want their vehicle to seamlessly blend into their digital life experience. Considering these aspects of user expectations, it is prudent to think of automotive display HMI software as a continuous design and evolving development process as opposed to launching once and then forgetting about it.

From an automaker’s point of view, vehicle display HMI is deployed using a few design strategies in mind, such as providing the right information at right time, prioritizing primary information (speed, location, braking, etc.) for safe driving while avoiding interference by secondary information (media player, telephony, etc). to distract drivers’ attention, avoiding information overload, placing information in the correct eyesight location, etc. Automakers are developing smart display HMI systems using artificial intelligence (AI) and driver workload algorithms to predict when the driver needs specific information and present it to them at the right time. Furthermore, the ‘Driver Monitoring System’ (DMS) is designed to track driver behaviours during certain driving circumstances and provide necessary alerts or recommendations of actions for safe driving. The market will witness a high growth of this safe and protective innovation in vehicle HMI and interiors in the midterm. Automakers are also getting serious about their display user interface which they see as a premium innovation platform for the interiors of their vehicles and help improve brand perception.

Source: Rightware

Growing display HMI development trend

In terms of new innovations within the coming years (between three to five) industry will witness several developments starting with VR (Virtual Reality), where virtual reality streaming content will be pushed inside a vehicle, using display HMI design & development tools such as Kanzi VR, automakers are developing simulated scenarios where users can see the user interface (UI) areas, how the driver can be distracted during display usage, are all elements of information is visible correctly, assessing the right impact of over or under information in the screen, etc.  The aim here is to get the design and development right the first time with the aid of VR HMI tools. Moreover, early simulation of HMI use cases reduces development lifecycles and leads to a faster tie market.

Source: Rightware, Kanzi VR feature pack

With multi-display development, automakers are taking advantage of hardware consolidation using domain controllers and centralizing HMI software platforms. For example, in today’s vehicle, the use of dedicated display per processing hardware is prevalent, meaning, an OEM would be using a tier-1 supplier for their Instrument Cluster (IC) development (from visual UI & software development to hardware integration) while a separate tier-1 supplier for Infotainment Center Stack Display (CSD). In such scenarios, development engagement includes two product groups and suppliers. This process is quite cumbersome apart from navigation where visual UI data is shared between two components, displays, and hardware. Going forward automakers are developing a third display for passengers in the same cockpit location. In doing so, automakers are utilizing single command and control for all of these displays. This development will bring a few benefits for automakers, for example, it will help reduce development and maintenance costs by integrating all HMI platforms under a single tier-1 supplier, as such, reducing overheads and integration costs.

However, multi-display development with central command and control offers some challenges for automakers. For example, this type of development is new to the industry and the market will witness the evolution of display HMI design, associated tools, and processes to support multiple displays in the coming years. To strike the right balance between graphical design, artists, and software development, HMI tool providers have started to develop a collaborative HMI design toolchain to cater to all stakeholders in the design process. With such a tool, designers, technical artists, and software developers can work in a collaborative environment that is non-existent today.

One of the HMI features that automakers are actively engaging in development is ‘3D virtual assistant’. This is one of the most popular demand features amongst automakers: virtual 3D characters that will talk to the driver using natural speech & recognition. However, the lack of HMI tools to support display has become a bottleneck for the industry. Although some automakers trying to develop such features using customized and unique toolsets, there is no widespread availability of similar toolsets. Among these examples, including a collaboration between German premium automaker Mercedes-Benz and Unity with their 3D HMI toolset for their virtual avatar inside the vehicle.

The auto industry market will also witness a higher level of adoption of ‘Augmented Reality’ (AR) content, either fusing GPS navigation data or Heads Up Display (HUD) display data with HMI content in the midterm. Data from S&P Mobility’s primary market study suggest the majority of the European & North American premium automakers will deploy AR technology between 2025 and 2027 into their high-segment vehicle Nameplates. Furthermore, automakers are devising future autonomy use cases to develop 3D animated driver assistance content to ease the use of future automated driving experiences. However, a few challenges need to be resolved before considering such features, for example, occlusion difficulties together with real-time ‘point of interest’ POI video feed of the navigation system. While the resolution for these challenges is under investigation, automakers are also planning to bring ‘photorealistic’ displays to their next generation of vehicles. For example, Swedish automaker Volvo announced that they will be using Epic Games’ free-to-use Unreal Engine to illustrate picture-perfect surroundings on vehicle visual screens. This type of solution allows for rendering visuals quickly and in high definition. Furthermore, the likes of General Motors, Nissan, SAIC IM Motors, and Xpeng Motors are also exploring 3D integrated user interaction methods using ‘Unreal Engine’ in their intelligent cockpit display systems.

Influence of software programming interface: OpenGL in display HMI development

The cross-language and HMI programming for rendering 2D and 3D like OpenGL ES driver libraries (managed by Kronos Group) are becoming increasingly ubiquitous among automakers. Almost all semiconductor chipsets facilitate OpenGL software libraries. Some of the advanced semiconductor chipsets (e.g., Intel, NVIDIA, Qualcomm, Mediatek, Samsung, etc.) also started to support Vulkan which offers higher performance and efficient usage of CPU and GPU compared to OpenGL. Vulkan also allows efficient multithreading capabilities with multi-core CPUs and GPUs on graphics processing that handles computation for graphics. Both cross-platform HMI design programming language API utilization is growing at a rate of 2.9% CAGR between 2021 and 2028, with OpenGL (0.9% CAGR) growing slowly due to the maturity of the technology compared to Vulkan (25.2% CAGR).

In the medium to long- term majority of automakers will start to discontinue low-end display HMI platforms and start adopting high-end display HMI. This is due to higher take rates in future cockpit displays. The market will witness a surge in vehicles with display-based driver information and center stack solutions. Furthermore, the S&P market study clearly shows, even the entry-level vehicle segments will integrate processing hardware that will be powerful enough to use OpenGL libraries for HMI development.

S&P research shows ‘Mid’ and ‘Standard’ price class vehicles will start to increase adoption of the display-based instrument cluster, and growth of OpenGL usage will corroborate fitment increase from the midterm, especially from 2025 onwards. With a compound annual growth rate of 16.1%, the volume of OpenGL usage in the display-based instrument cluster will hit 46m by 2028 according to the S&P market study. In current terms primarily high-end premium vehicle brands and segments are using high-end HMI display platforms with OpenGL backbone. Many of the current ‘Entry’, ‘Mid’, and ‘Standard’ price class vehicles are within the category of low-end display HMI: using a small textual display inside an instrument cluster with analog gauge instruments and CSD rendering without no 2D or 3D.

In such hardware, there is no support for OpenGL and have proprietary rendering process. Altia, QT, and Rightware-Kanzi Lite are a few toolchain providers for low-end HMI development with supporting proprietary APIs and development activities. S&P market study suggests that the majority of the HMI toolchain providers are not investing in the support of low-end HMI.


Lack of standard approach causing development complexity and costs

The majority of the silicon vendors are facilitating open-source and standard libraries in vector graphics that is helping to improve cost efficiencies and quality of display HMI and avoiding the bottleneck of underlying UX processing. However, in the coming years, the automotive industry will witness some changes in HMI feature & function development that will influence the display HMI development processes. For example, some of the silicon vendors are developing support with ray tracing for future automotive 3D computer graphics and rendering algorithms e.g., NVIDIA RTX technology.

Furthermore, automotive HMI is also evolving towards more intelligent user interaction methods with silicon vendors integrating and developing special features with Artificial Intelligence (AI). The growth with these new and special software features is fuelled by three key factors, 1) demand for immersive HMI experience within the vehicle, 2) support for the growing trend toward game-a-like display interaction and visual experience, and 3) growing trends towards AI in HMI technologies.

While these new features will certainly improve the quality of automotive display HMI user experience, however, the industry will also witness a development cost growth as well as an increase in complexities. For example, all of these new software features will have to be consolidated with the automotive HMI toolchain for rendering integration and vehicle UX processing which requires development resources. Furthermore, there is a lack of a common standard approach for developing these features and every silicon vendor and supplier is developing these using their own internal rules and standards. To make matter worse, the automotive industry is at the early transition stages with these technologies and features, as such, there is a high level of reliance on each specific silicon vendor and software supplier for their proprietary solutions. However, automakers together with their silicon vendor partners are working collaboratively to reduce dependencies on proprietary solutions and devise a common standard and collaborative approach that will reduce development costs and improve quality.


Author: Dr. Tawhid Khan, S&P Mobility

Guest Author: Giuseppe Maione, Rightware

preload preload preload preload preload preload