Microsoft teams up with VW to make HoloLens work in cars

Microsoft has officially announced a new “moving platform” feature for the HoloLens 2, which is designed to let the augmented reality headset work in places like cars. It addresses a long-standing HoloLens issue of moving environments confusing the headset’s sensors. The enhancement was developed in collaboration with Volkswagen, which has been experimenting with using the headset as a heads-up display in its vehicles.

As Microsoft’s blog post explains, its augmented reality headset tracks movement using a combination of camera sensors and an inertial measurement unit (which typically includes accelerometers and gyroscopes). But in a car, the readings from these two sensors can conflict; the headset senses movement but sees a static environment. In other words, it was getting car sick.

That’s what VW discovered after it started investigating the use of augmented reality headsets to teach drivers how to get around a racetrack faster. It started collaborating with Microsoft to fix the sensor problem in 2018, and, eventually, the two developed a prototype system that allowed a car to display real-time information on a connected headset.

The system allows virtual objects to be placed both inside and outside of the vehicle. One image released by Microsoft (above) shows the HoloLens 2 projecting a virtual map onto the dashboard of a car, with navigation arrows appearing ahead at key intersections. A second shows it alerting the driver to an upcoming pedestrian crossing.

Image: Microsoft

VW’s existing cars already integrate some augmented reality elements. Its recent ID electric cars feature an augmented reality heads-up display that projects data from the car — including current speed and navigation instructions — onto the windshield, where it’s easier for the driver to see without taking their eyes off the road.

Unsurprisingly, given Microsoft is firmly targeting the $3,500 HoloLens 2 at enterprise users, there’s no sign of it getting a consumer rollout anytime soon. (Although Microsoft’s Marc Pollefeys, who worked on the project, says it’s interested in consumer use cases “in the longer term.”) Instead, Microsoft suggests the first beneficiaries of the new functionality could be maritime companies that can connect workers with remote mechanical experts who can look through a worker’s HoloLens 2 to diagnose a problem.

Perhaps most importantly, it’s clear that Microsoft is still actively developing its augmented reality headset platform despite reports from earlier this year that the project was experiencing difficulties. The company has reportedly lost 70 employees from its HoloLens team over the past year, with over 40 heading to Meta to support its attempt to pivot into a metaverse company.