Thoughts on the HoloLens

Last week, Microsoft publicly announced their take on augmented reality, the HoloLens. Some people are already complaining about the technically incorrect use of the term "holographic", but certainly the ostensibly live demo is impressive. The combination of an augmented reality display with Kinect-style gesture recognition could be a significant step further towards the non-dystopic part of a Minority Report-like future.

There is one important technical issue though that I'm worried the Microsoft marketing machine may be trying to bury. Think about how the visual system works: it's all about photons arriving in your eye at the right time in the right place. Usually, those photons are reflections off objects in the world around us. Augmented reality of the type that Microsoft claims to have built here must allow a mixture of such reflected photons and photons that are generated by a process similar to computer screens and projectors.

Here's the thing: All existing augmented reality systems are additive in terms of brightness. They always let the real-world photons through, and add their own augmented-reality photons to do their thing. This means, for example, that this type of system cannot show black text on a white (real-world) wall. It also means that augmented-reality objects appear transparent.

The demo video pretends that the augmented-reality objects are (or can be) opaque, but keep in mind that you're looking at a rendered simulation. What you see in the video is not really what the AR user is seeing, even though the presenter claims that it is. The fact that the "visor" of the systems appears so dark could also hint at a "solution" which simply decides to make the real world quite dark, so that the augmented-reality objects can stand out better.

Now the laws of physics allow one to build a "perfect" AR system that can show opaque objects. It requires building a lens that can filter the real-world photons out selectively, on a per-pixel basis. This kind of technology isn't crazy science-fiction; after all, we can manipulate matter on a nanometer scale. It's just that as far as I know, nobody has built such a thing yet, and if Microsoft have done so, then why are they so quiet about this issue in their announcement?

So who knows. Maybe I missed something in the announcement. For now, I'm skeptical.