Games Alfresco

Harnessing the Power of Human Vision

By Mike Nichols, VP Content and Applications at SoftKinetic

For some time now, we’ve been in the midst of a transition away from computing on a single screen. Advances in technology, combined with the accessibility of the touch-based Human Machine Interface (HMI) have enabled mobile computing to explode. This trend will undoubtedly continue to evolve as we segue into more wearable Augmented Reality (AR) and Virtual Reality (VR) technologies. While AR and VR may provide substantially different experiences to their flat screen contemporaries, both AR and VR face similar issues of usability. Specifically, how and what are the most accessible ways to interact using these devices?

The history of HMI development for both AR and VR has iterated along similar paths of using physical controllers to provide user navigation. Although the use of physical controls has been a necessity in the past, if they remain the primary input, these tethered devices will only serve as shackles that prevent AR and VR from reaching full potential as wearable devices. While physical control devices can and do undoubtedly add a feeling of immersion to an experience, in particular with gaming, you would no more want a smart-phone that was only controllable via a special glove, then you would want to control your smart-glass through a tethered controller. As the technology for AR and VR continues to evolve it will eventually need embedded visual and audio sensors to support the HMI. In particular, visual sensors to support a full suite of human interactions that will integrate with our daily activities in a more natural and seamless way than our mobile devices do today.

In Depth

A depth sensor is the single most transformative technology for AR and VR displays because it is able to see the environment as 3-dimensional data, much like you or I do with our own eyes. It’s the key piece of technology that provides us with the building blocks needed to interact with our environments – virtual or otherwise. The depth sensor allows us to reach out and manipulate virtual objects and UI by tracking our hands and fingers

A scenes’ depth information can be used for surface and object detection, then overlaid with graphics displayed relative to any surface at the correct perspective to our heads’ position and angle. Depth recognition combined with AR and VR presents a profound change from the way we receive and interact with our 2D digital sources today. To simulate this effect, the video below shows an example of how a process known as projection mapping can transform even simple white cards in astonishing ways.

“Box” Bot & Dolly

It’s not hard to imagine how AR combined with depth can be used to transform our view of the world around us. To not only augment our world view with information, but even transform live entertainment such as theater, concerts, sporting events, even photography and more.

Take a more common example like navigation. Today, when we use our smart phones or GPS devices to navigate, our brain has to translate the 2D information on the screen into the real world. Transference of information from one context to another is a learned activity and often confusing for many people. We’ve all missed a turn from time-to-time and blamed the GPS for confusing directions. In contrast, when navigating with depth-enabled AR glasses the path will be displayed as if being projected into the environment, not overlaid on a flat simulated screen. Displaying projected graphics mapped to our environment creates more context aware interactions, and becomes easier to parse relevant information based on distance and view angle.

Bridge the gap

As we look to the future of AR and VR they will both certainly require new approaches to enable an accessible HMI. But that won’t happen overnight. With commercialized VR products from the likes of Oculus, Sony and more coming soon we’ll have to support an interactive bridge to a new HMI through existing controllers. Both Sony and Microsoft already offer depth cameras for their systems that support depth recognition and human tracking. The new Oculus development kit includes a camera for tracking head position.

We’re going to learn a lot about what interactions work well and those that do not over the next few years. With technology advances still a ways off to make commercial AR glass feasible as a mass market option, it’s even more important to learn from VR. Everything done to make VR more accessible will make AR better.

Stay tuned for our next guest post, where we’ll take a closer look at how depth will provide a deeper and more connected experience.

Filed under: Uncategorized |