Evolving the Human Machine Interface Part III

The concept of Presence in Virtual Reality (VR) has been gaining popularity over the past year, particularly within the gaming community. With consumer VR devices in development from Oculus, Sony, and more than likely Microsoft, Presence has become the metric by which we evaluate all VR experiences. But Presence is difficult to describe to someone who has never tried VR. “It’s like I was actually there. It made me feel like what I was seeing was actually happening to me, as though I was experiencing it for real,” is how one colleague described the experience.

Presence in VR triggers the same physical and emotional responses one would normally associate with a real world situation, it is the wonderfully magical experience of VR. But how is Presence achieved? While many research studies have provided a variety of subjective descriptions for Presence, there seem to be 3 common variables that affect tele or virtual Presence most:

1.  Sensory input: at minimum the ability to display spatial awareness

2.  Control: the ability to modify one’s view and interact with the environment

3.  Cognition: our individual ability to process and respond to sensory input

Because the nature of VR isolates the user from real world visual input, if the device’s sensory input and control are inadequate or missing then the effect of Presence fails, the results of which are oftentimes met will ill side effects: You feel sick to your stomach!

Sensory Input

For those who have tried VR, at some point or another you’ve felt queasy. That point which the experience turns from wonder to whoa, has been an unfortunate side effect throughout the development of VR. As Michael Abrash presented at Valve’s Steam Dev days, the hurdles needed to overcome VR sickness and achieve Presence are within reach. In the video below, Michael expertly summarizes the technical hurdles to achieve a believable sense of Presence in VR.

“What VR Could, Should, and Almost Certainly Will Be within Two Years” Steam Dev Days 2014

Michael Abrash, Valve Softwrae

Control

To achieve a minimum level of Presence, head tracking is used to display the VR image to match the users own head position and orientation. While this helps to create the sense of spatial awareness within VR, it still doesn’t make someone’s experience truly “Present.” To do that, we need to add the representation of ourselves, either through an avatar or our own body image. Viewing our physical presence in VR, known as body awareness, creates an instant sense of scale and helps to ground the user within the experience. In the video sample below, Untold Games is creating body awareness through avatar control.

“Loading Human, true body awareness” Untold Games 2014

Without body awareness, VR can feel more like an out-of-body experience. Everything “looks” real and the user has spatial awareness, but the user’s body and movements are not reflected therefore the user does not feel actually present. Combining body awareness with VR’s spatial awareness creates a strong bond between the user and the experience.

Cognition

The third perimeter of Presence is us. The feeling of Presence in VR is directly influenced by our personal ability to process and react to environmental changes in the real world. It’s likely that many of us will not have the same reactions to the experiences within VR. If you get sick riding in cars easily, then VR motion will give you the same sensation. If you’re afraid of heights, fire, spiders, etc. you’re going to have the same strong reactions and feelings in VR. Our individual real life experience influences our perception and reactions to VR. This can lead to some interesting situations, in particular with gaming. For example one player may be relatively unaffected by a situation or challenge, while another may be strongly affected.

Obviously the conditions of Presence are perceptual only. In most cases we’re not at the same physical risk in virtual environments as we would be in real life. But our own cognition coupled with VR’s ability to create Presence is why VR is such a popular field for everything from gaming and entertainment to therapy and rehabilitation.

Once we start to overcome these technical hurdles and provide a basic level of Presence, we next need to understand what it will ultimately enable. What does Presence provide for us in an experience other than merely perceiving the experience as real-like? We’ll explore that idea in the next segment, and try to understand where Presence will have the most impact.

Filed under: AR User Experience | 4 Comments »

How the World Is Finally Ready For Virtual and Augmented Reality

By Mike Nichols, VP, Content and Applications at SoftKinetic

The year is 1979 and Richard Bolt, a student at MIT, demonstrates a program that enables the control of a graphic interface by combining both speech and gesture recognition. As the video of his thesis below demonstrates, Richard points at a projected screen image and issues a variety of verbal commands like “put that there”, to control the placement of images within a graphical interface in what he calls a “natural user modality”.

“Put-That-There”: Voice and Gesture at the Graphics Interface

Richard A. Bolt, Architecture Machine Group

Massachusetts Institute of Technology – under contract with the Cybernetics Technology Division of the Defense Advanced Research Projects Agency, 1979.

What Bolt demonstrated in 1979 was the first natural user interface. A simple pointing gesture combined with a verbal command, while an innate task in human communication, was and still is difficult for machines to understand correctly. It would take another 30 years for a consumer product to appear that might just fulfill that vision.

A new direction

In the years following Richard’s research, technology would advance to offer another choice to improve the Human Machine Interface (HMI). By the mid 80’s the mouse, a pointing device for 2D screen navigation, had evolved to provide an accurate, cost effective, and convenient method for navigating a graphical interface. Popularized by Apple’s Lisa and Macintosh computers, and supported by the largest software developer Microsoft, the mouse would become the primary input for computer navigation over the next 20 years.

“The Macintosh uses an experimental pointing device called a ‘mouse’. There is no evidence that people want to use these things.”

San Francisco Examiner, John C. Dvorak – image provided by…

In 2007, technology advancements helped Apple once again popularize an equally controversial device, the iPhone. With its touch sensitive screen and gesture recognition, the touch interface in all its forms has now become the dominant form of HMI.

The rebirth of natural gesture

Although seemingly dormant throughout the 80’s and 90’s, research continued to refine a variety of methods for depth and gesture recognition. In 2003 Sony released the Eye Toy for use with the PlayStation2. The Eye Toy enabled Augmented Reality (AR) experiences and could track simple body motions. Then in 2005 Nintendo premiered a new console, the Wii, which used infrared in combination with handheld controllers to detect hand motions for video games. The Wii controllers, with their improved precision over Sony’s Eye Toy, proved wildly successful and set the stage for the next evolution in natural gesture.

In 2009 Microsoft announced the Kinect for Xbox 360, with its ability to read human motions to control our games and media user interface (UI), without the aid of physical controllers.

What Richard Bolt had demonstrated some 30+ years prior was finally within grasp. Since the premier of Kinect we’ve seen more progress in the development of computer vision and recognition technologies than in the previous 35 years combined. Products like the Asus Xtion, Creative Senz3D, and Leap Motion have inspired an energetic global community of developers to create countless experiences across a broad spectrum of use cases.

The future’s so bright

To this day, Richard’s research speaks to the core of what natural gesture technology aims to achieve, that “natural user modality”. While advances in HMI have continued to iterate and improve over time, the medium for our visual interaction has remained relatively intact: the screen. Navigation of our modern UI has been forced to work within the limits of the 2D screen. With the emergence of AR and VR, our traditional forms of HMI do not provide the same accessible input as the mouse and touch interfaces of the past. Our HMI must evolve to allow users the ability to interact to the scene and not the screen.

CES 2014, Road to VR, SoftKinetic premiers hand and finger tracking for VR.

Next, we’ll explore how sensors, not controllers, will provide the “natural user modality” that will propel AR and VR to become more pervasive than mobile is today. The answer, it seems, may be right in front of us…we just need to reach out and grab it.

Filed under: AR User Experience | 4 Comments »