Virtual and Augmented Reality

Test

A user drives a virtual car in a CAVE while speaking with an avatar controlled by a remote user. From Daily, M., Howard, M., Jerald, J., Lee, C., Martin, K., McInnes, D., Tinker, P., Smith, R. DDRIVE: Distributed Design Review In Virtual Environments. Proceedings of Collaborative Virtual Environments, 2000.

The term virtual reality is commonly used by the popular media to describe imaginary worlds that only exist in computers and our minds.  However, let us more precisely define the term.  According to Websters Dictionary, virtual is defined to be being in essence or effect but not in factReality is defined to be something that constitutes a real or actual thing as distinguished from something that is merely apparent; something that exists independently of ideas conceiving it.  Thus, virtual reality is a term that contradicts itself–an oxymoron!  Fortunately Websters has more recently defined the full term virtual reality to be an artificial environment which is experienced through sensory stimuli (as sights and sounds) provided by a computer and in which one’s actions partially determine what happens in the environment.  NextGen further defines a virtual reality to be a computer-generated environment that can be interacted with as if that environment was real.  A good virtual reality system will allow users to physically walk around objects and touch those objects as if they were real.

Ivan Sutherland, the creator of one of the world’s first virtual reality systems in the 1960s stated “The ultimate display would, of course, be a room within which the computer can control the existence of matter. A chair displayed in such a room would be good enough to sit in. Handcuffs displayed in such a room would be confining, and a bullet displayed in such a room would be fatal.”

We haven’t yet came any where near Ivan Sutherland’s vision (nor do we necessarily want to!) and perhaps we never will.  However, there are some engaging virtual realities today.

What is augmented reality?

Instead of replacing reality, augmented reality adds cues onto the already existing real world.  Computer graphics are embedded into the real world and ideally the human mind would not be able to tell the difference between computer-generated images and the real world.  This can take various forms, some of which are described below in Implementation.

The continuum from virtual reality to augmented reality

The differences between virtual and augmented reality is not often well defined.  True virtual reality completely blocks out the real world whereas augmented reality adds to the already existing real world.  Sometimes these forms that are somewhere between virtual and augmented reality are defined by other terms.  For example, mixed reality is a mix of a digitized model of the real world combined with computer-generated models.

Implementation

Today’s virtual and augmented reality systems are typically implemented in one of three ways:  head-mounted displays,  world-fixed displays, and hand-held displays.

Head-mounted displays

Head-Mounted Displays

The Oculus Rift, Google Glasses, the Joint-Force Fighter Helmet, and a custom built/modified HMD (custom HMD from Jerald, J., Fuller, A., Lastra, A., Whitton, M., Kohli, L., and Brooks, F. 2007. Latency Compensation by Horizontal Scanline Selection for Head-Mounted Displays. Proceedings of SPIE Stereoscopic Displays and Virtual Reality Systems, Volume 6490, pp. 64901Q)

Position and orientation tracking of the head is essential for head-mounted displays because the display/headphones move with the head.  For a virtual object to appear stable in space, the display must be appropriately updated as a function of the current pose of the head–for example as the user rotates his head to the left, the computer-generated image on the display should move to the right so that the image of the virtual objects appear stable in space, just as they would appear for real world objects.  Well implemented head-mounted displays typically provide the greatest amount of immersion.  However doing this well consists of many challenges such as accurate tracking, low latency, and careful calibration.

Head-mounted displays (HMDs) can be further broken down into three types:  non-see-through HMDs, video-see-through HMDs, and optical-see-through HMDs.  Non-see-through HMDs block out all cues from the real world and provide the most immersion for virtual reality.  Optical see-through HMDs enable computer generated cues to be overlaid onto the visual field and provides the ideal augmented reality experience.  Conveying the ideal augmented reality experience using  optical-see-through displays is extremely challenging due to various shortcomings (extremely low latency, extremely accurate tracking, optics, etc.).  Because of these challenges, video-see-through displays are sometimes used.  Video-see-through HMDs are typically considered to be augmented reality, although the advantages and disadvantages are somewhere between augmented reality and virtual reality.

World-fixed displays

CAVE and CABANA

Conceptual drawing of a CAVE and picture of a CAVE-like display. Users are surrounded with stereoscopic perspective-correct images displayed on the floor and walls that they interact with. The user feels as if immersed in a virtual world. The system on the right has moveable walls so that the display can be configured into different display shapes such as a wall or L-shape. From Cruz-Neira, C., Sandin, D.J., DeFanti, T.A., Kenyon, R., and Hart, J.C. The CAVE, Audio Visual Experience Automatic Virtual Environment. Communications of the ACM, June 1992, pp. 64-72 and Daily, M, Sarfaty, R, Jerald, J, McInnes, D., Tinker, P. “The CABANA”: A Re-configurable Spatially Immersive Display. Proceedings of the Third International Immersive Projection Technology Workshop, 1999.

World-fixed displays render graphics/audio via surfaces/speakers that do not move with the head. Displays take many forms, ranging from a standard monitor (also known as fish-tank VR) to displays that completely surround the user (e.g., CAVEs).  Display surfaces are typically flat surfaces, although more complex shapes can be used if those shapes are well defined or known.  Head tracking is important for world-fixed displays, but accuracy and latency requirements are typically not as is critical as they are for head-mounted displays.  High-end world-fixed displays with multiple surfaces and projectors can be extremely immersive but expensive.

World-fixed displays typically fall between augmented reality and virtual reality.  Often the intent is for the only real-world cue to be visible is the user himself.

Hand-held displays

Hand-held augmented reality

Zoo-AR from GeoMedia and the iKat app from Zenitum are examples of hand-held augmented reality.

Hand-held displays are tracked devices that can be held with the hand(s) and do not require precise alignment with the eyes (in fact the head is rarely tracked for hand-held displays).  Hand-held augmented reality, also called indirect augmented reality, has recently become popular due to the ease of access to smartphones and tablets.  In addition, system requirements are much less since viewing is indirect–rendering is independent of the user’s head/eyes.