Google Cardboard as "Augmented Reality" Headset

Google Virtual Reality headset evolved from Cardboard v1, to Cardboard v2, and to the most recent and elegant Daydream. Daydream supports limited devices, thus is excluded in the discussion here. For Cardboard v1 and v2, there have been a few discussion about the difference between them. As a VR / AR developer working with both headset, I looked at something different

Cardboard v1

Cardboard v2

Back view

Front view

Back and front view of Cardboard v1 and v2

Comments

What about Augmented Reality

Google Cardboard is basically a VR headset, here I am going to turn it into a “AR” headset!

Requirement

For an augmented reality application to run with Google Cardboard, there are a few requirement:

Pipeline

The combination of Cardboard v1 and Nexus 5 is chosen. Unity3D is the development platform. The rendering pipeline is:

  1. Grab a video frame
  2. Optional: do whatever augmented reality processing
  3. Select and crop the part of vision for two eyes
  4. Apply barrel distortion to both views (barrel distortion)
  5. Composite two views side by side and blit on screen

Video frame is accessed by WebcamTexture object of Unity3D engine.

Shader

Within this application, we are not displaying real 3D scene, what we have is only augmented 2D image. We select the part of image to display with disparity, faking an AR experience for the user, all the objects in the scene will be perceived at a fixed distance determined by the amount of disparity. We need fast pixel displacement!

Step 3, 4 and 5 are achieved by shader, especially fragment shader. Shader is very fast and powerful. Unity3D shaders can be categorized into vertex shaders, fragment shaders and surface shaders. Vertex shaders and fragment shaders are basic types of shaders that are inserted into graphics pipeline of most modern GPU. On mobile devices, OpenGL ES 2.0 supports both of them. Surface shader is a special type of shader dealing with lighting and textures. In compilation stage, surface shaders are separated into a vertex shader part and a fragment shader part.

In our application, we have video frame in a texture object, thus, we only need to displace the texture coordinates of screen plane to represent barrel distortion and disparity of two views.

Step 3, 4 and 5 can be implemented into three shaders, which is easy to maintain and modify but will require more render passes. I put them altogether into one shader, since they are all fragment shaders. Source code is on Github: cardboard_seethrough.

In order to render the texture directly on screen, Graphics.Blit function is called upon rendering, and the source texture object is replaced with our processed video, the target render object is set to null. According to Unity3D documentation, a null target render object represents the screen. Then the shader material is specified in order to achieve correct Cardboard style display.

Demo

Nexus 5 runs this application smoothly, with 20-30 fps.

With disparity

Without disparity

Screenshot with and without disparity

Several GUI elements are put on the GUI layer:

Source code is on Github: cardboard_seethrough.

Google VR SDK is not used here.

Acknowledgement

Thanks for reading!