SmartAR on HMD

This is a re-post of my project for EN.600.661 Computer Vision at Johns Hopkins University. Team members include me and Felix Jonathan.


The video is taken by a phone camera placed behind the smart glasses.

What is it?

SmartAR is an augmented reality application on smart glass Moverio BT-200. It is able to recognize the context in front of the glass via Bag-of-Words algorithm, track fiducial marker inside the scene, and display augmentation based on its understanding.



System architecture of SmartAR looks like this:

SmartAR architecture

ROS manages the messages within the network between PC and Moverio BT-200.

Object Recognition

The object recognition algorithm here is very basic bag-of-words algorithm. A database of object images is needed on PC.

SmartAR object recognition

Thanks to the simplicity of Bag-of-Words algorithm, the recognition of known objects in this application is real-time.

Marker Tracking

Aruco is used to track markers in the scene. In order to correctly overlay the graphics on top of the marker seen by the user, an optical see-through display calibration is needed. Here, the traditional Single Point Active Alignment Method is applied. Rendering is performed using OpenGL ES in Android OS. The result looks like this:

SmartAR marker tracking

Both geometric and context augmentation are provided at the same time. The performance is real-time as well.