Future Augmented Reality Surgery

In the paper of 2000, Optical Versus Video See-Through Head-Mounted Displays in Medical Visualization, the author illustrated the future of Augmented Reality technology in surgery like this:

Future Augmented Reality Surgery

16 years have passed, we are still far from that. Keep walking.

Thanks!


SmartAR on HMD

This is a re-post of my project for EN.600.661 Computer Vision at Johns Hopkins University. Team members include me and Felix Jonathan.

Demo

The video is taken by a phone camera placed behind the smart glasses.

What is it?

SmartAR is an augmented reality application on smart glass Moverio BT-200. It is able to recognize the context in front of the glass via Bag-of-Words algorithm, track fiducial marker inside the scene, and display augmentation based on its understanding.

Hardware

  • A PC with Internet
  • Moverio BT-200 head-mounted display

Software

System architecture of SmartAR looks like this:

SmartAR architecture

ROS manages the messages within the network between PC and Moverio BT-200.

Object Recognition

The object recognition algorithm here is very basic bag-of-words algorithm. A database of object images is needed on PC.

SmartAR object recognition

Thanks to the simplicity of Bag-of-Words algorithm, the recognition of known objects in this application is real-time.

Marker Tracking

Aruco is used to track markers in the scene. In order to correctly overlay the graphics on top of the marker seen by the user, an optical see-through display calibration is needed. Here, the traditional Single Point Active Alignment Method is applied. Rendering is performed using OpenGL ES in Android OS. The result looks like this:

SmartAR marker tracking

Both geometric and context augmentation are provided at the same time. The performance is real-time as well.

Thanks!