ARssist - Augmented Reality in Robotic Surgery

Recently I have been working on an augmented reality project: ARssist. It stands for “Augmented Reality Assistant” for robotic surgery (with da Vinci robot).

Bedside Assistant

In a da Vinci robotic surgery, a bedside-assistant (also called First Assistant) stands at the bedside, helpes the surgeon to complete the operation. The surgeon is seated in the surgeon console manipulating the instruments.

The bedside assistant needs to perform various tasks, for example:

In ARssist, we provide improved augmented reality visualization to the bedside assistant, to help him or her to accomplish the tasks better. We use HoloLens currently. One significant advantage of using HoloLens-like device for surgery task is that, it is fail-safe. Even if the system fails, the user can still finish the task in the same way he/she usually does without augmented reality.

Augmented Reality Visualization

ARssist provides the following visulizations:

Example See-Through View

The above video is captured using a camera behind the HoloLens. The robotic instruments and endoscope are correctly overlaid with the help of fiducial-tracking, and robot kinematics. The endoscopic video (bottom right corner) is streamed to the HoloLens, and projected in the field-of-view of the virtual endoscope. The system is perfectly real-time.

In this video, a robotic instrument is controlled by the bedside assistant, and its overlay is maintained well.


We have conducted experiments with novice users and experienced users, including surgeons and residents. Two specific tasks were chosen: instrument insertion and laparoscopic tool manipulation. We will publish more results of evaluation in the near future.

For more details, please refer to our papers:

  title={ARssist: augmented reality on a head-mounted display for the first assistant in robotic surgery},
  author={Qian, Long and Deguet, Anton and Kazanzides, Peter},
  journal={Healthcare technology letters},

# Another ICRA 2019 paper

Thank you for reading! LQ