ARssist - Augmented Reality in Robotic Surgery
09 Apr 2019Recently I have been working on an augmented reality project: ARssist. It stands for “Augmented Reality Assistant” for robotic surgery (with da Vinci robot).
Bedside Assistant
In a da Vinci robotic surgery, a bedside-assistant (also called First Assistant) stands at the bedside, helpes the surgeon to complete the operation. The surgeon is seated in the surgeon console manipulating the instruments.
The bedside assistant needs to perform various tasks, for example:
- Insufflation and trocar placement
- Dock and undock the robot
- Instrument exchange
- Manipulating laparoscopic instruments, e.g. stapler, grasper
In ARssist, we provide improved augmented reality visualization to the bedside assistant, to help him or her to accomplish the tasks better. We use HoloLens currently. One significant advantage of using HoloLens-like device for surgery task is that, it is fail-safe. Even if the system fails, the user can still finish the task in the same way he/she usually does without augmented reality.
Augmented Reality Visualization
ARssist provides the following visulizations:
- Virtual robotic instruments
- Virtual endoscope and its field-of-view
- Hand-held instrument if it is tracked
- Endoscopic video
Example See-Through View
The above video is captured using a camera behind the HoloLens. The robotic instruments and endoscope are correctly overlaid with the help of fiducial-tracking, and robot kinematics. The endoscopic video (bottom right corner) is streamed to the HoloLens, and projected in the field-of-view of the virtual endoscope. The system is perfectly real-time.
In this video, a robotic instrument is controlled by the bedside assistant, and its overlay is maintained well.
Evaluation
We have conducted experiments with novice users and experienced users, including surgeons and residents. Two specific tasks were chosen: instrument insertion and laparoscopic tool manipulation. We will publish more results of evaluation in the near future.
Link to Publications
For more details, please refer to our papers:
@article{qian2018arssist,
title={ARssist: augmented reality on a head-mounted display for the first assistant in robotic surgery},
author={Qian, Long and Deguet, Anton and Kazanzides, Peter},
journal={Healthcare technology letters},
volume={5},
number={5},
pages={194--200},
year={2018},
publisher={IET}
}
# Another ICRA 2019 paper
}
Thank you for reading! 