ARssist - Augmented Reality in Robotic Surgery

Recently I have been working on an augmented reality project: ARssist. It stands for “Augmented Reality Assistant” for robotic surgery (with da Vinci robot).

Bedside Assistant

In a da Vinci robotic surgery, a bedside-assistant (also called First Assistant) stands at the bedside, helpes the surgeon to complete the operation. The surgeon is seated in the surgeon console manipulating the instruments.

The bedside assistant needs to perform various tasks, for example:

  • Insufflation and trocar placement
  • Dock and undock the robot
  • Instrument exchange
  • Manipulating laparoscopic instruments, e.g. stapler, grasper

In ARssist, we provide improved augmented reality visualization to the bedside assistant, to help him or her to accomplish the tasks better. We use HoloLens currently. One significant advantage of using HoloLens-like device for surgery task is that, it is fail-safe. Even if the system fails, the user can still finish the task in the same way he/she usually does without augmented reality.

Augmented Reality Visualization

ARssist provides the following visulizations:

  • Virtual robotic instruments
  • Virtual endoscope and its field-of-view
  • Hand-held instrument if it is tracked
  • Endoscopic video

Example See-Through View

The above video is captured using a camera behind the HoloLens. The robotic instruments and endoscope are correctly overlaid with the help of fiducial-tracking, and robot kinematics. The endoscopic video (bottom right corner) is streamed to the HoloLens, and projected in the field-of-view of the virtual endoscope. The system is perfectly real-time.

In this video, a robotic instrument is controlled by the bedside assistant, and its overlay is maintained well.


We have conducted experiments with novice users and experienced users, including surgeons and residents. Two specific tasks were chosen: instrument insertion and laparoscopic tool manipulation. We will publish more results of evaluation in the near future.

For more details, please refer to our papers:

  title={ARssist: augmented reality on a head-mounted display for the first assistant in robotic surgery},
  author={Qian, Long and Deguet, Anton and Kazanzides, Peter},
  journal={Healthcare technology letters},

# Another ICRA 2019 paper

Thank you for reading! LQ

A Metamask Donation Button

Ethereum is cool!

Tip me with ether is better than buy me a coffee nowadays.

The community around Ethereum is awesome. The MetaMask is a great browser plugin for the Ethereum blockchain, serving many Ðapps, like some collectable, breedable and adorable creatures.

MetaMask can serve the tipping purpose easily. This post is largely adapted from their own demo: TipButton. This post demos how to embed such donation link in a Jekyll Github page, very straightforward.

The Tip Button

Here we go:

It is SAFE to click it. An additional window will prompt up BEFORE you actually send the 0.005 ether to me, which is equivalent to a coffee by today 2018-04-16.

If you don’t have MetaMask installed, please consider visiting MetaMask.

What is it doing?

  1. Metamask provides the web3.js object for your browser that communicates with the blockchain in the air, manages your accounts, and most importantly protects (hopefully) your private keys.
  2. When you initiate a transaction on the Ethereum blockchain, it prompts a window to ask for your permission to sign the transaction.
  3. The tipping transaction is constructed in the embedded javascript code.
  4. If you hit confirm it, I will get the 0.005 ether after this transaction is mined into the chain.
  5. Thanks!

The Embedded JavaScript

The script is very simple and straightforward:

var tipButton = document.querySelector('.tip-button');
tipButton.addEventListener('click', function() {
  if (typeof web3 === 'undefined') {
    return alert('You need to install MetaMask to use this feature.')
  var user_address = web3.eth.accounts[0];
  if (typeof user_address === 'undefined') {
    return alert('You need to log in MetaMask to use this feature.')
    to: "0x66454C561Cf137F53321945758b0E4645E9EEae8",
    from: user_address,
    value: web3.toWei('0.005', 'ether'),
  }, function (err, transactionHash) {
    if (err) return alert('Thanks for trying out!');
    alert('Thanks for the generosity!!');

Enjoy the blockchain! LQ