Coorindate Systems in HoloLensARToolKit v0.2

This post is part of documentation of HoloLensARToolKit, version v0.2. The documentation for the coordinate system in HoloLensARToolKit v0.1 is here.

Coordinate System of Unity3D

Unity3D uses left-hand coordinate systems to define transformations.

The above is a screenshot of a Unity transformation. The x, y, z axis directions are visualized with different colors, and are following left-land rules. In addition, rotation along axis is applied clockwise, unlike counter-clockwise in right-hand coordinate systems. If you are accustomed to right-hand coordinate systems as I do, please pay more attention when working with raw 4x4 matrices.

Coordinate System of ARToolKit

ARToolKit uses right-hand coordinate systems. Since OpenGL is right-handed as well, it is very comfortable to visualize the tracking result of ARToolKit by OpenGL.

In the official Unity package of ARToolKit, arunity, there exists a utility function

public static Matrix4x4 LHMatrixFromRHMatrix(Matrix4x4 rhm)

that converts the tracking result of ARToolKit to Unity environment.

Coordinate System of HoloLensARToolKit

It took me a while to figure out the complicated conversions happened inside ARToolKit, inside Unity, and in-between. Actually in HoloLensARToolKit v0.1, the coordinate system is not very clear nor consistent between different kinds of markers. In the current version v0.2, the design of coordinate system is improved, and a special Unity scene is designed to help visualize the coordinate system associated with the marker.

In this project, the tracking result is converted into left-handed coordinate system, with y-axis flipped. This is performed in the function ARUWPUtils.ConvertARUWPFloatArrayToMatrix4x4(). When the tracking result is applied to a Visualization Target of the current marker, there is no need to worry about any conversion.

The following examples show the coordinate system associated with each kind of marker.

Single Pattern Marker

Hiro marker and Kanji marker are examples of single pattern marker. They are defined using binary files.

The coordinate system associated with a single pattern marker looks like this:

Single Matrix Marker

Single matrix marker is pre-defined in ARToolKit. User is only required to provide the ID and type of matrix code associated when using them in applications. A full list of matrix markers are available in ARToolKit repository. The coordinate system associated with a matrix marker looks like this:

Multi Matrix Marker

Multi matrix marker is defined by a configuration file, that contains

  • total number of matrix markers
  • the ID of each matrix marker
  • the transformation from the multi-matrix marker to individual matrix marker.

Multi-matrix marker is useful for robust tracking, thus is much more useful for accuracy-critical augmented reality applications. Iterative Closest Point (ICP) algorithm is the underlying method for the fusion of tracking information.

For example, for a multi-barcode-4x3 configuration, the coordinate system of the multi-marker looks like this:

Marker Configuration File

In HoloLensARToolKit v0.1, users need to adjust the multi-marker configuration file provided by ARToolKit, like this, to make it work with HoloLensARToolKit. Starting from v0.2, this step is not needed. The multi marker configuration file of ARToolKit and HoloLensARToolKit is EXACTLY SAME, which means, the description of multi marker is actually using right-hand coordinate system.

Finally

You can access more articles describing the implementation details of HoloLensARToolKit in my blog, simply clicking on the tag: hololens-artoolkit.

Thanks for reading!


ARUWPController Options in HoloLensARToolKit v0.2

This post is part of documentation of HoloLensARToolKit, version v0.2. The ARUWPController options documentation for v0.1 is here.

ARUWPController

ARUWPController.cs is one of the main scripts used in HoloLensARToolKit. In this post, the options of this script are listed and discussed, along with common usecases.

Each Unity project using HoloLensARToolKit package must and must only have one ARUWPController component.

ARUWPController is very similar to ARController in ARToolKit, one of the major difference is that ARUWPController targets only at Universal Windows Platform, while ARController also handles Android, iOS, standalone and even editor. Therefore, ARUWPController has fewer attributes than ARController in general.

When ARUWPController script is attached to some Unity GameObject, its inspector window looks like this:

Use Camera Param

  • The checkbox to indicate how to initialize the camera parameters
  • If this box is check, then the field Camera Param Filename will appear. If not, then users must manually initialize the camera parameter via setting a byte buffer, by calling ARUWPController.SetCameraParamBuffer()

Camera Param Filename

  • This field specifies the name of camera calibration file, contained in the path Assets/StreamingAssets/.
  • The camera calibration file must be ARToolKit format. It is a binary file, instead of XML or YAML for OpenCV. Please refer to HoloLens Camera Calibration project for more details.

Pattern Detection Mode

This field configures what kind of marker does the detection algorithm look for.

  • If pattern markers only, e.g. Hiro or Kanji, then AR_TEMPLATE_MATCHING_COLOR or AR_TEMPLATE_MATCHING_MONO is enough.
  • If matrix marker only, e.g. 3x3 code marker, then AR_MATRIX_CODE_DETECTION is sufficient.
  • If there is need to detect both kinds of marker, then AR_TEMPLATE_MATCHING_COLOR_AND_MATRIX pr AR_TEMPLATE_MATCHING_MONO_AND_MATRIX must be chosen.

Matrix Code Type

This field will appear if the Pattern Detection Mode involves the detection of matrix marker. There are many types of matrix marker, some of them are supported by ARToolKit and this project. The most common marker set is AR_MATRIX_CODE_3x3. All available options are:

  • AR_MATRIX_CODE_3x3
  • AR_MATRIX_CODE_3x3_PARITY65
  • AR_MATRIX_CODE_3x3_HAMMING63
  • AR_MATRIX_CODE_4x4
  • AR_MATRIX_CODE_4x4_BCH_13_9_3
  • AR_MATRIX_CODE_4x4_BCH_13_5_5

Track FPS Holder (optional)

This field is looking for a Unity.UI.Text object to print out tracking frame rate. It is very useful for debugging, or inspecting the performance of the application. In the sample scenes provided by HoloLensARToolKit, this text field is at the top-right corner for the user. Because it is not a required component for tracking to run, this field can be left blank.

Render FPS Holder (optional)

Similar to Track FPS Holder (optional), this field is optional, and is able to visualize the frame rate of rendering of the application. The rendering here means the refreshing of the whole application, but not the refreshing of video.

Advanced Options

The above options are essential, and should be taken care for each application. If Advanced Options is checked, more options will be listed for users to configure the performance of ARUWPController. The following screenshot shows the full list of options:

Border Size

The percentage of border of the marker, by default, it is 0.25. For example, in a 80cm width marker, the border is 20cm (25%).

Labeling Mode

It configures the color of the border of the marker. AR_LABELING_BLACK_REGION is the default.

Image Proc Mode

The mode of image processing, by default, AR_IMAGE_PROC_FRAME_IMAGE is chosen.

According to ARToolKit documentation,

When the mode is AR_IMAGE_PROC_FIELD_IMAGE, ARToolKit processes pixels in only every second pixel row and column. This is useful both for handling images from interlaced video sources (where alternate lines are assembled from alternate fields and thus have one field time-difference, resulting in a “comb” effect) such as Digital Video cameras.

Thresholding Mode

You can choose different thresholding algorithm from the droplist, same as ARToolKit. Available options are:

  • AR_LABELING_THRESH_MODE_MANUAL
  • AR_LABELING_THRESH_MODE_AUTO_MEDIAN
  • AR_LABELING_THRESH_MODE_AUTO_OTSU
  • AR_LABELING_THRESH_MODE_AUTO_ADAPTIVE
  • AR_LABELING_THRESH_MODE_AUTO_BRACKETING

Threshold

This field appears when the Thresholding Mode is set to AR_LABELING_THRESH_MODE_MANUAL. ARToolKit first thresholds the grayscale image before corner extraction. It is easy to understand that the value applied here will be used as the threshold to obtain black and white image.

Finally

You can access more articles describing the implementation details of HoloLensARToolKit in my blog, simply clicking on the tag: hololens-artoolkit.

Thanks for reading!