5.9 KiB

Raw Blame History

Using a RealSense Camera

After making sure that everything works by running the EuRoC datasets, it should be possible to use the realsense driver from Monado to get any RealSense camera that has an IMU and one or more cameras to get tracked with SLAM. However, this was only tested on a D455, so if you are having problems with another device, please open an issue. Also, open an issue if you manage to make it work with other devices so that I can add it to this README.

Index

Using a RealSense Camera

Overview of the Setup (D455)

Let's first assume you have a RealSense D455, which is the one that works with the defaults. Even if you have another RealSense device follow this section, you might at least get something working, although not at its best.

SLAM-Tracked RealSense Driver

Set these environment variables:

export RS_HDEV_LOG=debug: Make our realsense device logs more verbose
export RS_SOURCE_INDEX=0: Indicate that we want to use the first RealSense device connected as data source
export RS_TRACKING=2: Only try to use "host-slam". See other options here.
export SLAM_CONFIG=$bsltdeps/basalt/data/monado/d455.toml: Configuration file for Basalt and the D455.

RealSense-Tracked Qwerty Driver

You now have a RealSense device that you can use to track another device, for example, let's track a keyboard-and-mouse controlled HMD provided by the qwerty driver.

Set these environment variables to enable the qwerty driver and stream to the SLAM system on start:

export QWERTY_ENABLE=true QWERTY_COMBINE=true SLAM_SUBMIT_FROM_START=true

And then modify your tracking overrides in your monado configuration file (~/.config/monado/config_v0.json) by updating the json object with:

{
  "tracking": {
    "tracking_overrides": [
      {
        "target_device_serial": "Qwerty HMD", // Or "Qwerty Left Controller"
        "tracker_device_serial": "Intel RealSense Host-SLAM",
        "type": "direct",
        "offset": {
          "orientation": { "x": 0, "y": 0, "z": 0, "w": 1 },
          "position": { "x": 0, "y": 0, "z": 0 }
        },
        "xrt_input_name": "XRT_INPUT_GENERIC_TRACKER_POSE"
      }
    ],
  }
}

And that's it! You can now start an OpenXR application with Monado and get your view tracked with your D455 camera.

Non-D455 RealSense Devices

While I was unable to test other devices because I don't have access to them, it should be possible to make them work by:

Configuring the RealSense Pipeline

These fields determine your RealSense streaming configuration, and these are their current defaults that work on a D455. You can change those fields by setting any of them in your config_v0.json inside a config_realsense_hdev field. Also note that as we already set RS_HDEV_LOG=debug, you should see the values they are currently taking at the start of Monado.

For example, let's say you have a realsense device which has two fisheye cameras that support streaming 640x360 at 30fps (a T265 I think), then a configuration like this should work:

"config_realsense_hdev": {
  "stereo": true,
  "video_format": 9, // 9 gets casted to RS2_FORMAT_Y8 (see https://git.io/Jzkfw), grayscale
  "video_width": 640, // I am assuming the T265 supports 640x360 streams at 30fps
  "video_height": 360,
  "video_fps": 30,
  "gyro_fps": 0, // 0 indicates any
  "accel_fps": 0,
  "stream_type": 4, // 4 gets casted to RS2_STREAM_FISHEYE (see https://git.io/Jzkvq)
  "stream1_index": -1, // If there were more than one possible stream with these properties select them, -1 is for auto
  "stream2_index": -1,
}

The particular values you could set here are very dependent on your camera. I recommend seeing the values that get output by running the rs-sensor-control example from the RealSense API.

Configuring Basalt

As you might've noticed, we set SLAM_CONFIG to $bsltdeps/basalt/data/monado/d455.toml which is this config file that I added for the D455. This file points to a calibration file and a VIO configuration file.

For the tracking to be as good as possible you should set the intrinsics/extrinsics of the device in a similar calibration file and point to it with the SLAM_CONFIG config file. You can obtain that information from the previously mentioned rs-sensor-control utility. Issues like this (T265) and this (D435) provide configuration files tried by other users. Additionally Basalt provides custom calibration tools that can work for any camera-IMU setup or tools like basalt_rs_t265_record or the euroc_recorder in Monado that can help creating an initial calibration file for RealSense devices.

5.9 KiB Raw Blame History