Slamcore Visualiser

The Slamcore Visualiser is a tool that provides a 3D interface for evaluating the behaviour of the visual-inertial tracking system in real time. It can be used to view:

• A 3D plot of the trajectory.

• A 3D plot of the features in the map.

• Live state info of the camera position and orientation, tracking status and number of features in the map.

• A camera feed with feature detection overlaid (picture-in-picture).

Usage - SLAM

Run SLAM on Live Camera Feed

Plug in the camera to a USB 3 port on your computer.

From a terminal window, run:

$ slamcore_visualiser

Note

The software automatically detects the type of camera that is plugged in. In case multiple cameras of the same type are plugged in it will arbitrarily select one.

This behaviour is consistent across all the SLAM client applications.

See the troubleshooting page to help diagnose common errors.

Run SLAM on a Recorded Dataset

The Slamcore Visualiser supports processing from datasets. To start the Visualiser to process a dataset run the following command:

$ slamcore_visualiser dataset -u <path/to/dataset/>

SLAM, Localisation, Mapping and Odometry Mode

There are four main ways to run our system:

Mode	Description
SLAM	A sparse map is built on the go, without any prior knowledge about the environment.
Localisation	Use a sparse map previously created in SLAM mode.
Height Mapping	Build a height and occupancy map for autonomous navigation.
Panoptic Segmentation	Segment and detect object instances. See Panoptic Segmentation API for requirements and installation.
Odometry	Positioning without using a map. See Positioning Mode to use this mode.

SLAM mode

By default, the system is configured to run in SLAM mode. To run in live SLAM mode just press the START button, and start moving the camera. An example of a SLAM run is shown in the image below.

Fig. 3 Slamcore Visualiser - Running SLAM

To generate a session map, click on the GENERATE button, select a location to save the session file, and press OK. Pressing GENERATE will automatically stop the SLAM system. A progress bar will be displayed informing you about the status of session generation and saving.

Warning

Depending on the size of the map as well as the length of the trajectory, generating a session can take several minutes.

Fig. 4 Slamcore Visualiser - Generating a session

Once the session has been written to disk, the application will display a pop-up with the path to the saved map, as in the figure below.

Fig. 5 Slamcore Visualiser - Session successfully saved

After saving a map, the SLAM run may be continued by clicking on the START button again.

Localisation mode

Warning

Slamcore SDK v23.01 brings several breaking changes, one of them being the session file format. Session files (.session) generated using older software versions (v21.06) will no longer be usable with v23.01 and above. You may use the new software to generate a new session file if you have retained the original dataset.

Please contact support@slamcore.com, providing your Slamcore account email address, if you require access to the v21.06 SDK packages and its supporting documentation to use in conjunction with your old session files.

In localisation mode, our system localises against a previously created map that was saved in a session file. To run in this mode, load a session file using the LOAD button and press START. The visualisation in the localisation mode differs from the one in SLAM mode. There are three main components as shown in the image below:

• The loaded sparse map is displayed in grey;

• temporary landmarks used for tracking not present in the loaded map are shown in red; and

• landmarks that are currently recognised from the map are visualised in cyan.

Fig. 6 Slamcore Visualiser - Relocalisation using preloaded map

Note

Pressing RESET will clear the map and configure the system to run in SLAM mode.

Height Mapping Mode

In height mapping mode, our system runs in SLAM mode but also generates a height map and an occupancy map which can be used in autonomous navigation.

Note

See the 2D Occupancy Mapping tutorial for a detailed walkthrough of the mapping feature.

To run in this mode, launch the Visualiser with the -m or --generate-map2d flag, with depth stream enabled:

$ slamcore_visualiser -m

Press the START button and move the camera around to begin mapping.

Hit the R key on the keyboard or select View and Show Right Sidebar to reveal visualisation options. Under the “2D Map” field is displayed, switch between “Height Map” and “Occupancy Map” to view the 2D/3D height map or the occupancy map.

Fig. 7 Slamcore Visualiser - Height map built in mapping mode

Fig. 8 Slamcore Visualiser - Occupancy map built in mapping mode

To generate and save a session map, click on the GENERATE button, select a location to save the session file, and press OK.

Note

The 2D map visualised may not be representative of the final map produced as further optimisation is carried out during session generation.

Panoptic Segmentation

Note

A separate Debian package installation of the Slamcore Panoptic Segmentation Plugin is required to run panoptic segmentation. Please check the system requirements and follow the installation steps at the Panoptic Segmentation API page before running this mode.

To launch the panoptic plugin with the Visualiser, connect an Intel RealSense D435i or D455 and run with the following flags:

$ slamcore_visualiser --enable-panoptic-plugin --color --depth

Using a pre-generated optimised network
Warming up device
Device warmed

Note

If you did not run the check_panoptic_segmentation script during installation, Slamcore Visualiser will instead perform the set up of the network file during the first run of this command. This process takes 10 - 15 minutes to complete before you can use the network and run SLAM.

The --enable-panoptic-plugin flag enables the use of the plugin.

The color stream is required and enabled with the --color flag for the network to run inference on.

The depth stream is optional and can be enabled with the --depth flag to project the predictions into 3D space. When enabled, the point cloud of the detected object(s) will be colored in the Visualiser’s 3D view to indicate the segmentation class, as well as a 3D bounding box around each detected object.

If only the color stream is enabled and the depth stream disabled, only the 2D semantic segmentation masks will be computed and shown in the Segmentation PiP RGB stream.

Press the START button and move the camera around to start SLAM and panoptic segmentation.

Fig. 9 Slamcore Visualiser - Panoptic Segmentation Plugin

The Network Inference Time (ms) is provided on the left information panel of the Slamcore Visualiser. This is the time taken by the panoptic plugin to return the inference results from an image.

Note

The network inference time is affected by other processes running on the GPU, such as visualization with the Slamcore Visualiser, ROS RViz, etc. If the inference time takes longer than the SLAM time, you may see this warning:

[Warning 2023-07-18T18:06:44+0100] Inference took longer than the expected
frame period (66 ms), SLAM results may be affected

If this occurs frequently, set your platform to the highest power mode, and set the GPU and CPU clocks to the maximum by running:

sudo jetson_clocks

A Semantics drop-down is available on the right sidebar to configure the semantics visualization settings. You may toggle the visualization of the detected objects’ instance point clouds and 3D bounding boxes in the 3D view.

In the main 3D view, when the depth stream has been enabled, a pink “instance point cloud” will be shown containing only points with detected object instances.

In the PiP view, the RGB streams Segmentation(Visible.0) and Segmentation blend(Visible.0) contain overlays of the person segmentation mask in pink, background in light gray and unknown pixels in dark gray.

Usage - Dataset Recording while running SLAM

The Slamcore Visualiser also supports recording a dataset while simultaneously running SLAM on a live camera. To do so, specify the directory to write the dataset to with the --dataset-write-dir flag:

$ slamcore_visualiser --dataset-write-dir </output/directory>

Note

Recording a dataset while simultaneously running SLAM on a live camera is compatible with any of the Slamcore software modes: SLAM, Localisation, Mapping and Odometry Mode.

Saving Trajectories

For each run, the SLAM trajectory may be exported as two CSV files for future analysis: optimised and optimised trajectory.

To do so, click on the EXPORT button at any point of the run, select a location to save the trajectory files and press OK. See Guide to Trajectories for more information about the trajectories.

The Application

Fig. 10 Slamcore Visualiser - User Interface

On the left is a panel which shows the instantaneous state of the system, such as:

• Current position away from origin (the position at the start of the run),

• Current orientation (in Euler angles) with respect to that at the origin,

• Distance travelled,

• Tracking status,

• Time elapsed since the start of the run,

• Current frame rate,

• Number of dropped frames, landmarks and features,

• Map information.

Below this are the controls to START, STOP and RESET the tracking. Under the “Session” section are the buttons to GENERATE and LOAD a session. Under the “Trajectory” section is the button to EXPORT the optimised and unoptimised trajectories.

To the right is the 3D view. Initially this only displays a grid to represent the ground plane and an origin gizmo. When the tracking is running the following are also visible:

• Pose displayed as three-dimensional axes,

• Cameras displayed as pyramids,

• Trajectory displayed as a yellow line,

• Sparse map landmarks displayed as points in the 3D space,

• Local point cloud also displayed as points in the 3D space,

• 2D map, displaying the height map or occupancy map.

The visibility of these items can be toggled using the options under the button and the Right Sidebar (hit the R key on the keyboard).

The visualisation perspective of the 3D view can be changed under the button to Top, Front, Right or First Person View. Chase mode can also be enabled to follow the camera such that the camera frame is always at the centre of the window.

Fig. 11 Slamcore Visualiser - 3D View Features

3D Camera Controls

To control the position and orientation of the 3D view in the application, use the mouse or trackpad.

The view can be rotated by holding down the left mouse button while hovering over the 3D window. Moving the mouse up and down will then tilt the view while a left/right movement will rotate the view around a point central to the window.

The view can be panned around in 3D space by holding down the right mouse button.

The view can be moved directly forwards and backwards by using the mouse wheel or the scroll region of the trackpad.

Menus

File:

• Quit: exits the application.

View:

• Show Left Sidebar: display the tracking controls, state information and map information on the left-hand side of the window.

• Show Right Sidebar: display the controls for the visualisation options in the right-hand side of the window.

Fig. 12 Slamcore Visualiser - Menus

Help:

• Controls: description of the keyboard and mouse controls for the application.

• About: version and licensing information.

Command-Line Options

At start-up, the Slamcore Visualiser accepts the following command-line options:

-h,--help

Print this help message and exit

--help-all

Expand all help

-v,--version

Display the version

-c,--config-file :FILE ...

Path to the configuration file(s)

-k,--write-kinematics,--no-k{false},--no-kinematics{false}

Enable/disable kinematics sensor stream

--enable-panoptic-plugin

Enable Panoptic Segmentation plugin

-l,--load-session :FILE

Path to the session file to load

--dataset-write-dir

Directory to write dataset to

-m,--generate-map2d

Produce a new 2D map

Dataset Subcommand Options

The dataset subcommand allows dataset processing via the Slamcore Visualiser tool.

-u,--euroc-dataset :DIR

Dataset reader path

--ts,--timescale FLOAT

Timescale for dataset reader, for real time processing set to 1.0

$ slamcore_visualiser dataset -u <path/to/dataset/>

RealSense Subcommand Options

The following options only apply to RealSense cameras.

realsense

Detect RealSense camera

--dev,--device

Integer device index for multiple reader types

--color,--no-color

Enable or disable the colour stream

--depth,--no-depth

Enable or disable the depth stream (See FAQs for detailed explanation)

--ir,--no-ir

Enable or disable the infrared stream

--fps

Override reader’s FPS (See FAQs for options)

OAK-D S2 Fixed-Focus Subcommand Options

The following options only apply to OAK-D S2 Fixed-Focus cameras.

oakd

Detect OAK-D S2 Fixed-Focus camera

--dev,--device

Integer device index for multiple reader types

--fps

Override reader’s FPS (See FAQs for options)

--rectify,--no-rectify{false}

Whether the reader should provide rectified stereo images