Panoptic Segmentation API

The panoptic segmentation API allows for panoptic segmentation to be run as part of the position SLAM system, providing semantic segmentation, 3D bounding boxes and the position of object instances relative to the camera that a user can use in their applications.

_images/panoptic_visualiser.png

Fig. 19 Slamcore Visualiser - Panoptic Segmentation Plugin

As part of the API, we provide a person detector via a pre-built dynamic link library (DLL) that can segment and detect people in the environment from the RGB images. This is available as a separate Debian package from the Slamcore Portal to be installed alongside our APIs or Tools.

Current we only support the pre-built DLL that we’ve provided for person detection. In the future, users may register custom inference callbacks or plugin DLLs to use with the panoptic segmentation API, as long as it complies with the same interface as our internal plugin.

There is currently no instance tracking in the API. The 3D pose and bounding box for each detected object is reported relative to the Body frame.

Requirements

Supported Platforms

The person detection API is supported on the following platforms with JetPack 5.1 or 5.1.1 installed:

  • NVIDIA Jetson Xavier NX

  • NVIDIA Jetson Xavier AGX

  • NVIDIA Jetson Orin Nano

  • NVIDIA Jetson Orin NX

  • NVIDIA Jetson Orin AGX

Supported Sensors

The API can be used with the Intel RealSense D435i or D455 with the color stream enabled and optionally the depth stream for 3D bounding box visualization and position estimation.

Note

As the RGB camera in the D455 is global shutter, the segmentation performance is expected to be better on the D455 than on the D435i which has a rolling shutter RGB.

Supported Packages

The panoptic plugin can be used with any of the following Slamcore tools or APIs:

  • Slamcore Tools (Slamcore Visualiser)

  • Slamcore C++ API

  • Slamcore Python API

  • Slamcore ROS2 Foxy and Galactic

Refer to Requirements for more details about system requirements.

Installation

From the Slamcore Portal, download the Slamcore Panoptic Segmentation Plugin, and any of the Slamcore SDK packages that you want to run the panoptic plugin on.

  1. Install the Slamcore SDK package, for example:

    $ sudo apt install </path/to/slamcore-tools>.deb

  2. Install the panoptic plugin:

    $ sudo apt install </path/to/slamcore-panoptic-segmentation>.deb

During this installation step, you may be informed about some CUDA dependency packages that will be installed.

  1. To set up the segmentation model on your machine, run the check_panoptic_segmentation script which was installed by the plugin.

    Note

    This process takes 10-15 minutes to complete.

    $ check_panoptic_segmentation

Generating an optimised network for your platform. This could take several minutes ...
Successfully created a TensorRT engine at: ~/.slamcore/slamcore_panoptic_segmentation_plugin_network.engine
Warming up device
Device warmed

    During this step, the embedded network is converted and optimised for the GPU platform of your machine.

Usage

To use panoptic segmentation with the Slamcore Visualiser, see Panoptic Segmentation.

To use panoptic segmentation with the C++ API, refer to the detect_objects c++ example.

To use panoptic segmentation with the ROS 2 wrapper, refer to the ROS 2 Panoptic Segmentation section.