Panoptic Segmentation API
The panoptic segmentation API allows for panoptic segmentation to be run as part of the position SLAM system, providing semantic segmentation, 3D bounding boxes and the position of object instances relative to the camera that a user can use in their applications.
As part of the API, we provide a person detector via a pre-built dynamic link library (DLL) that can segment and detect people in the environment from the RGB images. This is available as a separate Debian package from the Slamcore Portal to be installed alongside our APIs or Tools.
Current we only support the pre-built DLL that we’ve provided for person detection. In the future, users may register custom inference callbacks or plugin DLLs to use with the panoptic segmentation API, as long as it complies with the same interface as our internal plugin.
There is currently no instance tracking in the API. The 3D pose and bounding box
for each detected object is reported relative to the
The person detection API is supported on the following platforms with JetPack 5.1 or 5.1.1 installed:
NVIDIA Jetson Xavier NX
NVIDIA Jetson Xavier AGX
NVIDIA Jetson Orin Nano
NVIDIA Jetson Orin NX
NVIDIA Jetson Orin AGX
The API can be used with the Intel RealSense D435i or D455 with the color stream enabled and optionally the depth stream for 3D bounding box visualization and position estimation.
As the RGB camera in the D455 is global shutter, the segmentation performance is expected to be better on the D455 than on the D435i which has a rolling shutter RGB.
From the Slamcore Portal, download the Slamcore Panoptic Segmentation Plugin, and any of the Slamcore SDK packages that you want to run the panoptic plugin on.
Install the Slamcore SDK package, for example:
$ sudo apt install </path/to/slamcore-tools>.deb
Install the panoptic plugin:
$ sudo apt install </path/to/slamcore-panoptic-segmentation>.deb
During this installation step, you may be informed about some CUDA dependency packages that will be installed.
To set up the segmentation model on your machine, run the
check_panoptic_segmentationscript which was installed by the plugin.
This process takes 10-15 minutes to complete.
$ check_panoptic_segmentation Generating an optimised network for your platform. This could take several minutes ... Successfully created a TensorRT engine at: ~/.slamcore/slamcore_panoptic_segmentation_plugin_network.engine Warming up device Device warmed
During this step, the embedded network is converted and optimised for the GPU platform of your machine.