Release 0.13.0 (#614)

See CHANGELOG.rst for details!

Co-authored-by: Tom Slankard <[email protected]>

.github/workflows/build_test.yaml

@@ -18,7 +18,7 @@ jobs:
     - name: install python-deps
       run: pip3 install mypy flake8
     - name: python-lint
-      run: cd python && mypy ./src ./tests && flake8
+      run: cd python && mypy ./src ./tests ../tests/hil ../tests/integration && flake8
   linux-build:
     runs-on: ubuntu-latest
     steps:

.gitmodules

@@ -0,0 +1,4 @@
+[submodule "sdk-extensions"]
+	path = sdk-extensions
+	url = ../sdk-extensions.git
+	branch = master

CHANGELOG.rst

@@ -2,6 +2,100 @@
 Changelog
 =========
 
+**Important: as of 0.13.0, the SDK is no longer compatible with firmware versions older than 2.1.0.**
+
+[20240702] [0.13.0]
+======================
+
+ouster_osf
+------------------------
+* Add full index of both receive and sensor timestamps to metadata
+* Speed up opening of OSF files with index
+
+* OSF now saves alert flags, thermal countdown and status, shot limiting countdown and status from ``LidarScan``.
+* [BUGFIX] Fix OSF being unable to load LidarScans containing only custom fields
+* [BUGFIX] Fix OSF not flushed when the user pressed CTRL-C more than once
+* [BUGFIX] Fix improper timestamps when saving OSF on MacOS(m-series) and Windows
+* [BUGFIX] Fix an issue with destaggering images after modifying ``SensorInfo`` in an ``OsfScanSource``.
+* [BUGFIX] Fix an issue loading extrinsics from OSF metadata into a ``SensorInfo`` in ``OsfScanSource``.
+* [BREAKING] Remove ``ChunksLayout`` and ``ChunkRef`` from Python API.
+
+ouster_client/Python SDK
+------------------------
+
+* Add support for reading and writing ROS1 and ROS2 bag files.
+* Add new sensor client interface ``ouster::sensor::SensorClient`` which supports multiple sensors as well as multiple sensors and IMU data on the same port
+* Add higher level sensor client interface ```ouster::sensor::SensorScanSource`` which produces ``LidarScan`` s from multiple sensors
+* Add ``ouster.sdk.client.SensorPacketSource`` which receives packets from multiple sensors
+* Add support for multiple sensors to ``ouster.sdk.sensor.SensorScanSource``
+* Greatly reduced redundant HTTP API calls to the sensor during initialization
+* Deserialize FLAGS fields in each profile by default
+* Add support for IPv6 multicast
+* Add ``field_names`` argument to each scan source and to ``open_source`` to specify which fields to decode
+* Add metadata validation functionality
+* Add vendored json library
+* Improved multi sensor pcap reading
+* Improve ``ScanBatcher`` to release ``LidarScan`` as soon as they are completed
+* ``ScanBatcher`` now adds alert flags, thermal countdown, and shot limiting countdown to ``LidarScan``.
+* Use index to speed up ``ouster-cli source .osf info``
+* Use index to speed up slicing of indexed OSF sources when sliced immediately after the ``source`` command
+* Add ``LidarScan.get_first_valid_column_timestamp()``
+* Add ``crc`` and ``calculate_crc`` methods to ``ouster::sensor::packet_format`` for obtaining or calculating (respectively) the CRC64 of a packet.
+* ``scan_to_packets`` now creates packets with alert flags, thermal countdown and status, shot limiting countdown and status, and CRC64.
+* Add ``ouster::pose_util::dewarp`` C++ function to de-warp a ``LidarScan`` (similar to ``ouster.sdk.pose_util`` in the Python API.)
+* Add a constructor ``LidarScan(const ouster::sensor::sensor_info&)``.
+* Always use ``nonstd::optional`` instead of drop-in ``std::optional`` from https://github.com/martinmoene/optional-lite.git to reduce issues associated with mixing C++14 and C++17.
+* Add ``w()`` and ``h()`` methods to ``sensor_info`` in C++ and ``w`` and ``h`` properties to ``SensorInfo`` in Python.
+* [BUGFIX] fix automatic UDP dest for FW 2.3 sensors.
+
+
+* [BREAKING] Remove ``ouster::make_xyz_lut(const ouster::sensor::sensor_info&)``. (Use ``make_xyz_lut(const sensor::sensor_info& sensor, bool use_extrinsics)`` instead.)
+* [BREAKING] changed REFLECTIVITY channel field size to 8 bits. (Important - this makes the SDK incompatible with FW 2.0 and 2.1.)
+* [BREAKING] Removed ``UDPPacketSource`` and ``BufferedUDPSource``.
+* [BREAKING] Removed ``ouster.sdk.util.firmware_version(hostname)`` please use ``ouster.sdk.client.SensorHttp.create(hostname).firmware_version()`` instead
+* [BREAKING] ``open_source`` no longer automatically finds and applies extrinsics from ``sensor_extrinsics.json`` files. Use the ``extrinsics`` argument instead to specify the path to the relevant extrinsics file instead.
+* [BREAKING] Deprecated ``osf.Scans(...)`` for ``osf.OsfScanSource(...).single_source(0)```.
+* [BREAKING] Deprecated ``client.Sensor(...)`` for ``client.SensorPacketSource(...).single_source(0)```.
+* [BREAKING] Deprecated ``pcap.Pcap(...)`` for ``pcap.PcapMultiPacketReader(...).single_source(0)```.
+* [BREAKING] Deprecated ``ScanBatcher::ScanBatcher(size_t, const packet_format&)`` for ``ScanBatcher::ScanBatcher(const sensor_info&)``.
+* [FUTURE BREAKING] Removing all instances of jsoncpp's ``Json::Value`` from the public C++ API methods in favor of ``std::string``.
+
+ouster_viz
+----------
+
+* ``LidarScanViz`` now supports multi-sensor datasets.
+* Add Python callback registration methods for mouse button and scroll events from ``PointViz``.
+* Add Python and C++ callback registration methods for frame buffer resize events.
+* Add ``MouseButton``, ``MouseButtonEvent``, and ``EventModifierKeys`` enums.
+* Add methods ``aspect_ratio``, ``normalized_coordinates``, and ``window_coordinates`` to ``viz::WindowCtx``.
+* Add method ``window_coordinates_to_image_pixel`` to ``viz::Image``. (See ``viz_events_example.cpp`` for an example.)
+* Add ``current_camera()`` method to ``PointViz``.
+* [BREAKING] ``SimpleViz`` no longer accepts a ``ScansAccumulator`` instance and now accepts scan/map accumulation parameters as keyword args in its constructor.
+* [BREAKING] ``ScansAccumulator`` is split into several different classes: ``ScansAccumulator``, ``MapAccumulator``, ``TracksAccumulator``, and ``LidarScanVizAccumulators``.
+* [BREAKING] changed ``PointViz`` mouse button callback to fire for both mouse button press and release events.
+* [BREAKING] changed ``PointViz`` mouse button callback signature to use the new enums.
+* [BREAKING] removed ``bool update_on_input()`` and ``update_on_input(bool)`` methods from ``PointViz``.
+* [BUGFIX] SimpleViz throws a 'generator already executing' exception.
+
+ouster_cli
+----------
+
+* Add support for reading and writing ROS1 and ROS2 bag files.
+* Add support for working with multi scan sources.
+* Add ``--fields`` argument to ``ouster-cli source`` to specify which fields to decode.
+* Add metadata validation utility.
+* [BUGFIX] Program doesn't terminate immediately when pressing CTRL-C the first time when streaming from a live sensor.
+* [BUGFIX] Fix some errors that appeared when running ``ouster-cli util benchmark``
+* [BREAKING] ``source`` no longer automatically finds and applies extrinsics from ``sensor_extrinsics.json`` files. Use the ``-E`` argument instead to specify the path to the relevant extrinsics file instead.
+* [BREAKING] Moved raw recording functionality for BAG and PCAP to ``ouster-cli source ... record_raw`` command.
+* [BREAKING] CLI plugins now need to handle a list of Optional[LidarScan] instead of a single LidarScan to support multi sources.
+
+mapping
+-------
+
+* Update KissICP version from 0.4.0 to 1.0.0.
+* Add multi-sensor support.
+
 [20240702] [0.12.0]
 ===================
 
@@ -64,7 +158,9 @@ ouster-cli
 * Add chainable ``ouster-cli source ... stats`` command
 * Add chainable ``ouster-cli source ... clip`` command to discard points outside a provided range
 * Add ``--rate max`` option to ``ouster-cli source ... viz```
-* Improve argument naming and descriptions for ``ouster-cli source ... viz`` map and accum options
+* Improve argument naming and descriptions for ``ouster-cli source ... viz`` map and accum options:
+  ``--accum-map`` is now called ``--map`` and ``--accum-map-ratio`` is now called ``--map-ratio``.
+* New ``--map-size`` argument to set the maximum number of points used when ``--map`` is specified.
 
 * [BUGFIX] Prevent dropped frames from live sensors by consuming scans as fast as they come in rather than sleeping
 
@@ -137,20 +233,18 @@ Python SDK
 * Add support for python 3.12, including wheels on pypi
 * Updated VCPKG libraries to 2023.10.19
 * New ``ScanSource`` API:
-  * Added new ``MultiScanSource`` that supports streaming and manipulating LidarScan frames
-    from multiple concurrent LidarScan sources
-    * For non-live sources the ``MultiScanSource`` have the option to choose LidarScan(s) by index
-    or choose a subset of scans using slicing operation
-    * The ``MultiScanSource`` interface has the ability to fallback to ``ScanSource`` using the 
-      ``single_source(sensor_idx)``, ``ScanSource`` interface yield a single LidarScan on iteration
-      rather than a List
-    * The ``ScanSource`` interface obtained via ``single_source`` method supports same indexing and
-      and slicing operations as the ``MultiScanSource``
-  * Added a generic ``open_source`` that accepts sensor urls, or a path to a pcap recording
-    or an osf file
+
+   * Added new ``MultiScanSource`` that supports streaming and manipulating LidarScan frames from multiple concurrent LidarScan sources
+
+     * For non-live sources the ``MultiScanSource`` has the option to choose LidarScan(s) by index or choose a subset of scans using slicing operation
+     * The ``MultiScanSource`` interface has the ability to fallback to ``ScanSource`` using the ``single_source(sensor_idx)``, ``ScanSource`` interface yield a single LidarScan on iteration rather than a List
+     * The ``ScanSource`` interface obtained via ``single_source`` method supports same indexing and and slicing operations as the ``MultiScanSource``
+
+  * Added a generic ``open_source`` that accepts sensor urls, or a path to a pcap recording or an osf file
   * Add explicit flag ``index`` to index unindexed osf files, if flag is set to ``True`` the osf file
     will be indexed and the index will be saved to the file on first attempt
   * Display a progress bar during index of pcap file or osf (if unindexed)
+
 * Improved the robustness of the ``resolve_metadata`` method used to
   automatically identify the sensor metadata associated with a PCAP source.
 * [bugfix] SimpleViz complains about missing fields
@@ -240,6 +334,7 @@ Known issues
 * ouster-cli when combining ``slice`` command with ``viz`` the program will
   exit once iterate over the selected range of scans even when
   the ``--on-eof`` option is set to ``loop``.
+
   - workaround: to have ``viz`` loop over the selected range, first perform a
     ``slice`` with ``save``, then playback the generated file.
 

CMakeLists.txt

@@ -13,10 +13,10 @@ include(DefaultBuildType)
 include(VcpkgEnv)
 
 # ==== Project Name ====
-project(ouster_example VERSION 20231031)
+project(ouster_sdk VERSION 0.13.0)
 
 # generate version header
-set(OusterSDK_VERSION_STRING 0.12.0)
+set(OusterSDK_VERSION_STRING 0.13.0)
 include(VersionGen)
 
 # ==== Options ====

Jenkinsfile

@@ -0,0 +1,30 @@
+node {
+    withEnv(["ARTIFACT_DIR=${WORKSPACE}/artifacts"]) {
+        withCredentials([string(credentialsId: 'LIBRARY_PATH', variable: 'LIBRARY_PATH')]) {
+            if (params.CLEAN_WORKSPACE) {
+                echo "CLEAN_WORKSPACE set: deleting everything"
+                cleanWs()
+            }
+            dir('ouster-sdk') {
+                checkout([
+                        $class           : 'GitSCM',
+                        branches         : [[name: "*/${BRANCH_NAME}"]],
+                        extensions       : scm.extensions + [
+                                [$class: 'CleanCheckout'],
+                                [$class             : 'SubmoduleOption',
+                                 disableSubmodules  : false, parentCredentials: true,
+                                 recursiveSubmodules: true,
+                                 trackingSubmodules : true
+                                ]
+                        ],
+                        userRemoteConfigs: scm.userRemoteConfigs
+                ])
+            }
+
+            dir(env.ARTIFACT_DIR) { deleteDir() }
+
+            run_pipeline = load "${LIBRARY_PATH}"
+            run_pipeline()
+        }
+    }
+}

LICENSE

@@ -1,7 +1,7 @@
 Format: https://www.debian.org/doc/packaging-manuals/copyright-format/1.0/
 Upstream-Name: OusterSDK
 Upstream-Contact: Ouster Sensor SDK Developers <[email protected]>
-Source: https://github.com/ouster-lidar/ouster_example
+Source: https://github.com/ouster-lidar/ouster_sdk
 
 Files: *
 Copyright: 2018-2022 Ouster, Inc
@@ -11,6 +11,10 @@ Files: include/optional-lite/*
 Copyright: 2014-2021 Martin Moene
 License: BSL-1.0
 
+Files: include/jsoncons*
+Copyright Daniel Parker 2013 - 2020
+License: BSL-1.0
+
 License: BSD-3-Clause
  Redistribution and use in source and binary forms, with or without
  modification, are permitted provided that the following conditions are met:

LICENSE-bin

@@ -71,6 +71,35 @@ License: MIT
     CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
     SOFTWARE.
 
+Name: jsoncons
+Description: statically linked vendored header only library
+Availability: https://github.com/danielaparker/jsoncons/
+Copyright Daniel Parker 2013 - 2020
+License: BSL-1.0
+Boost Software License - Version 1.0 - August 17th, 2003
+
+Permission is hereby granted, free of charge, to any person or organization
+obtaining a copy of the software and accompanying documentation covered by
+this license (the "Software") to use, reproduce, display, distribute,
+execute, and transmit the Software, and to prepare derivative works of the
+Software, and to permit third-parties to whom the Software is furnished to
+do so, all subject to the following:
+
+The copyright notices in the Software and this entire statement, including
+the above license grant, this restriction and the following disclaimer,
+must be included in all copies of the Software, in whole or in part, and
+all derivative works of the Software, unless such copies or derivative
+works are solely in the form of machine-executable object code generated by
+a source language processor.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE, TITLE AND NON-INFRINGEMENT. IN NO EVENT
+SHALL THE COPYRIGHT HOLDERS OR ANYONE DISTRIBUTING THE SOFTWARE BE LIABLE
+FOR ANY DAMAGES OR OTHER LIABILITY, WHETHER IN CONTRACT, TORT OR OTHERWISE,
+ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
+DEALINGS IN THE SOFTWARE.
+
 Name: libpcap
 Description: statically linked
 Availability: https://github.com/the-tcpdump-group/libpcap

README.rst

@@ -1,4 +1,4 @@
-.. figure:: https://github.com/ouster-lidar/ouster_example/raw/master/docs/images/Ouster_Logo_TM_Horiz_Black_RGB_600px.png
+.. figure:: https://github.com/ouster-lidar/ouster_sdk/raw/master/docs/images/Ouster_Logo_TM_Horiz_Black_RGB_600px.png
 
 ------------------------------------------------------
 
@@ -22,6 +22,8 @@ reading and visualizing data.
 * `ouster_osf <ouster_osf/>`_ contains C++ OSF library to store ouster sensors data
 * `ouster_viz <ouster_viz/>`_ contains a customizable point cloud visualizer
 * `python <python/>`_ contains the code for the ouster sensor python SDK (``ouster-sdk`` Python package)
+* `sdk-extenstions` is a submodule of the ouster-sdk repository which is currently for internal use only.
+The submodule cannot be cloned or updated.
 
 .. note::
     Ouster ROS driver code has been moved out to a separate GitHub repository. To get started using the
@@ -31,7 +33,7 @@ reading and visualizing data.
 Contact
 =======
 
-For support of the Ouster SDK, please use `Github Issues <https://github.com/ouster-lidar/ouster_example/issues>`_ in this repo.
+For support of the Ouster SDK, please use `Github Issues <https://github.com/ouster-lidar/ouster_sdk/issues>`_ in this repo.
 
 For support of Ouster products outside of the SDK, please use `Ouster customer support <https://ouster.atlassian.net/servicedesk/customer/portal/8/group/22/create/86>`_.
 

Security.md

@@ -0,0 +1,39 @@
+# Ouster Security Policy
+
+## Overview
+
+This security policy outlines the security support commitments for ouster-sdk users.
+
+[Email]([email protected]) to learn more about Ouster's security SLAs and process.
+
+## BSD-3-Clause License Users
+
+- **Security SLA:** No security Service Level Agreement (SLA) is provided.
+- **Release Schedule:** Releases are planned every 3 months. These releases
+  will contain all security fixes implemented up to that point.
+- **Version Support:** Security patches are only provided for the current
+  release version.
+
+## Reporting a Vulnerability
+
+Please [email]([email protected]) to report a security vulnerability.
+
+Please include the requested information listed below (as much as you can provide) to help us better understand 
+the nature and scope of the possible issue:
+
+- Type of issue (e.g. buffer overflow, SQL injection, cross-site scripting, etc.)
+- Full paths of source file(s) related to the manifestation of the issue
+- The location of the affected source code (tag/branch/commit or direct URL)
+- Any special configuration required to reproduce the issue
+- Step-by-step instructions to reproduce the issue
+- Proof-of-concept or exploit code (if possible)
+Impact of the issue, including how an attacker might exploit the issue
+- This information will help us triage your report more quickly.
+
+### Preferred Languages
+We prefer all communications to be in English.
+
+### Response time 
+You should receive a response within 24 hours. 
+
+If for some reason you do not, please follow up via email to ensure we received your original message.

clang-linting.sh

@@ -1,5 +1,7 @@
 #!/bin/bash
 
+set -euo pipefail
+
 help='false'
 apply='false'
 check='true'
@@ -28,15 +30,23 @@ if [[ $apply == 'true' ]]; then
   SUBCOMMAND=''
 fi
 
-
 # Ignore the ./ouster_client/include/optional-lite/nonstd/optional.hpp file while formatting files
-CMD="find . -regex '.*\.\(cpp\|hpp\|cu\|c\|h\)' -not -path ./ouster_client/include/optional-lite/nonstd/optional.hpp \
--exec clang-format ${SUBCOMMAND} -style=file -i {} \;"
-OUTPUT=$(eval "$CMD" 2>&1 | grep error)
+run_command() {
+  find . -regex '.*\.\(cpp\|hpp\|cu\|c\|h\)' -not -path './ouster_client/include/optional-lite/nonstd/optional.hpp' \
+  -and -not -path './thirdparty/*' \
+  -exec clang-format ${SUBCOMMAND} -style=file -i {} \;
+   echo "Exit code: $?"
+}
+# grep returns an exit code of 1 if it finds no matches.
+set +e
+OUTPUT=$(run_command 2>&1 | grep error)
+set -e
+echo "OUTPUT $OUTPUT"
+
 if [ -z "$OUTPUT" ]; then
   echo "clang-format check passed! No errors"
   exit 0;
 else
-  eval "$CMD"
+  run_command
   exit 1;
-fi
+fi

cmake/FindOusterSDK.cmake

@@ -1,5 +1,5 @@
 # Allow downstream code to depend on source transparently
 if(NOT TARGET EXAMPLE_INCLUDED)
   add_custom_target(EXAMPLE_INCLUDED)
-  add_subdirectory(${CMAKE_CURRENT_LIST_DIR}/.. ouster_example EXCLUDE_FROM_ALL)
+  add_subdirectory(${CMAKE_CURRENT_LIST_DIR}/.. ouster_sdk EXCLUDE_FROM_ALL)
 endif()

conanfile.py

@@ -11,7 +11,7 @@ class ousterSdkRecipe(ConanFile):
     package_type = "library"
     license = "BSD 3-Clause License"
     author = "Ouster, Inc."
-    url = "https://github.com/ouster-lidar/ouster_example"
+    url = "https://github.com/ouster-lidar/ouster_sdk"
     description = "Ouster SDK - tools for working with Ouster Lidars"
     topics = ("lidar", "driver", "hardware", "point cloud", "3d", "robotics", "automotive")
     settings = "os", "compiler", "build_type", "arch"
@@ -43,6 +43,7 @@ class ousterSdkRecipe(ConanFile):
         "ouster_osf/*",
         "ouster_viz/*",
         "tests/*",
+        "thirdparty/*",
         "CMakeLists.txt",
         "CMakeSettings.json",
         "LICENSE",