Improve NuScenes example with more geo data & blueprint (#8130)

### What https://github.com/user-attachments/assets/0d09ae1b-100e-4c9e-bca6-de0f38dacff5 Draft todo: * [x] update thumbnail screenshots * [x] ? ### Checklist * [x] I have read and agree to [Contributor Guide](https://github.com/rerun-io/rerun/blob/main/CONTRIBUTING.md) and the [Code of Conduct](https://github.com/rerun-io/rerun/blob/main/CODE_OF_CONDUCT.md) * [x] I've included a screenshot or gif (if applicable) * [x] I have tested the web demo (if applicable): * Using examples from latest `main` build: [rerun.io/viewer](https://rerun.io/viewer/pr/8130?manifest_url=https://app.rerun.io/version/main/examples_manifest.json) * Using full set of examples from `nightly` build: [rerun.io/viewer](https://rerun.io/viewer/pr/8130?manifest_url=https://app.rerun.io/version/nightly/examples_manifest.json) * [x] The PR title and labels are set such as to maximize their usefulness for the next release's CHANGELOG * [x] If applicable, add a new check to the [release checklist](https://github.com/rerun-io/rerun/blob/main/tests/python/release_checklist)! * [x] If have noted any breaking changes to the log API in `CHANGELOG.md` and the migration guide - [PR Build Summary](https://build.rerun.io/pr/8130) - [Recent benchmark results](https://build.rerun.io/graphs/crates.html) - [Wasm size tracking](https://build.rerun.io/graphs/sizes.html) To run all checks from `main`, comment on the PR with `@rerun-bot full-check`. To deploy documentation changes immediately after merging this PR, add the `deploy docs` label.
rerun-io · Nov 14, 2024 · 21f4ddd · 21f4ddd
1 parent 0e72fb3
commit 21f4ddd
Show file tree

Hide file tree

Showing 3 changed files with 96 additions and 59 deletions.
diff --git a/examples/python/nuscenes_dataset/README.md b/examples/python/nuscenes_dataset/README.md
@@ -1,20 +1,20 @@
 <!--[metadata]
 title = "nuScenes"
 tags = ["Lidar", "3D", "2D", "Object detection", "Pinhole camera", "Blueprint"]
-thumbnail = "https://static.rerun.io/nuscenes/9c50bf5cadb879ef818ac3d35fe75696a9586cb4/480w.png"
-thumbnail_dimensions = [480, 480]
+thumbnail = "https://static.rerun.io/nuscenes_dataset/3724a84d6e95f15a71db2ccc443fb67bfae58843/480w.png"
+thumbnail_dimensions = [480, 301]
 channel = "release"
 build_args = ["--seconds=5"]
 -->
 
 Visualize the [nuScenes dataset](https://www.nuscenes.org/) including lidar, radar, images, and bounding boxes data.
 
-<picture data-inline-viewer="examples/nuscenes_dataset">
-  <img src="https://static.rerun.io/nuscenes/64a50a9d67cbb69ae872551989ee807b195f6b5d/full.png" alt="">
-  <source media="(max-width: 480px)" srcset="https://static.rerun.io/nuscenes/64a50a9d67cbb69ae872551989ee807b195f6b5d/480w.png">
-  <source media="(max-width: 768px)" srcset="https://static.rerun.io/nuscenes/64a50a9d67cbb69ae872551989ee807b195f6b5d/768w.png">
-  <source media="(max-width: 1024px)" srcset="https://static.rerun.io/nuscenes/64a50a9d67cbb69ae872551989ee807b195f6b5d/1024w.png">
-  <source media="(max-width: 1200px)" srcset="https://static.rerun.io/nuscenes/64a50a9d67cbb69ae872551989ee807b195f6b5d/1200w.png">
+<picture>
+  <img src="https://static.rerun.io/nuscenes_dataset/3724a84d6e95f15a71db2ccc443fb67bfae58843/full.png" alt="">
+  <source media="(max-width: 480px)" srcset="https://static.rerun.io/nuscenes_dataset/3724a84d6e95f15a71db2ccc443fb67bfae58843/480w.png">
+  <source media="(max-width: 768px)" srcset="https://static.rerun.io/nuscenes_dataset/3724a84d6e95f15a71db2ccc443fb67bfae58843/768w.png">
+  <source media="(max-width: 1024px)" srcset="https://static.rerun.io/nuscenes_dataset/3724a84d6e95f15a71db2ccc443fb67bfae58843/1024w.png">
+  <source media="(max-width: 1200px)" srcset="https://static.rerun.io/nuscenes_dataset/3724a84d6e95f15a71db2ccc443fb67bfae58843/1200w.png">
 </picture>
 
 ## Used Rerun types
@@ -83,6 +83,18 @@ rr.log(
 )
 ```
 
+#### GPS data
+
+GPS data is calculated from the scene's reference coordinates and the transformations (starting map point + odometry).
+The GPS coordinates are logged as [`GeoPoints`](https://www.rerun.io/docs/reference/types/archetypes/geopoints?speculative-link).
+
+```python
+rr.log(
+    "world/ego_vehicle",
+    rr.GeoPoints([[lat, long]]),
+)
+```
+
 ### LiDAR data
 LiDAR data is logged as [`Points3D`](https://www.rerun.io/docs/reference/types/archetypes/points3d) archetype.
 ```python
@@ -101,18 +113,6 @@ Radar data is logged similar to LiDAR data, as [`Points3D`](https://www.rerun.io
 rr.log(f"world/ego_vehicle/{sensor_name}", rr.Points3D(points, colors=point_colors))
 ```
 
-### GPS data
-
-GPS data is calculated from the scene's reference coordinates and the transformations (starting map point + odometry).
-The GPS coordinates are logged as [`GeoPoints`](https://www.rerun.io/docs/reference/types/archetypes/geopoints?speculative-link).
-
-```python
-rr.log(
-    "world/ego_vehicle/gps",
-    rr.GeoPoints([[lat, long]]),
-)
-```
-
 ### Annotations
 
 Annotations are logged as [`Boxes3D`](https://www.rerun.io/docs/reference/types/archetypes/boxes3d), containing details such as object positions, sizes, and rotation.
@@ -129,6 +129,8 @@ rr.log(
 )
 ```
 
+GPS coordinates are added to the annotations similarly to the vehicle.
+
 ### Setting up the default blueprint
 
 The default blueprint for this example is created by the following code:
@@ -138,6 +140,10 @@ sensor_space_views = [
     rrb.Spatial2DView(
         name=sensor_name,
         origin=f"world/ego_vehicle/{sensor_name}",
+        # Set the image plane distance to 5m for all camera visualizations.
+        defaults=[rr.components.ImagePlaneDistance(5.0)],
+        # TODO(#6670): Can't specify rr.components.FillMode.MajorWireframe right now, need to use batch type instead.
+        overrides={"world/anns": [rr.components.FillModeBatch("solid")]},
     )
     for sensor_name in nuscene_sensor_names(nusc, args.scene_name)
 ]
@@ -147,7 +153,7 @@ blueprint = rrb.Vertical(
         rrb.Vertical(
             rrb.TextDocumentView(origin="description", name="Description"),
             rrb.MapView(
-                origin="world/ego_vehicle/gps",
+                origin="world",
                 name="MapView",
                 zoom=rrb.archetypes.MapZoom(18.0),
                 background=rrb.archetypes.MapBackground(rrb.components.MapProvider.OpenStreetMap),

diff --git a/examples/python/nuscenes_dataset/nuscenes_dataset/__main__.py b/examples/python/nuscenes_dataset/nuscenes_dataset/__main__.py
@@ -58,7 +58,7 @@ def ensure_scene_available(root_dir: pathlib.Path, dataset_version: str, scene_n
         raise ValueError(f"{scene_name=} not found in dataset")
 
 
-def nuscene_sensor_names(nusc: nuscenes.NuScenes, scene_name: str) -> set[str]:
+def nuscene_sensor_names(nusc: nuscenes.NuScenes, scene_name: str) -> list[str]:
     """Return all sensor names in the scene."""
 
     sensor_names = set()
@@ -75,7 +75,16 @@ def nuscene_sensor_names(nusc: nuscenes.NuScenes, scene_name: str) -> set[str]:
                 sensor_names.add(sensor_name)
                 current_camera_token = sample_data["next"]
 
-    return sensor_names
+    # For a known set of cameras, order the sensors in a circle.
+    ordering = {
+        "CAM_FRONT_LEFT": 0,
+        "CAM_FRONT": 1,
+        "CAM_FRONT_RIGHT": 2,
+        "CAM_BACK_RIGHT": 3,
+        "CAM_BACK": 4,
+        "CAM_BACK_LEFT": 5,
+    }
+    return sorted(list(sensor_names), key=lambda sensor_name: ordering.get(sensor_name, float("inf")))
 
 
 def log_nuscenes(nusc: nuscenes.NuScenes, scene_name: str, max_time_sec: float) -> None:
@@ -110,7 +119,7 @@ def log_nuscenes(nusc: nuscenes.NuScenes, scene_name: str, max_time_sec: float)
     log_lidar_and_ego_pose(location, first_lidar_token, nusc, max_timestamp_us)
     log_cameras(first_camera_tokens, nusc, max_timestamp_us)
     log_radars(first_radar_tokens, nusc, max_timestamp_us)
-    log_annotations(first_sample_token, nusc, max_timestamp_us)
+    log_annotations(location, first_sample_token, nusc, max_timestamp_us)
 
 
 def log_lidar_and_ego_pose(
@@ -119,6 +128,8 @@ def log_lidar_and_ego_pose(
     """Log lidar data and vehicle pose."""
     current_lidar_token = first_lidar_token
 
+    ego_trajectory_lat_lon = []
+
     while current_lidar_token != "":
         sample_data = nusc.get("sample_data", current_lidar_token)
         sensor_name = sample_data["channel"]
@@ -131,23 +142,29 @@ def log_lidar_and_ego_pose(
 
         ego_pose = nusc.get("ego_pose", sample_data["ego_pose_token"])
         rotation_xyzw = np.roll(ego_pose["rotation"], shift=-1)  # go from wxyz to xyzw
+        position_lat_lon = derive_latlon(location, ego_pose)
+        ego_trajectory_lat_lon.append(position_lat_lon)
+
         rr.log(
             "world/ego_vehicle",
             rr.Transform3D(
                 translation=ego_pose["translation"],
                 rotation=rr.Quaternion(xyzw=rotation_xyzw),
+                axis_length=10.0,  # The length of the visualized axis.
                 from_parent=False,
             ),
+            rr.GeoPoints(lat_lon=position_lat_lon, radii=rr.Radius.ui_points(8.0), colors=0xFF0000FF),
         )
-        current_lidar_token = sample_data["next"]
-
-        # log GPS data
-        (lat, long) = derive_latlon(location, ego_pose)
+        # TODO(#6889): We don't want the radius for the trajectory line to be the same as the radius of the points.
+        # However, rr.GeoPoints uses the same `rr.components.Radius` for this, so these two archetypes would influence each other
+        # if logged on the same entity. In the future, they will have different tags, which will allow them to live side by side.
         rr.log(
-            "world/ego_vehicle/gps",
-            rr.GeoPoints(lat_lon=[[lat, long]]),
+            "world/ego_vehicle/trajectory",
+            rr.GeoLineStrings(lat_lon=ego_trajectory_lat_lon, radii=rr.Radius.ui_points(1.0), colors=0xFF0000FF),
         )
 
+        current_lidar_token = sample_data["next"]
+
         data_file_path = nusc.dataroot / sample_data["filename"]
         pointcloud = nuscenes.LidarPointCloud.from_file(str(data_file_path))
         points = pointcloud.points[:3].T  # shape after transposing: (num_points, 3)
@@ -193,7 +210,7 @@ def log_radars(first_radar_tokens: list[str], nusc: nuscenes.NuScenes, max_times
             current_camera_token = sample_data["next"]
 
 
-def log_annotations(first_sample_token: str, nusc: nuscenes.NuScenes, max_timestamp_us: float) -> None:
+def log_annotations(location: str, first_sample_token: str, nusc: nuscenes.NuScenes, max_timestamp_us: float) -> None:
     """Log 3D bounding boxes."""
     label2id: dict[str, int] = {}
     current_sample_token = first_sample_token
@@ -207,6 +224,7 @@ def log_annotations(first_sample_token: str, nusc: nuscenes.NuScenes, max_timest
         centers = []
         quaternions = []
         class_ids = []
+        lat_lon = []
         for ann_token in ann_tokens:
             ann = nusc.get("sample_annotation", ann_token)
 
@@ -218,6 +236,7 @@ def log_annotations(first_sample_token: str, nusc: nuscenes.NuScenes, max_timest
             if ann["category_name"] not in label2id:
                 label2id[ann["category_name"]] = len(label2id)
             class_ids.append(label2id[ann["category_name"]])
+            lat_lon.append(derive_latlon(location, ann))
 
         rr.log(
             "world/anns",
@@ -226,14 +245,13 @@ def log_annotations(first_sample_token: str, nusc: nuscenes.NuScenes, max_timest
                 centers=centers,
                 quaternions=quaternions,
                 class_ids=class_ids,
-                fill_mode=rr.components.FillMode.Solid,
             ),
+            rr.GeoPoints(lat_lon=lat_lon),
         )
         current_sample_token = sample_data["next"]
 
-    # skipping for now since labels take too much space in 3D view (see https://github.com/rerun-io/rerun/issues/4451)
-    # annotation_context = [(i, label) for label, i in label2id.items()]
-    # rr.log("world/anns", rr.AnnotationContext(annotation_context), static=True)
+    annotation_context = [(i, label) for label, i in label2id.items()]
+    rr.log("world/anns", rr.AnnotationContext(annotation_context), static=True)
 
 
 def log_sensor_calibration(sample_data: dict[str, Any], nusc: nuscenes.NuScenes) -> None:
@@ -296,26 +314,39 @@ def main() -> None:
         rrb.Spatial2DView(
             name=sensor_name,
             origin=f"world/ego_vehicle/{sensor_name}",
+            contents=["$origin/**", "world/anns"],
+            # TODO(#6670): Can't specify rr.components.FillMode.MajorWireframe right now, need to use batch type instead.
+            overrides={"world/anns": [rr.components.FillModeBatch("majorwireframe")]},
         )
         for sensor_name in nuscene_sensor_names(nusc, args.scene_name)
     ]
-    blueprint = rrb.Vertical(
-        rrb.Horizontal(
-            rrb.Spatial3DView(name="3D", origin="world"),
-            rrb.Vertical(
-                rrb.TextDocumentView(origin="description", name="Description"),
-                rrb.MapView(
-                    origin="world/ego_vehicle/gps",
-                    name="MapView",
-                    zoom=rrb.archetypes.MapZoom(18.0),
-                    background=rrb.archetypes.MapBackground(rrb.components.MapProvider.OpenStreetMap),
+    blueprint = rrb.Blueprint(
+        rrb.Vertical(
+            rrb.Horizontal(
+                rrb.Spatial3DView(
+                    name="3D",
+                    origin="world",
+                    # Set the image plane distance to 5m for all camera visualizations.
+                    defaults=[rr.components.ImagePlaneDistance(5.0)],
+                    # TODO(#6670): Can't specify rr.components.FillMode.MajorWireframe right now, need to use batch type instead.
+                    overrides={"world/anns": [rr.components.FillModeBatch("solid")]},
+                ),
+                rrb.Vertical(
+                    rrb.TextDocumentView(origin="description", name="Description"),
+                    rrb.MapView(
+                        origin="world",
+                        name="MapView",
+                        zoom=rrb.archetypes.MapZoom(18.0),
+                        background=rrb.archetypes.MapBackground(rrb.components.MapProvider.OpenStreetMap),
+                    ),
+                    row_shares=[1, 1],
                 ),
-                row_shares=[1, 1],
+                column_shares=[3, 1],
             ),
-            column_shares=[3, 1],
+            rrb.Grid(*sensor_space_views),
+            row_shares=[4, 2],
         ),
-        rrb.Grid(*sensor_space_views),
-        row_shares=[4, 2],
+        rrb.TimePanel(state="collapsed"),
     )
 
     rr.script_setup(args, "rerun_example_nuscenes", default_blueprint=blueprint)

diff --git a/examples/python/nuscenes_dataset/nuscenes_dataset/download_dataset.py b/examples/python/nuscenes_dataset/nuscenes_dataset/download_dataset.py
@@ -10,16 +10,16 @@
 import tqdm
 
 MINISPLIT_SCENES = [
-    "scene-0061",
-    "scene-0103",
-    "scene-0553",
-    "scene-0655",
-    "scene-0757",
-    "scene-0796",
-    "scene-0916",
-    "scene-1077",
-    "scene-1094",
-    "scene-1100",
+    "scene-0061",  # Driving around a corner.
+    "scene-0103",  # Driving straight on a city road.
+    "scene-0553",  # Standing in front of a traffic light.
+    "scene-0655",  # Drive straight only.
+    "scene-0757",  # Goes straight rather slowly.
+    "scene-0796",  # Drive straight only.
+    "scene-0916",  # Driving in a circle on a parking lot.
+    "scene-1077",  # Straight drive on an artery road at night.
+    "scene-1094",  # Slow drive at night.
+    "scene-1100",  # Standing on front of traffic light at night.
 ]
 MINISPLIT_URL = "https://www.nuscenes.org/data/v1.0-mini.tgz"