Rerun Viewer breaks after connection from client terminated

[dominikandreas]

Describe the bug
When logging data from a python client and killing the python client mid process, the rerun viewer does not recover and is unable to display recordings from new connections.

To Reproduce
Steps to reproduce the behavior:

1. Start rerun
2. Connect with a python client using rr.connect_grpc(f"rerun+http://{host_ip}:9876/proxy")
3. Kill the python client while streaming data
4. Restart client from step 2. -> the rerun visualizer no longer shows data from the second run

Expected behavior
Rerun shows data from the second run

Backtrace
Viewer logs

[2025-06-04T06:35:54Z DEBUG tonic::codec::decode] decoder inner stream error: Status { code: Unknown, message: "h2 protocol error: error reading a body from connection", source: Some(Error { inner: Status { code: Unknown, message: "h2 protocol error: error reading a body from connection", source: Some(hyper::Error(Body, Error { kind: Io(Custom { kind: BrokenPipe, error: "stream closed because of a broken pipe" }) })) } }) }
[2025-06-04T06:35:54Z ERROR re_grpc_server] Error while receiving messages: status: Unknown, message: "h2 protocol error: error reading a body from connection", details: [], metadata: MetadataMap { headers: {} }

Full log:

Full debug log of the viewer

$ RUST_LOG=debug rerun 
[2025-06-04T06:48:46Z DEBUG rerun::commands::entrypoint] Detected 16 cores. Using 14 compute threads.
[2025-06-04T06:48:46Z DEBUG rerun::commands::entrypoint] Parsing memory limit for Viewer
[2025-06-04T06:48:46Z DEBUG re_memory::memory_limit] Setting memory limit to 16.4 GiB, which is 75% of total available memory (21.8 GiB).
[2025-06-04T06:48:46Z DEBUG rerun::commands::entrypoint] Parsing memory limit for gRPC server
[2025-06-04T06:48:46Z DEBUG re_memory::memory_limit] Setting memory limit to 5.5 GiB, which is 25% of total available memory (21.8 GiB).
[2025-06-04T06:48:46Z DEBUG eframe] Using the wgpu renderer
[2025-06-04T06:48:46Z INFO  re_grpc_server] Listening for gRPC connections on 0.0.0.0:9876. Connect by running `rerun --connect rerun+http://127.0.0.1:9876/proxy`
[2025-06-04T06:48:46Z DEBUG eframe::native::wgpu_integration] Event::Resumed
[2025-06-04T06:48:46Z DEBUG eframe::native::file_storage] Loading app state from "C:\\Users\\<redacted>\\AppData\\Roaming\\rerun\\data\\app.ron"…
[2025-06-04T06:48:46Z DEBUG egui_wgpu::setup] Creating wgpu instance with backends Backends(VULKAN | GL | METAL)
[2025-06-04T06:48:47Z INFO  egui_wgpu] There were 2 available wgpu adapters: {backend: Vulkan, device_type: IntegratedGpu, name: "AMD Radeon (TM) Graphics", driver: "AMD proprietary driver", driver_info: "24.3.1 (AMD proprietary shader compiler)", vendor: AMD (0x1002), device: 0x15E7}, {backend: Gl, device_type: Other, name: "AMD Radeon (TM) Graphics", driver_info: "4.6.0 Compatibility Profile Context 24.3.1.240216"}
[2025-06-04T06:48:47Z DEBUG egui_wgpu] Picked wgpu adapter: backend: Vulkan, device_type: IntegratedGpu, name: "AMD Radeon (TM) Graphics", driver: "AMD proprietary driver", driver_info: "24.3.1 (AMD proprietary shader compiler)", vendor: AMD (0x1002), device: 0x15E7
[2025-06-04T06:48:47Z DEBUG re_renderer::context] wgpu adapter backend: Vulkan, device_type: IntegratedGpu, name: "AMD Radeon (TM) Graphics", driver: "AMD proprietary driver", driver_info: "24.3.1 (AMD proprietary shader compiler)"
[2025-06-04T06:48:47Z WARN  re_viewer::ui::welcome_screen::example_section] Failed to load examples: https://app.rerun.io/version/0.23.1/examples_manifest.json: Connection Failed: tls connection init failed: invalid peer certificate: UnknownIssuer
[2025-06-04T06:48:47Z WARN  wgpu_hal::vulkan] Suboptimal present of frame 0
[2025-06-04T06:48:49Z DEBUG re_analytics::native::sink] Failed to send analytics down the sink: https://tel.rerun.io/: Connection Failed: tls connection init failed: invalid peer certificate: UnknownIssuer
[2025-06-04T06:49:13Z DEBUG tonic_web::service] kind="other h2" content_type=Some("application/grpc")
[2025-06-04T06:49:13Z DEBUG re_analytics::native::pipeline] dropped event, analytics channel is disconnected
[2025-06-04T06:49:13Z DEBUG egui::context] Ignoring call request_discard, because max_passes=2. Requested from [crates\viewer\re_ui\src\list_item\scope.rs:79 Missing re_ui::LayoutStatistics, crates\viewer\re_ui\src\list_item\scope.rs:79 Missing re_ui::LayoutStatistics, crates\viewer\re_ui\src\list_item\scope.rs:79 Missing re_ui::LayoutStatistics, C:\Users\runneradmin\.cargo\registry\src\index.crates.io-6f17d22bba15001f\egui-0.31.1\src\grid.rs:452 new Grid, crates\viewer\re_ui\src\list_item\scope.rs:79 Missing re_ui::LayoutStatistics, crates\viewer\re_ui\src\list_item\scope.rs:79 Missing re_ui::LayoutStatistics, crates\viewer\re_ui\src\list_item\scope.rs:79 Missing re_ui::LayoutStatistics, crates\viewer\re_ui\src\list_item\scope.rs:79 Missing re_ui::LayoutStatistics]
[2025-06-04T06:49:13Z DEBUG reqwest::connect] starting new connection: https://tile.openstreetmap.org/
[2025-06-04T06:49:13Z DEBUG reqwest::connect] starting new connection: https://tile.openstreetmap.org/
[2025-06-04T06:49:13Z DEBUG reqwest::connect] starting new connection: https://tile.openstreetmap.org/
[2025-06-04T06:49:13Z DEBUG reqwest::connect] starting new connection: https://tile.openstreetmap.org/
[2025-06-04T06:49:13Z DEBUG reqwest::connect] starting new connection: https://tile.openstreetmap.org/
[2025-06-04T06:49:13Z DEBUG reqwest::connect] starting new connection: https://tile.openstreetmap.org/
[2025-06-04T06:49:14Z DEBUG tonic::codec::decode] decoder inner stream error: Status { code: Unknown, message: "h2 protocol error: error reading a body from connection", source: Some(Error { inner: Status { code: Unknown, message: "h2 protocol error: error reading a body from connection", source: Some(hyper::Error(Body, Error { kind: Io(Custom { kind: BrokenPipe, error: "stream closed because of a broken pipe" }) })) } }) }
[2025-06-04T06:49:14Z ERROR re_grpc_server] Error while receiving messages: status: Unknown, message: "h2 protocol error: error reading a body from connection", details: [], metadata: MetadataMap { headers: {} }
[2025-06-04T06:49:18Z DEBUG re_viewer::app] Saved blueprint for <redacted>_20250604_084912 to "C:\\Users\\<redacted>\\AppData\\Roaming\\rerun\\data\\blueprints\\20240911_123354_mtcejw1.000064-001512_20250604_084912-3291aedc451ce35a.rbl"

Note: the TLS issues in the logs can be ignored, this is caused by running the viewer viewer behind a corporate proxy

Desktop (please complete the following information):

• OS: Windows 11 (Viewer), Ubuntu 22.04 (client)

Rerun version
Windows Host

> rerun --version
rerun-cli 0.23.1 (default map_view nasm native_viewer release web_viewer) [rustc 1.84.0 (9fc6b4312 2025-01-07), LLVM 19.1.5] x86_64-pc-windows-msvc release-0.23.1 87bdacb, built 2025-04-25T11:15:37Z
Video features: av1 default ffmpeg nasm serde

Client machine:

$rerun --version
rerun-cli 0.23.1 (default map_view nasm native_viewer release web_viewer) [rustc 1.84.0 (9fc6b4312 2025-01-07), LLVM 19.1.5] x86_64-unknown-linux-gnu release-0.23.1 87bdacb, built 2025-04-25T11:04:25Z
Video features: av1 default ffmpeg nasm serde

It seems that after the client connection was terminated mid process, the rerun viewer does not accept any new connections.

[Wumpf]

I can repro this specifically on Windows when opening Rerun from the same process as python (via spawn or init) but not when opening Rerun separately and then connecting. Which then would make this a duplicate of

• [Windows] Killing python script spawning Rerun Viewer also closes Rerun #6356

@dominikandreas Can you confirm that first starting Rerun separately (via just rerun) and then connecting to it doesn't exhibit this issue?

[dominikandreas]

Yes, I am running rerun separately and then connecting to it from another machine and consistently run into this issue every single time.

To add more detail to how I'm currently running things: I have a remote machine (linux) that logs the data and somehow needs to stream it to my local viewer (windows). Since the remote can't access the local machine directly (company network restrictions), I use ssh to remote forward the local port. But this breaks after a single run, actually independent of whether I kill the client or not.

I don't know what the role of the SSH remote port forwarding is, I assumed that this is not the source of error because the connection is stable and the connection to the viewer breaks consistently after each run.

I can only access the remote via ssh, http, or https. I wonder if there's another better way to what I need?

With version 0.22 it was possible to run a http proxy that routes traffic to websocket and webviewer over the same port. With version 0.23 that doesn't work anymore. Any alternative?

edit: I found a way to make this work that seems to be reliable:

• start the rerun grpc server on the remote
• use ssh to forward the grpc port to localhost
• start rerun with the localhost url as a parameter to connect to

But still I wonder why the reverse doesn't work, starting the grpc server locally together with the viewer and using remote port forwaring to stream data to it.