Rust SDK panic on `internal error: entered unreachable code`

[AngelOnFira]

Continued from discussion on SpacetimeDB Discord

I'm running into an issue where this unreachable!() is being hit, which implies that the invariant here isn't being upheld. From some debugging on my end, I've discovered a few things:

• This seems to happen after a connection hiccup
• I can reproduce this consistently with the repro below
• I see this once every ~1h when I leave a single instance of my game running locally

Here is a repro where I see this behaviour within 5 seconds of starting the client. However, @gefjon hasn't been able to see this same behaviour. For context, I'm running this on a Hetzner server with an older i7 and 64GB ram, I can share more about this setup if needed.

I used this branch to get some extra logs from my repro. At the bottom of several log files is the line a delete update should correspond to an existing row in the table cache. You can view those here: Archive.zip

As an example:

`handle_delete` for table with row type repro_cache_bug_client::module_bindings::player_type::Player: a delete update should correspond to an existing row in the table cache, but the row WithBsatn { bsatn: b"\x03\xfc\xb2T4\xbd\xcb\xfc\x83r\xed\xd46=\x93\xa7\x8c\xaa;{\xc6 \xb7\xd0\xf2\xd3N\x8a\xaab\0\xc2\r\0\0\0Player-c20062\x04\0\0\0\0\0\0\0\xed\x03\0\0\0\0\0\0\x01", row: Player { identity: Identity(c20062aa8a4ed3f2d0b720c67b3baa8ca7933d36d4ed7283fccbbd3454b2fc03), name: "Player-c20062", x: 4, credits: 1005, online: true } } was not present
table contents:
	b"\x03\xfc\xb2T4\xbd\xcb\xfc\x83r\xed\xd46=\x93\xa7\x8c\xaa;{\xc6 \xb7\xd0\xf2\xd3N\x8a\xaab\0\xc2\r\0\0\0Player-c20062\0\0\0\0\0\0\0\0\xe8\x03\0\0\0\0\0\0\x01"
		Player { identity: Identity(c20062aa8a4ed3f2d0b720c67b3baa8ca7933d36d4ed7283fccbbd3454b2fc03), name: "Player-c20062", x: 0, credits: 1000, online: true }
		ref_count 1
	b"\x03\xfc\xb2T4\xbd\xcb\xfc\x83r\xed\xd46=\x93\xa7\x8c\xaa;{\xc6 \xb7\xd0\xf2\xd3N\x8a\xaab\0\xc2\r\0\0\0Player-c20062\x05\0\0\0\0\0\0\0\xed\x03\0\0\0\0\0\0\x01"
		Player { identity: Identity(c20062aa8a4ed3f2d0b720c67b3baa8ca7933d36d4ed7283fccbbd3454b2fc03), name: "Player-c20062", x: 5, credits: 1005, online: true }
		ref_count 1

There are still a few areas that I want to rule out from my own circumstances:

• I'm currently using the free tier on Maincloud, though I expect to upgrade rather soon, so I'll find out if there are issues there
• My application is using Ratatui, and I'm wondering if any potential issues with how the main application updates or sleeps could cause issues with the SDK (however, the repro isn't set up like this, so this might not matter)

Happy to help debug this more as needed. I'm not sure how much this will happen as I start running playtests, but I'll be collecting telemetry that will give me more data.