backend/rs: Make sure max message size is handled like `libwayland`, with tests

[ids1024]

I noticed this when working on #851.