Ability to serialize/deserialize Schema on the server side

[jrabek]

Thanks again @endel for the great multiplayer game framework! If you would provide some basic guidance on this enhancement I would be happy to implement it in a PR.

Summary

This is a request for a Schema method for completely serializing/deserializing the server Schema (i.e. game state) so that it can be persisted in a database.

Motivation

In the current colyseus server architecture, the game state (i.e. some Schema derived class) persists in memory in one specific server.

One of the shortcomings is that it prevents seamless deploys and crash recovery since the game state is in memory and therefore lost when the server process is killed. For certain game types (e.g. long running games, games involving money) it is critical that game state is not lost.

To support a more robust game state, the ability to fully serialize the server Schema in a way that it could be stored in a database is needed.

Currently there are two ways to serialize the server Schema neither of which are meant for deserializing it later.

1. toJSON: This method is meant for logging / debugging
2. encodeAll / decode: These are meant to send state updates to the clients and not for recreating the entire server Schema.

For (2) this is what happens when the server state is serialized/deserialized using encodeAll / decode: https://gist.github.com/jrabek/6303d3f94b82c692de7721f0fc331f8c.

[jrabek]

Some related discord discussions:

Original conversation: https://discord.com/channels/525739117951320081/526083188108296202/880867521824170005

Follow up when encodeAll / decode wasn't working: https://discord.com/channels/525739117951320081/526083188108296202/889878308983046215.

[Wenish]

as a workaround:

in the onCreate function of the room call a loadState function which gets the room state from a database and maps it to the state (mapping has to be manual)

and in the onDispose function of the room call a saveState function which save the room in a database (mapping has to be manual)

at least thats how i do it :)

and maybe its worth for your game to auto save the state every 5 minutes in case server crashes. so only 5 minutes of game gets lost.

[jrabek]

@Wenish so I am not using the rest of colyseus, just the schema. Also I don't think what you are proposing will work because not all the internal values are saved (most importantly the ids of the fields meaning the diff that gets sent after reloading the data breaks). Check out the gist I had sent before: https://gist.github.com/jrabek/6303d3f94b82c692de7721f0fc331f8c

[Wenish]

@jrabek im talking about a workaround where you write your own mapper from the schema to your database model and visa versa. not using any built in function you try to use.
and it does work. i load hole maps from a json file

[lpsandaruwan]

Hi @jrabek

Still investigating this issue. Hopefully plan to fix this in future. You will be able to use this workaround for now.

 // Simulate a serialization to a database
  if (doDecode) {
    state = new State();
    state.decode(encodedState);
    state = state.clone();
  }

cc @endel

[jrabek]

Thanks for the update @lpsandaruwan! Definitely would love to try out colyseus again!

I must have missed clone before. Was that added more recently?

With this approach will the cloned state work with per client filters?

[lpsandaruwan]

Hi @jrabek
Cloning the state and reassigning, was a shot in the dark :-D. It resolved the undefined property exception when decoding state back.
It should work without any interruption. I'm still testing client's behavior against this. But if you have any inputs/help to offer on this please feel free to mention.

cc @endel

[jrabek]

@lpsandaruwan I had a chance to play with your suggestion and while I don't see the exception I also don't see the correct behavior when decoding from an encoded state and then cloning.

You can check out my experiment here https://github.com/jrabek/colyseus-schema-experiments/tree/main. I put a few notes in the README.md.

[lpsandaruwan]

Thank you @jrabek. Will look into this.

[lpsandaruwan]

Hi @jrabek
From your provided sample, I think after cloning the schema you might have to set the schema inside the serializer object again since the serializer object is referenced to the old schema object, not the cloned one. Please let us know if that worked for you. In, https://github.com/jrabek/colyseus-schema-experiments/blob/def4ed67724ce2da52e5bd6491fb4a7fc41136fd/server.ts#L23

    if (this.doDecode) {
      this.state = new GameState();
      this.state.decode(encodedState);
      this.state = this.state.clone();
      this.serializer = new Serializer(this.state);
    }

[jrabek]

Thanks for the follow-up! I'll check when I have a second this week.