Thoughts on a solid-based game engine

[viridia]

For the last several months, I've been trying to decide whether to migrate my game engine (currently at around 70k lines of code) to solid.js. This post is a list of some of the pros and cons I am weighing in this decision. Those who only care about using Solid for building websites should probably stop reading now :)

First some background: the engine itself is based on three.js and rapier physics. For purposes of this discussion, the engine consists of three parts:

• The simulator / renderer, which is written on top of a reactive framework that is similar to solid.js.
• The game HUD, which is written in React (mainly because I didn't know about Solid when I started this project 3 years ago).
• The Editor / IDE, which is also written in React and which embeds a live copy of the game as a preview window - also written in React.

The engine also has a suite of separate offline editing tools such as the texture generator (Vortex), the tree grown simulator (Seedling), and the model animation previewer (Catwalk). All of these apps use the "real" Solid.js. The game is not open source yet (because the source code contains many spoilers) but will be eventually once it is complete.

All of the internal data structures of the engine are both reactive and reflectable - this includes Actors, Buildings, Terrain, Quests, Dialogues, ParticleEmitters, LootTables, CutScenes, Physics Colliders, and many more.

The original design was based on something like MobX observables, but after reading Ryan's articles on how to construct a reactive framework about a year ago, I build my own "solid-like" framework and migrated to it. This framework is 90% the same as "real" Solid.js, but it is neither a strict subset or superset.

Reactivity is ubiquitous throughout the engine, but it shows up most prominently in two areas: character reactions and property editing.

Characters have hierarchical goals which compete for priority - when a character is attacked, for example, the defense goals take priority over other goals such as eating and sleeping. All of these goals are reactive, driven by signals that are properties of the various objects in the world. For example, the character's inventory is a reactive store, and when a quest item is added to the inventory, this can trigger a reaction which advances the state of the associated quest.

Here's an example of a goal for an NPC inkeeper working in the kitchen - this goal is a "component" which represents one small facet of that character's behavior. This involves (a) walking over to the work table (scenery interaction), (b) looking in the direction of the table's 'use' mark, (c) equipping the kitchen knife, (d) animating a chopping motion with the knife for 4 seconds, and finally (e) putting the knife away:

  return G(SceneryInteraction, {
    ...props,
    target: fixtureInstance,
    children: (position: Vector3) =>
      G(Contingent, {}, [
        G(LookAt, { target: position }),
        G(Equip, { archetype: 'archetypes/weapons:KitchenKnife', instant: true }),
        G(Pose, {
          delay: 4,
          animation: {
            type: 'stand',
            clipNames: ['KitchenWork', 'Stand'],
            repeat: true,
          },
        }),
        G(Unequip, { instant: true }),
      ]),
  });

The function G() can be thought of as the equivalent of JSX.createElement() - it creates an instance of a component with reactive children. A 'contingent' goal is a control-flow component that provides backtracking - if any goal fails, it can revert back to previous goals in the sequence, or after a number of retries, bounce the failure higher up the goal stack.

Game objects are edited in the editor using a reflective, type-driven property sheet editor, which can be seen in the left side of the screenshot below:

Most game objects are JavaScript classes which support prototype-based inheritance using Archetypes. Both instances and archetypes have editable properties which are created using Object.defineProperty(), each backed by a signal instance. In addition, each property has reflective type information (stored as an attribute on the getter), so the property grid knows what type of editor widget to create based on the property type. Note: I could have used decorators for this, however given the current state of decorators is in flux, I decided to go with a more pedestrian makeObservable(instance, propertyDefs). Still, decorators would be nice to have at some point.

The reflection metadata is also used for serialization. A given game level or archetype is saved as a JSON file (or for very large assets, a .msgpack file). There is also a complete set of json-schema files for all asset types, which helps when debugging the raw JSON.

So, with that, what are the advantages of migrating to "real" Solid.js as opposed to my home-grown "solid-like" framework?

Pros:

• The real solid framework has a large user base and has been thoroughly battle-tested, whereas my own framework probably has bugs. I'm particularly lacking confidence in the code that bridges between the reactive world and the React world.
• Performance. The UI of this app is extremely dynamic, and a virtual DOM is not needed. Note however, that the UI is not really on the critical path - I get 120FPS on my MacBook in most cases, unless a lot of characters are on the screen.

Cons:

• There are a number of features in my framework which Solid doesn't have, and some of them are primitives which are not easy to build in "real" solid.
• The migration is an enormous amount of work, involving several thousand source files. It would probably take several months.
• I would also need to get rid of any other packages that depend on React - in particular, migrating all components from emotion to vanilla-extract.

One problematic area has to do with how I use contexts - for example, when an actor is evaluating their goals, the goal code can call useContext() to get access to the character that the goal is attached to, as well as other contextual information about the local environment. Solid.js only supports setting contexts within a JSX component, and JSX isn't used in the goal system. (Actually, an earlier version did use JSX to define nested goals, but using custom JSX factories in Vite simply proved to be too unwieldy. So now JSX is only used for the game HUD and editor, there is no longer any JSX in the simulator.)

There are a few other primitives in my system that could be implemented on top of Solid:

• createWhen(condition, action) - waits until the condition is true, then runs the action.
• createMutex() - this is used to prevent multiple characters from speaking at the same time. createMutex() returns a function which returns a lock object that is initially unlocked but reacts when the lock is acquired.
• setDebugName() associates a text name with the current reactive scope, which is useful when debugging.
• createQuery() is an extremely minimalist cousin of tanstack query, used in cases where an actor goal requires an asyncrhonous resource such as a list of navigation waypoints. It's designed to work in non-JSX contexts.
• waitFor(condition) transforms a reactive condition into a promise. (I think solid-primitives has a version of this.)
• createAtom() is similar to the MobX function of the same name, and is used to add reactivity to large complex data structures that don't need fine-grained reactivity. It's like a signal that always reacts when written. So for example, there's an atom associated with the list of Biome types, something that only changes in the editor, and only rarely, so a full re-render of the world is acceptable even if the smallest change is made.

In some cases a reaction needs to be deferred until the next requestAnimationFrame, because the reaction needs to know the time-delta in order to know how to scale the effect. At the moment this works simply by having the reaction set a 'needsUpdate' flag, and then processing the side-effects in the next RAF, but I suspect that this could be improved.

Procedural generation: the game engine procedurally generates certain resources at runtime, such as navigation maps. Because the geometry calculation are expensive, it's done in a web worker; heavyweight calculations like this require extraordinary levels of batching, so to avoid redundant calculations the engine has a framework for managing these that is similar to "make" - that is, classes that represent 'build targets' which have explicit dependencies and version numbers, and which react when a dependent's version number changes, but only when all dependents are in a "ready" state.