Restore the ability to run 32-bit level scripts

[Lgt2x]

This has been mentioned multiple times, so here is a master issue to talk about it and suggest possible solutions.

The problem is the following: Descent 3 relies on pre-compiled scripts inside of levels to run the logic: open doors, pick-up objects, etc. For a long time now, we're compiling the main campaign's level scripts and put them in an OS-dependent file: d3-linux.hog/d3-osx.hog/d3-win.hog. This means that level scripts should be compiled again when the game ABI is updated.

Outside of the main campaigns, many user levels have been created over the years using the Descent 3 level editor, generating level scripts binary blobs that are now incompatible with the current version of the game. Especially, the game is now compiled exclusively to 64-bit architectures, instead of the original 32-bit it originally ran on. This means that user-created level scripts that have not been compiled recently against the 64-bit Descent 3 game cannot be played at all.

So far, it has been proposed to use a virtual machine able to run 32-bit black-box scripts/blobs, in particular using the Box86 or loadlibrary projects, but no proof of concept has been made. More investigation is needed to find the best solution we can get to get old user-created levels to run again.

Related to #583 among others

[JeodC]

Since you have the osiris source code, even from the 32bit era, is it possible to create a conversion tool to decompile and recompile scripts? If so, I would keep such a thing separate from the game itself, so external websites can perform the conversions and host levels, and the game will not be bogged down with the process during runtime.

[ccfly42]

The best option is of course to load the 32-bit versions.
Whether a web service or the level designers themselves
update their scripts to 64-bit, we should set conditions for this so that older versions (32-bit) are not affected.
We should make sure that the 64-bit script files do not replace the 32-bit files,
otherwise we will create confusion and incompatibilities.

I don't know exactly whether the script file is loaded via the .d3l file name or whether the
script file name is entered into the .d3l file. If the former is the case,
a postfix "e.g. {filename}_x64.dll/.so" should be used.

Until the problem is fixed, I recommend making a 32-bit version available on a regular basis.
According to #631 , this seems to work. On the one hand, singleplayer players could play their levels again
until the problem is fixed, and on the other hand, server administrators could specifically use 32-bit versions
for multiplayer levels that contain 32-bit scripts.

[SiriusTR]

I would suggest that the solution involve sandboxing/virtual machines, for security reasons - honestly even third-party 64-bit scripts should use this. The current implementation is a giant remote code execution vulnerability.

[bryanperris]

Since you have the osiris source code, even from the 32bit era, is it possible to create a conversion tool to decompile and recompile scripts? If so, I would keep such a thing separate from the game itself, so external websites can perform the conversions and host levels, and the game will not be bogged down with the process during runtime.

Sure it is possible, but we don't need to boil the whole ocean here. Going forward D3 needs a modern and practical approach to running level logic, Lua is already an easy fit for that. As for external MP maps, that is an entire another giant beast to deal with. A server should only provide Lua level scripts, nothing else -- no remote code execution. Even for Lua scripts, things should be restricted even there too. One could ask the user to permit executing some classic DLLs in a wine based sandbox, but then even that is going to be a pain to maintain when people play D3 on ARM64 or other CPU arches. I don't even like to call the precompiled level scripts scripts, they aren't scripts to begin with, its just level logic. D3 was going to use a real scripting engine but it was scrapped.

The best option is of course to load the 32-bit versions. Whether a web service or the level designers themselves update their scripts to 64-bit, we should set conditions for this so that older versions (32-bit) are not affected. We should make sure that the 64-bit script files do not replace the 32-bit files, otherwise we will create confusion and incompatibilities.

I don't know exactly whether the script file is loaded via the .d3l file name or whether the script file name is entered into the .d3l file. If the former is the case, a postfix "e.g. {filename}_x64.dll/.so" should be used.

Until the problem is fixed, I recommend making a 32-bit version available on a regular basis. According to #631 , this seems to work. On the one hand, singleplayer players could play their levels again until the problem is fixed, and on the other hand, server administrators could specifically use 32-bit versions for multiplayer levels that contain 32-bit scripts.

Since this open source project is not using some modern tool such as Unreal Engine and no one is getting paid, Lua is the best way to go for now and yeah level designers will have to do their stuff in that scripting language or the map editor will have to be updated to generate Lua based on the visual editor.