Windows: declare UTF-8 through manifest

[scurest]

Fixes #61. Fixes #86.

Attaches a manifest to Windows .exes which makes char*s (narrow char strings) like argv be encoded in UTF-8, allowing tools to operate on files that can't be written in the system's narrow char encoding (eg. Japanese filenames on a Latin-1 system).

This is basically copy-pasted from the way libjxl/libavif does it.

All the tools are changed except xyz-thumbnailer (I didn't look into that one).

For lmu2png specifically, I believe it also fixes a bug where a narrow char path and a UTF-8 string are concatenated. This implicitly assumes the narrow char encoding is UTF-8. Previously, Japanese games wouldn't work on Windows even when the system narrow char encoding was Shift JIS.

More points:

• Will not work on Windows older than Windows 10 2019 update.
• I only tested lmu2png and png2xyz.
• This is cmake only, I have no idea how to update autotools. (Does autotools run on Windows?)
• As Ghabry says, mingw is not tested.
• The manifest files are located in resources/. Tell me if you want them somewhere else.

[Ghabry]

Jenkins: test this please

Thanks. Though we have no way to test the mingw path

[carstene1ns]

The relative paths to the manifest will break when you try to build a tool separate from e.g. archive checkout. For the "external" folder, I used symlinks, since they are supported on desktop platforms and git will make copies when archiving.

Do not worry too much about autotools, it will be removed in the future anyway. Mingw building is unsupported, so can be fixed when actually used.

[scurest]

The relative paths to the manifest will break when you try to build a tool separate from e.g. archive checkout.

Can you explain this in more detail please?

---

I do notice two problems. My intention was to give the path to the manifest relative to the Modules/ConfigureWindows.cmake file. But

1. Using CMAKE_CURRENT_LIST_DIR is wrong for this. What I want is CMAKE_CURRENT_FUNCTION_LIST_DIR, which gives the path relative to where the function is defined, not called. Since that requires CMake 3.17 and this repo uses 3.16, I can use the pattern with a global variable shown on that doc page instead. But if I do that...
2. Since Modules/ is included via symlink, the relative path (eg) lmu2png/CMakeModules/../resources/ leads to the non-existent lmu2png/resources/ instead of the desired Modules/../resources/. So I need to take the realpath to the ConfigureWindows.cmake dir too.

Putting those together, gives

get_filename_component(_CONFIGURE_WINDOWS_BASE_DIR "${CMAKE_CURRENT_LIST_DIR}" REALPATH)

function(target_use_utf8_codepage_on_windows target)
	set(RESOURCES_DIR "${_CONFIGURE_WINDOWS_BASE_DIR}/../resources")
	...

Does that look right?

[Ghabry]

He means that all files you access from within a tool must be inside the tool directory. So e.g. for xyz2png the resource must be somewhere in the xyz2png directory.

Iirc the command to create directory symlinks on Windows is mklink /D

So in every directory that references the resources you must do resources you must do mklink /D resources ..\resources. Then git add them.

And in ConfigureWindows change the target sources to resources/utf8.manifest.

---

In case this doesn't work because symlinks on Windows are a bit weird: I can add the commit for you.

[scurest]

He means that all files you access from within a tool must be inside the tool directory.

Why?

[Ghabry]

As c1 explained

The relative paths to the manifest will break when you try to build a tool separate from e.g. archive checkout.

The tool we use to create the final distribution archive resolves the symlinks and converts them back to "normal" files

[scurest]

I'm not sure why you use symlinks at all then but I guess it doesn't matter here.

So I should symlink resources->../resources in all the tool folders? Alternatively the utf8.manifest could be moved to Modules/. And do you want the .rc file if mingw isn't supported? (it's only used for that)

[Ghabry]

yeah pls symlink resources in all folders. You can keep the mingw rc file. It doesn't hurt even if untested :)

[scurest]

Will CI run automatically or...?

[scurest]

Thanks.

Okay, how does that look? I can confirm it still works for me. Can anyone else test it?

[fdelapena]

Sorry for the late reply, any suggested game file for testing this quickly?

[Ghabry]

You can use the Japanese version of Yume Nikki from here https://rmarchiv.de/games/1089

2007-10-01 Vollversion - 0.10

just e.g. pass a Japanese filename to xyz2png etc.

[fdelapena]

Tested latest pr build from https://ci.easyrpg.org/job/tools-win32-pr/211/

Tried xyzcrush on Windows 11 24H2. I guess globbing is broken (possibly unrelated different bug), but on a codepage 1252 is not showing characters at least. But converting with xyz2png works as expected and the output file now exists with same Japanese filename with extension .png.

lmu2png also shows garbled characters without setting encoding at least.
When adding -e 932, it works without errors and outputs Map0001.png as expected:

[fdelapena]

The visual glitch goes away with chcp 65001, but commands work as expected.

[scurest]

@fdelapena

Yes, that's all correct.

Globbing is the responsibility of each program on Windows, the shell doesn't do it. There's an .obj you can link to to supposedly do it for you though, /link setargv.obj. But yeah, that's a different issue.

[carstene1ns]

Looks good to me.