[ Intl ] Fix Intl feature inside Blueprints in Playground

[mho22]

Motivation for the change, related issues

Based on this issue.

Related to this function.

It looks like the ICU data is not found when running the below Blueprint. Even if the file is located in /internal/shared before running the playground.run(...) in run-php.ts.

Implementation details

• Added a new Intl classes should work when intl is enabled test in playground/website/playwright/e2e/blueprints.spec.ts

Blueprint

{
	"features": {
		"intl": true
	},
	"steps": [
		{
			"step": "runPHP",
			"code": "<?php \n\n$data = array(\n    'F' => 'Foo',\n    'Br' => 'Bar',\n    'Bz' => 'Bz',\n);\n\n$collator = new Collator('en_US');\n$collator->asort($data, Collator::SORT_STRING);\n\nthrow new Exception( json_encode($data, JSON_PRETTY_PRINT ) );"
		}
	]
}

Should throw :

Uncaught Exception: {
    "Br": "Bar",
    "Bz": "Bz",
    "F": "Foo"
} 

But throws :

Uncaught IntlException: Constructor failed

[mho22]

I found the root cause. The above blueprint will run successfully if I comment the proxyFileSystem(...) function in playground-worker-endpoint.ts :

if (!isPrimary) {
	const pathsToShareBetweenPhpInstances = [
		'/tmp',
		requestHandler.documentRoot,
		'/internal/shared',
		'/internal/symlinks',
	];
	const pathsToProxy = pathsToShareBetweenPhpInstances.filter(
		(path) => !isPathToSharedFS(php, path)
	);

	// TODO: Document that this shift is a breaking change.
	// Proxy the filesystem for all secondary PHP instances to
	// the primary one.
	proxyFileSystem(
		await requestHandler.getPrimaryPhp(),
		php,
		pathsToProxy
	);
}

This means something is happening to the /internal/shared/icudt74l.dat file during the proxy.

If I add these lines in the proxyFileSystem(...) function :

+ if( path !== '/internal/shared' ){
	// @ts-ignore
	replica[__private__symbol].FS.mount(
		// @ts-ignore
		replica[__private__symbol].PROXYFS,
		{
			root: path,
			// @ts-ignore
			fs: sourceOfTruth[__private__symbol].FS,
		},
		path
	);
+ }

It runs correctly. I wonder if mounting a file used by PHP internals like the intl extension will corrupt it.

[mho22]

@adamziel I created a new directory named /internal/private which will be used to store everything related to dynamic extensions. This directory shouldn't be proxied between PHP instances since there seems to be file corruption with the icu.dat file when it is done in /internal/shared.

I tagged you because I wanted to know if /internal/private is a meaningful path to you.

[mho22]

Maybe /internal/libs ?

[adamziel]

Hm, not mounting that file worries me. First, what corrupts that file? Could it affect other files and corrupt other data? Second, the system is wired for sharing files between php instances and I worry about the consequences of not sharing them.

The reason we have these mounts is to share a set of files between all the php instances - they're created once in the primary env and then every other php instance sources them from that primary php's filesystem. If we stop mounting, what would happen to icu.dat in non-primary php instances or after the php env is rotated?

[mho22]

@adamziel Here’s my current understanding of why we were still hitting the Intl issue and how it relates to filesystem sharing.

Each PHP instance is initialized via loadWebRuntime. During that phase, the instance loads and links against its intl.so extension and the corresponding icu.dat file [ which is fetched and memoized at runtime ]. Those links are established during initialization, before any filesystem proxying happens.

Later on, when proxyFileSystem is called, we mount /internal/shared from the primary PHP instance onto the secondary ones using FS.mount. Although the icu.dat file is still visible after the mount, this operation effectively replaces the underlying filesystem for that path. As a result, the ICU runtime inside the PHP instance can no longer use the icu.dat file it was originally linked to during loadWebRuntime, and Intl starts failing [ e.g. Collator constructor errors ].

Even though I can’t say exactly what happens internally inside FS.mount, the observed behavior is consistent : after proxying /internal/shared, PHP’s intl extension can no longer access ICU data reliably. This strongly suggests that remounting that directory breaks the early filesystem bindings created during PHP initialization.

To fix this, my proposal is to isolate native extensions .so and their required runtime assets [ like icu.dat ] into a dedicated, instance-local directory, such as : /internal/private, /internal/libraries or /internal/libs

Each PHP instance would load its extensions from this directory during startup, and that directory would never be proxied or remounted. This keeps ICU and other native extensions stable and instance-safe.

In contrast, /internal/shared should only be used for files that are not tightly coupled to a specific PHP instance and are safe to share across instances.

What do you think ?

[akirk]

@mho22 Were you able to verify your theory by doing a checksum of the file with and without the mounting?

[mho22]

@akirk The checksum is always the same. I was wrong about data corruption. But I think I found out what was the issue :

This is the proxy-file-system.ts proxyFileSystem modified function :

export async function proxyFileSystem(
	sourceOfTruth: PHP,
	replica: PHP,
	paths: string[]
) {
	const __private__symbol = Object.getOwnPropertySymbols(sourceOfTruth)[0];
	for (const path of paths) {
		if (!replica.fileExists(path)) {
			replica.mkdir(path);
		}
		if (!sourceOfTruth.fileExists(path)) {
			sourceOfTruth.mkdir(path);
		}


+		async function checksumSHA256(data: Uint8Array): Promise<string>
+		{
+			const hashBuffer = await crypto.subtle.digest( 'SHA-256', data as BufferSource );
+
+			const hashArray = Array.from(new Uint8Array(hashBuffer));
+
+			return hashArray.map(b => b.toString(16).padStart(2, '0')).join('');
+		}


+		if( replica.fileExists( '/internal/shared' ) ) console.log( 'BEFORE', await checksumSHA256( replica.readFileAsBuffer( '/internal/shared/icudt74l.dat' ) ) );

		// @ts-ignore
		replica[__private__symbol].FS.mount(
			// @ts-ignore
			replica[__private__symbol].PROXYFS,
			{
				root: path,
				// @ts-ignore
				fs: sourceOfTruth[__private__symbol].FS,
			},
			path
		);

+		if( replica.fileExists( '/internal/shared' ) ) console.log( 'AFTER ', await checksumSHA256( replica.readFileAsBuffer( '/internal/shared/icudt74l.dat' ) ) );
	}
}

These are the logs when the icudt74l.dat file is in the /internal/shared/ directory.

BEFORE dc778b9ffe18ed319ad3fb70754f80e51cf7b6dbfff38fc0c0a5f27bb5463dad
AFTER  dc778b9ffe18ed319ad3fb70754f80e51cf7b6dbfff38fc0c0a5f27bb5463dad
BEFORE dc778b9ffe18ed319ad3fb70754f80e51cf7b6dbfff38fc0c0a5f27bb5463dad
AFTER  dc778b9ffe18ed319ad3fb70754f80e51cf7b6dbfff38fc0c0a5f27bb5463dad
BEFORE dc778b9ffe18ed319ad3fb70754f80e51cf7b6dbfff38fc0c0a5f27bb5463dad
AFTER  dc778b9ffe18ed319ad3fb70754f80e51cf7b6dbfff38fc0c0a5f27bb5463dad
BEFORE dc778b9ffe18ed319ad3fb70754f80e51cf7b6dbfff38fc0c0a5f27bb5463dad
AFTER  dc778b9ffe18ed319ad3fb70754f80e51cf7b6dbfff38fc0c0a5f27bb5463dad

PHP Fatal error:  Uncaught IntlException: Constructor failed in /internal/eval.php:5
Stack trace:
#0 /internal/eval.php(5): Collator->__construct('en_US')
#1 {main}
  thrown in /internal/eval.php on line 5

And these are the logs when the icudt74l.dat file is in the /internal/private/ directory. [ Not proxied ].

BEFORE dc778b9ffe18ed319ad3fb70754f80e51cf7b6dbfff38fc0c0a5f27bb5463dad
AFTER  dc778b9ffe18ed319ad3fb70754f80e51cf7b6dbfff38fc0c0a5f27bb5463dad
BEFORE dc778b9ffe18ed319ad3fb70754f80e51cf7b6dbfff38fc0c0a5f27bb5463dad
AFTER  dc778b9ffe18ed319ad3fb70754f80e51cf7b6dbfff38fc0c0a5f27bb5463dad
BEFORE dc778b9ffe18ed319ad3fb70754f80e51cf7b6dbfff38fc0c0a5f27bb5463dad
AFTER  dc778b9ffe18ed319ad3fb70754f80e51cf7b6dbfff38fc0c0a5f27bb5463dad
BEFORE dc778b9ffe18ed319ad3fb70754f80e51cf7b6dbfff38fc0c0a5f27bb5463dad
AFTER  dc778b9ffe18ed319ad3fb70754f80e51cf7b6dbfff38fc0c0a5f27bb5463dad

There is no Uncaught IntlException anymore. This is certainly not enough to verify my theory. So this time I focused on where and how that icu file was reached and it looks like, when I replace the logs with this in the proxy-file-system.ts file :

// @ts-ignore
const FS1 = replica[__private__symbol].FS;
const node1 = FS1.lookupPath('/internal/shared/icudt74l.dat').node;

console.log('is MEMFS?', node1.mount.type === FS1.filesystems.MEMFS);
console.log('is PROXYFS?', node1.mount.type === FS1.filesystems.PROXYFS);

I'll get these logs :

is MEMFS? true
is PROXYFS? false
is MEMFS? true
is PROXYFS? false
is MEMFS? false
is PROXYFS? true
is MEMFS? false
is PROXYFS? true

So at one point in time, we reach data from PROXYFS instead of MEMFS and, based on what GPT says, this could be the reason why Intl crashes.

MEMFS owns the bytes; PROXYFS forwards the bytes.
Intl (ICU) needs ownership, not forwarding.

[adamziel]

This Blueprint:

<?php 

require_once '/wordpress/wp-load.php';

try {
  $collator = new Collator( get_locale() );
} catch(Exception $e) {
  var_dump([
     'msg' => intl_get_error_message(),
     'code' => intl_get_error_code(),
  ]);
  throw $e;
}

Gave me some more info:

array(2) {
["msg"]=> string(65) "collator_create: unable to open ICU collator: U_FILE_ACCESS_ERROR"
["code"]=> int(4)
}

It seems like something fundamental with file access is broken in PROXYFS. I'd rather not work around that but pinpoint the exact problem. I've built a bunch of tests to see how PROXYFS deals with different FS operations and found that this test fails with errno 8:

beforeEach(async () => {
	phpA = new PHP(await loadNodeRuntime(phpVersion));
	phpB = new PHP(await loadNodeRuntime(phpVersion));

	// Mount /internal/shared from phpA to phpB using PROXYFS
	proxyFileSystem(phpA, phpB, [sharedPath]);
});

it('should open a file in read-write mode (O_RDWR)', () => {
	const testFile = `${sharedPath}/rdwr-test.txt`;
	phpA.writeFile(testFile, 'initial content');

	const fsB = getFS(phpB);
	const stream = fsB.open(testFile, O_RDWR);

	try {
		// Read from the file
		const readBuffer = new Uint8Array(8);
		fsB.read(stream, readBuffer, 0, 8, 0);
		expect(new TextDecoder().decode(readBuffer)).toBe('initial ');

		// Write to the file
		const writeContent = 'new';
		const writeBuffer = new TextEncoder().encode(writeContent);
		fsB.write(stream, writeBuffer, 0, writeBuffer.length, 8); // Write after "initial "

		fsB.close(stream);

		expect(phpA.readFileAsText(testFile)).toBe('initial newnt'); // 'content' becomes 'newnt'
	} finally {
		fsB.close(stream);
	}
});

There could be more problems in there.

[adamziel]

This one also seems to be failing:

it('should change file access and modification times through PROXYFS', () => {
	const testFile = `${sharedPath}/utime-test.txt`;
	phpA.writeFile(testFile, 'utime test');

	const fsB = getFS(phpB);
	const oldStat = fsB.stat(testFile);

	// Set new access and modification times (e.g., 1 day later)
	const newAtime = (oldStat.atime.getTime() / 1000) + 86400;
	const newMtime = (oldStat.mtime.getTime() / 1000) + 86400;

	fsB.utime(testFile, newAtime, newMtime);

	const newStat = fsB.stat(testFile);
	// Emscripten's FS.utime expects seconds, but stat returns milliseconds.
	// Compare with a small tolerance due to potential floating point inaccuracies.
	expect(newStat.atime.getTime() / 1000).toBeCloseTo(newAtime);
	expect(newStat.mtime.getTime() / 1000).toBeCloseTo(newMtime);
});