Slow Buffer.from when generating large web pages

[D-Sketon]

Check List

Feature Request

hexo/lib/hexo/router.ts

Line 33 in c69553c

return Buffer.from(data); // Assume string is UTF-8 encoded string

When the generated web page has a large size, this becomes a performance bottleneck.

I don't quite understand why we need to convert the string to a buffer and then write it back to the file. Is it possible to skip the process of converting to a buffer?

[SukkaW]

I don't quite understand why we need to convert the string to a buffer and then write it back to the file. Is it possible to skip the process of converting to a buffer?

Might have something to do with Readable? Have you tested new TextEncoder().encode?

TBH, I am more than happy to drop the entire Buffer from Hexo. I myself have been going full Uint8Arrays for quite a long time now.

[D-Sketon]

I conducted a simple test and found that there is little difference in speed between Buffer.from and new TextEncoder().encode.

[SukkaW]

I don't quite understand why we need to convert the string to a buffer and then write it back to the file. Is it possible to skip the process of converting to a buffer?

So I just dug around. The Buffer.from is used by this RouteStream:

hexo/lib/hexo/router.ts

Line 11 in c69553c

class RouteStream extends Readable {

This RouteStream is only used by Route.prototype.get:

hexo/lib/hexo/router.ts

Line 118 in c69553c

get(path: string): RouteStream {

And Route.prototype.get is only used by Generator.prototype.writeFile:

hexo/lib/plugins/console/generate.ts

Lines 75 to 79 in c69553c

	writeFile(path: string, force?: boolean): Promise<boolean> {
	const { route, log } = this.context;
	const publicDir = this.context.public_dir;
	const Cache = this.context.model('Cache');
	const dataStream = this.wrapDataStream(route.get(path));

So before we write a file, we actually do some hashing here, assuming we are comparing the SHA1 hash to avoid writing the same file twice (if it is already written in the previous hexo g). So if you want to skip Buffer.from, you will need to make sure both SHA1 hash and Buffer.concat support string.