Support loading from zipped .wasm files

[NickCrews]

Possibly related:

• Cloudflare workers #430

I want to deploy evidence.dev on cloudflare pages or workers. When I build my evidence app, it outputs a static site. One of the largest files in the output is a duckdb.wasm, which evidence bundles for the browser to use. This is 34 MB. When I try to deploy this site to cloudflare pages or workers, I get an error, because cloudflare has a 25MB limit for static files. If you search the evidence.dev slack, you can find people coming up with all sorts of absurd workaround for this, like hosting the wasm file on s3 and then rewriting urls with sed.

If I zip the wasm file on my laptop, it is only 7MB. This makes me think it would be possible to:

• build and publish a zipped version of the wasm files, eg with the same filenames, but with a .wasm.zip extension
• in AsyncDuckDB.instantiate(mainModuleURL), accept a url that points to this zip file. I assume that all the platforms that the js runs on will have a zlib implementation to do the unzipping on the fly?

Then, user code that looks like

const DUCKDB_CONFIG = await duckdb.selectBundle({
            mvp: {
                mainModule: './duckdb-mvp.wasm',
                mainWorker: './duckdb-browser-mvp.worker.js',
            },
            ...
        });

could switch to

const DUCKDB_CONFIG = await duckdb.selectBundle({
            mvp: {
                mainModule: './duckdb-mvp.wasm.zip',
                mainWorker: './duckdb-browser-mvp.worker.js',
            },
			...
        });

and everything would work the same, but with less network load, and now duckdb-wasm apps can get deployed on cloudflare.

If you give me a rough outline of how this should be done, I can try to write a PR.

[domoritz]

Would it be possible to server the wasm file from a server with gzip/brotli compression which would be handled transparently by the browser? Or is that not enough compression?

[NickCrews]

Maybe I don't understand what you are suggesting. Here is my understanding. When I do wrangler deploy, it looks through the public/ folder for static assets. If any of them are >25MB, it fails. So I would need to pre-compress the .wasm using gzip, brotli, etc. So then there is this compressed wasm file sitting on my origin server. When a browser makes the request, it goes to cloudflare's CDN, which then queries the origin, and then pipes the response through, adding caching and compression. I am basing this off of https://blog.cloudflare.com/this-is-brotli-from-origin. I don't think there is a way to tell cloudflare "hey, this file is compressed at-rest, but when you serve it, please decompress it on the fly". Are you saying you think this is possible?

[davidgasquez]

hey, this file is compressed at-rest, but when you serve it, please decompress it on the fly

that would be the perfect solution, if is possible!

[carlopi]

I think it could make sense to add a flag about compression, or probably more general allow a custom callback to provide the actual payload.

I don't see this exactly end-to-end, but has to be viable.