feat: patch @endo/bundle-source to esbuild if bundle is too big

[michaelfig]

closes: #12193

[Best reviewed commit-by-commit.]

Description

Patches @endo/bundle-source to automatically retry with esbuild minimisation when bundles exceed 500KB, helping to prevent RPC node max_body_bytes rejections until bundle chunking is available.

Patch implementation (.yarn/patches/@endo-bundle-source-npm-4.1.2-*.patch):

• Adds byteLimit option (default 500kB) and esbuildPath option
• On overflow, invokes esbuild --bundle --minify-whitespace and re-bundles the output
• Creates a temporary package.json if needed for @endo/compartment-mapper's module resolution

Prevent esbuild from bundling dynamic import expressions (~20 files):

• Pattern: const mod = '@pkg/name'; await import(mod) prevents esbuild from traversing @pkg/name
• Applied to: deploy scripts, test utilities, and dynamic imports that @endo/bundle-source already treats as external

Kernel bundle exemption:

• Set byteLimit: Infinity in SwingSet bundling paths (kernel bundles don't transit RPC)

Test fidelity:

• fakeVatAdmin.registerBundle() now generates bundleIDs containing the bundle hash
• Ensures test bundleIDs don't unnecessarily collide with different content for endoZipBase64 bundles

Security Considerations

Introduces esbuild as a dependency for minimisation. Only activated when standard Endo bundles exceed 500KB.

Scaling Considerations

Raises the effective contract size ceiling without requiring on-chain changes (even though this PR has kernel changes, they only need to be deployed during the next coordinated chain software update). Bundles under 500KB are unchanged; larger bundles trade legibility for deployability.

Documentation Considerations

Transparent to consumers—no API changes required. The intermediate-variable pattern for dynamic imports is fairly self-explanatory. It revealed a lenience that @endo/bundle-source has, but other conventional JS bundlers do not.

Testing Considerations

Existing tests pass. Enhanced bundleID generation improves test realism without causing behaviour regressions.

Upgrade Considerations

The patch itself is temporary and may be removable once bundle chunking ships. No migration needed—byteLimit is optional and backward-compatible.

[socket-security]

All alerts resolved. Learn more about Socket for GitHub.

This PR previously contained dependency changes with security issues that have been resolved, removed, or ignored.

View full report

[michaelfig]

@copilot please summarize this PR by writing a comment with the same format as the PR's current description.

[Copilot]

@michaelfig I've opened a new pull request, #12227, to work on those changes. Once the pull request is ready, I'll request review from you.

[Copilot]

Pull Request Overview

This PR patches @endo/bundle-source to automatically fall back to esbuild-based bundling when bundles exceed 500KB, addressing bundle size issues. The patch is applied via Yarn's patch mechanism and includes updates to the test infrastructure to properly handle bundle IDs that now incorporate hashes.

Key Changes

• Patches @endo/bundle-source to add esbuild fallback when bundles exceed 500KB (configurable via byteLimit option)
• Adds registerBundle and getBundleID helpers to fakeVatAdmin for test code that automatically append hash suffixes to bundle IDs for endoZipBase64 bundles
• Uses dynamic imports with avoidBundling variable pattern to prevent bundlers from including modules intended to be loaded dynamically

Reviewed Changes

Copilot reviewed 60 out of 63 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
.yarn/patches/@endo-bundle-source-npm-4.1.2-80da9522ea.patch	Core patch adding esbuild fallback logic and new options
package.json	Adds patch reference for @endo/bundle-source
yarn.lock	Updates lock file with patch reference and adds esbuild dependencies
packages/zoe/tools/fakeVatAdmin.js	Adds registerBundle and getBundleID helpers, improves bundle mismatch error messages
packages/zoe/test/**/*.test.js	Updates tests to use registerBundle instead of installBundle directly
packages/SwingSet/**/*.js	Adds byteLimit: Infinity to disable limits for kernel/system bundles
Various *.build.js files	Applies avoidBundling pattern to prevent bundling of dynamic imports
multichain-testing/ and a3p-integration/	Updates to use patched @endo/bundle-source

[turadg]

I'm surprised to see bundle-source patched instead of wrapped. We could have a local module that wraps it and use resolutions to wire it in wherever bundle-source is imported.

Since this is a stopgap I won't block on that.

But I do want to avoid the code changes to accomodate esbuild.

[turadg]

I suppose esbuild it bundling node-util when it sees the literal. But the code here shouldn't be concerned with that.

Instead please configure esbuild with the --external option to exclude what shouldn't be bundled.

[michaelfig]

Thanks for raising this.

I started with surgical use of --external, but the list was growing too large for my comfort, and it was fragile.

When digging a little deeper, I see that I can use (await import('node:module')).builtinModules and 'node:*' to catch 'em all.

[michaelfig]

Oops, now I hit a different hurdle that makes --external fail for a different reason. We need to package the esbuild output with the original @endo/bundle-source implementation, and that chokes on all the modules that esbuild has passed through.

It seems @endo/compartment-mapper hasn't sufficiently implemented externals for this use, but I'll ask @kriskowal if there's a way to make this work.

[michaelfig]

Still digging. I hope to have a solution by tomorrow.