Loading 100s of workers?

[tom-sherman]

I've tried to load hundreds of relatively large workers, and I get an OOM exception when starting workerd:

> workerd serve --verbose --experimental app.capnp


<--- Last few GCs --->


<--- JS stacktrace --->


#
# Fatal javascript OOM in Executable MemoryChunk allocation failed during deserialization.
#

The generated capnp binary config is around 2.5GB, so I assume this OOM is coming from V8 itself.

Could workerd load this whole config into memory, and then manage creating and destroying v8 isolates for me? This seems like a decent solution to this kind of scaling problem without adding multithreading to workerd itself.

Without this built in support, I'm going to need to spread my workers over X number of workerd processes or use some heuristic to restart workerd with only active workers.

[kentonv]

Hmm. By default, V8 puts all isolates in the process into a single "pointer cage" of 4GB, which makes it hard to load a huge number of isolates in one process. This might be what you're hitting here. V8 can be configured (at build time) to not use pointer cages. However, this requires turning off pointer compression as well, so overall memory usage will be higher even for a single isolate. But maybe we should flip that flag for workerd?

In our edge runtime we actually use an unsupported middle-ground configuration in which we ask V8 to create a separate pointer cage for each isolate. For workerd I wanted to avoid using a V8 configuration that the V8 team isn't committed to supporting. We will probably end up working with the V8 team to find a compromise that we can upstream later on, to get ourselves out of unsupported configuration hell, and then hopefully our solution there can apply to workerd as well. But in the meantime, maybe we should simply disable pointer compression for workerd?

[tom-sherman]

This sounds ok to me. It'd make the memory use more intuitive as there would be no hidden limit within workerd. Not ideal that memory usage would go up, but that can be solved separately as you mention.

What are you thoughts on adding LRU isolate eviction to workerd?

[kentonv]

So, this may be a bit disappointing, but I'll be honest... LRU eviction is something we aren't planning to implement in workerd, and the reason is annoying business reasons rather than technical ones. In order to be able to open source this code at all, we had to make sure it wasn't something that a competitor could easily just take and use to run their competing edge compute platform. To that end we focused on making workerd useful for people trying to self-host their own applications and for people trying to test code locally, but not for people trying to build a hosting platform for third-party code. LRU eviction of isolates was something we decided was a key capability for edge compute and hosting platforms, but usually not needed for the self-hosting case, so we decided to keep it out. (Plus, to be honest, our code for this is horridly complex and tuned for our environment... I'm not sure it would make sense for most users.)

With that said, it is totally possible to implement your own LRU eviction on top of this codebase -- that's exactly what we do, after all. The general idea here would be to fork server.c++ and modify it to do what you want. The rest of workerd essentially becomes a library that you depend on. However, I do have to caution that we don't guarantee API stability for that "library", so you would have to either pin to a particular version of the workerd code or be ready to update your code periodically to account for changes in our APIs.

[gerrywastaken]

Man, I was thinking that workerd would be an efficient alternative to enforcing containers in a CI/CD system where you end up with similar (but non competing) needs to what is needed for an edge compute platform. It is frustrating that this part hasn't been included, but also very understandable. I appreciate your and Cloudflare's openness in regard to this and all the work that has been shared.