Implement in-memory worker file system

So this is a fun one. In preparation for the
implementation of node:fs (and eventually also the
Web file system API), we need to implement a
virtual file system for the worker.

There's quite a bit to unpack here so let's go
through it a bit.

First, let's talk about the file system itself.
Every worker instance will have its own root
directory (`/`). In this root directory we will
have at least two special directories, the
"bundle" root and the "temp" root.

* The bundle root is where all of the modules that
  are included in the worker bundle will be
  accessible. Everything in this directory will be
  strictly read-only. The contents are populated
  by the worker configuration bundle. By default,
  the bundle root will be `/bundle` but this can
  be overridden.

* The temp root is where we will allow temporary
  files to be created. Everything in this
  directory is read-write but will be transient.
  By default, the temp root will be `/tmp` but
  this can also be overridden.

Let's imagine the following simple workerd
configuration:

```
const helloWorld :Workerd.Worker = (
  modules = [
    (name = "worker",
     esModule = embed "worker.js"),
    (name = "foo", text = "Hello World!"),
  ],
  compatibilityDate = "2023-02-28",
);
```

Given this configuration, the worker fs will
initially have the following structure:

```
/
├── bundle
│   ├── worker
│   └── foo
└── tmp
```

We can then use the `jsg::Lock&` to access the
file system at the C++ level. We reuse the
existing `kj::Filesystem` API to provide the
interface.

```cpp
jsg::Lock& js = ...
// Resolve the root of the file system
KJ_IF_SOME(node,
           js.resolveVfsNode("file:///"_url)) {
  auto& dir =
    kj::downcast<const kj::ReadableDirectory>(
      node);
  // List the contents of the directory
  KJ_DBG(dir.listNames());
}

KJ_IF_SOME(node, js.resolveVfsNode(
    "file:///bundle/worker"_url)) {
  auto& file =
    kj::downcast<const kj::ReadableFile>(node);
  KJ_DBG(file.readAllText());
}

KJ_IF_SOME(node,
    js.resolveVfsNode("file:///tmp"_url)) {
  auto& dir =
    kj::downcast<const kj::Directory>(node);
  auto tmpFile = dir.createTemporary();
  // tmpFile is an anonymous temporary file
}
```

The temporary file directory is a bit special in
that the contents are fully transient based on
whether there is or is not an active IoContext.
We use a special RAII `TempDirStoreScope` scope to
manage the contents of the temporary directory.
For example,

```cpp
KJ_IF_SOME(node,
    js.resolveVfsNode("file:///tmp"_url)) {
  auto& dir =
    kj::downcast<const kj::Directory>(node);
  kj::Path path("a/b/c/foo.txt")
  {
    TempDirStoreScope temp_dir_scope;
    auto tmpFile = dir.openFile(path,
        kj::FileMode::CREATE);
    KJ_ASSERT(tmpFile.write(0,
        "Hello World!"_kjb) == 12);
    KJ_DBG(dir.exists(path)); // true!
  }
  // The temp dir scope is destructed and the file
  // is deleted
  KJ_DBG(dir.exists(path)); // false!
}
```

However, if there is an active IoContext, the
temporary file will instead be created within
that IoContext's TempDirStoreScope, and will be
deleted when the IoContext is destructed. This
allows us to have a single virtual file system
whose temporary directories are either deleted
immediately as soon as the execution scope is
exited, or are specific to the IoContext and are
deleted when the IoContext is destructed. This
mechanism allows us to have multiple IoContexts
active at the same time while still having a
single virtual file system whose contents
correctly reflect the current IoContext.

Temporary files can be created, copied, etc only
within the temporary directory. All other
directories are read-only.

When there is no active IoContext, all temporary
files will have a timestamp set to the Unix epoch.
When there is an active IoContext, the temporary
files will acquire the current timestamp from the
IoContext ensuring that the file system's view of
time is consistent within a request.

The design here is intended to be extensible. We
can add new root directories in the future with
different semantics and implementations. For
example, to support python workers we can
introduce a new root directory that is backed,
for instance, by a tar/zip file containing the
python standard library, etc.

What isn't implemented yet? Great question! The
following todos are remaining:

* Implementing memory accounting for the temporary
  file system. Currently the implementation is
  using the default in-memory file factory
  provided by kj. This is not ideal as it doesn't
  provide any mechanisms for memory accounting
  that integrates with the Isolate heap limits.
  As a next step, before this work is complete,
  we will need to implement a custom in-memory
  file factory that does integrate with the memory
  accounting system. The intention is that the
  total size of the temporary files will be
  limited to the Isolate heap limit.

* Implmenting "file descriptors". Currently the
  file system will return `kj::none` for all file
  descriptors. As a next step, the file system
  will implement a custom file descriptor counter
  that will ensure that all files are assigned a
  unique monotonically increasing file descriptor.
  When the counter reaches a maximum threshold of
  opened files, the worker will be condemned. We
  can also just track the number of active
  `kj::Own<const kj::File>` objects and use that
  internally for accounting but the `node:fs` impl
  does a lot with integer fds so we will need them
  at some point for that.

The implementation currently does not support
symbolic links in any way. I do not anticipate
implementing this in the future.

The implementation is split across several
files/components:

`worker-fs.h/c++` - These provide the main
interfaces for the worker file system.

`bundle-fs.h/c++` - These provide the interfaces
for interfacing the workerd configuration with
the worker file system. A similar interface will
need to be provided for the internal repo since
the two use different configuration schemas.

The integration via the `jsg::Lock&` is provided
as the most convenient way to access the file
system where it is needed.

Author: jasnell

src/workerd/api/tests/new-module-registry-test.js

@@ -13,7 +13,7 @@ import { foo as foo2, default as def2 } from 'bar';
 import { createRequire } from 'module';
 
 // Verify that import.meta.url is correct here.
-strictEqual(import.meta.url, 'file:///worker');
+strictEqual(import.meta.url, 'file:///bundle/worker');
 
 // Verify that import.meta.main is true here.
 ok(import.meta.main);
@@ -44,7 +44,8 @@ console.log(import.meta);
 // Verify that import.meta.resolve provides correct results here.
 // The input should be interpreted as a URL and normalized according
 // to the rules in the WHATWG URL specification.
-strictEqual(import.meta.resolve('./.././test/.././../foo'), 'file:///foo');
+strictEqual(import.meta.resolve('./.././test/.././.%2e/foo'), 'file:///foo');
+strictEqual(import.meta.resolve('foo'), 'file:///bundle/foo');
 
 // There are four tests at this top level... one for the import of the node:assert
 // module without the node: prefix specifier, two for the imports of the foo and
@@ -58,7 +59,7 @@ strictEqual(def2, 2);
 strictEqual(fs, 'abc');
 
 // Equivalent to the above, but using the file: URL scheme.
-import { foo as foo3, default as def3 } from 'file:///foo';
+import { foo as foo3, default as def3 } from 'file:///bundle/foo';
 strictEqual(foo, foo3);
 strictEqual(def, def3);
 
@@ -78,7 +79,7 @@ import { default as cjs2 } from 'cjs2';
 strictEqual(cjs2.foo, 1);
 strictEqual(cjs2.bar, 2);
 strictEqual(cjs2.filename, 'cjs1');
-strictEqual(cjs2.dirname, '/');
+strictEqual(cjs2.dirname, '/bundle');
 strictEqual(cjs2.assert, assert);
 
 // CommonJS modules can define named exports.
@@ -91,7 +92,7 @@ const customRequireCjs = myRequire('cjs1');
 strictEqual(customRequireCjs.foo, cjs1foo);
 strictEqual(customRequireCjs.bar, cjs1bar);
 
-await rejects(import('file:///cjs3'), {
+await rejects(import('file:///bundle/cjs3'), {
   message: 'boom',
 });
 
@@ -109,7 +110,7 @@ await rejects(import('invalid-json'), {
 });
 
 await rejects(import('module-not-found'), {
-  message: /Module not found: file:\/\/\/module-not-found/,
+  message: /Module not found: file:\/\/\/bundle\/module-not-found/,
 });
 
 // Verify that a module is unable to perform IO operations at the top level, even if
@@ -160,10 +161,10 @@ export const queryAndFragment = {
     notStrictEqual(c, d);
 
     // The import.meta.url for each should match the specifier used to import the instance.
-    strictEqual(a.bar, 'file:///foo?query');
-    strictEqual(b.bar, 'file:///foo#fragment');
-    strictEqual(c.bar, 'file:///foo?query#fragment');
-    strictEqual(d.bar, 'file:///foo');
+    strictEqual(a.bar, 'file:///bundle/foo?query');
+    strictEqual(b.bar, 'file:///bundle/foo#fragment');
+    strictEqual(c.bar, 'file:///bundle/foo?query#fragment');
+    strictEqual(d.bar, 'file:///bundle/foo');
   },
 };
 

src/workerd/io/BUILD.bazel

@@ -52,6 +52,7 @@ wd_cc_library(
         "io-util.c++",
         "tracer.c++",
         "worker.c++",
+        "worker-fs.c++",
     ] + ["//src/workerd/api:srcs"],
     hdrs = [
         "compatibility-date.h",
@@ -64,6 +65,7 @@ wd_cc_library(
         "promise-wrapper.h",
         "tracer.h",
         "worker.h",
+        "worker-fs.h",
     ] + ["//src/workerd/api:hdrs"],
     # global CPU-specific options are inconvenient to specify with bazel – just set the options we
     # need for CRC32C in this target for now.
@@ -368,3 +370,10 @@ kj_test(
         ":frankenvalue",
     ],
 )
+
+kj_test(
+    src = "worker-fs-test.c++",
+    deps = [
+        ":io",
+    ],
+)

src/workerd/io/io-context.c++

@@ -131,6 +131,8 @@ IoContext::IoContext(ThreadContext& thread,
     kj::Maybe<Worker::Actor&> actorParam,
     kj::Own<LimitEnforcer> limitEnforcerParam)
     : thread(thread),
+      clock(*this),
+      tempDirStoreScope(TempDirStoreScope::create(clock)),
       worker(kj::mv(workerParam)),
       actor(actorParam),
       limitEnforcer(kj::mv(limitEnforcerParam)),

src/workerd/io/io-context.h

@@ -14,6 +14,7 @@
 #include <workerd/io/io-thread-context.h>
 #include <workerd/io/io-timers.h>
 #include <workerd/io/trace.h>
+#include <workerd/io/worker-fs.h>
 #include <workerd/jsg/async-context.h>
 #include <workerd/jsg/jsg.h>
 #include <workerd/util/exception.h>
@@ -224,6 +225,20 @@ class IoContext final: public kj::Refcounted, private kj::TaskSet::ErrorHandler
  public:
   class TimeoutManagerImpl;
 
+  class IoContextClock: public kj::Clock {
+   public:
+    IoContextClock(IoContext& context): context(context) {}
+    kj::Date now() const override {
+      if (context.isCurrent()) {
+        return context.now();
+      }
+      return kj::UNIX_EPOCH;
+    }
+
+   private:
+    IoContext& context;
+  };
+
   // Construct a new IoContext. Before using it, you must also create an IncomingRequest.
   IoContext(ThreadContext& thread,
       kj::Own<const Worker> worker,
@@ -623,6 +638,15 @@ class IoContext final: public kj::Refcounted, private kj::TaskSet::ErrorHandler
   // Access the event loop's current time point. This will remain constant between ticks.
   kj::Date now();
 
+  // Returns a kj::Clock that returns the same value as now().
+  const kj::Clock& getClock() const {
+    return clock;
+  }
+
+  const TempDirStoreScope& getTempDirStoreScope() const {
+    return *tempDirStoreScope;
+  }
+
   // Returns a promise that resolves once `now() >= when`.
   kj::Promise<void> atTime(kj::Date when) {
     return getIoChannelFactory().getTimer().atTime(when);
@@ -839,6 +863,9 @@ class IoContext final: public kj::Refcounted, private kj::TaskSet::ErrorHandler
 
   kj::Own<WeakRef> selfRef = kj::refcounted<WeakRef>(kj::Badge<IoContext>(), *this);
 
+  IoContextClock clock;
+  kj::Own<TempDirStoreScope> tempDirStoreScope;
+
   kj::Own<const Worker> worker;
   kj::Maybe<Worker::Actor&> actor;
   kj::Own<LimitEnforcer> limitEnforcer;

src/workerd/io/worker-fs-test.c++

@@ -0,0 +1,121 @@
+#include "worker-fs.h"
+
+#include <kj/debug.h>
+#include <kj/test.h>
+
+namespace workerd {
+namespace {
+
+kj::Own<FsMap> createTestFsMap() {
+  FsMap::Builder builder;
+  builder.setPathForKnownRoot(FsMap::KnownRoot::BUNDLE, "/mything/bundle");
+  builder.setPathForKnownRoot(FsMap::KnownRoot::TEMP, "/mything/temp");
+  return builder.finish();
+}
+
+KJ_TEST("FsMap") {
+  auto fsMap = createTestFsMap();
+
+  // Check that the paths are correct.
+  KJ_EXPECT(
+      fsMap->getPathForKnownRoot(FsMap::KnownRoot::BUNDLE) == kj::Path({"mything", "bundle"}));
+
+  KJ_EXPECT(fsMap->getPathForKnownRoot(FsMap::KnownRoot::TEMP) == kj::Path({"mything", "temp"}));
+
+  KJ_EXPECT(
+      fsMap->getUrlForKnownRoot(FsMap::KnownRoot::BUNDLE).equal("file:///mything/bundle"_url));
+
+  KJ_EXPECT(fsMap->getUrlForKnownRoot(FsMap::KnownRoot::TEMP).equal("file:///mything/temp"_url));
+
+  // Check that the known root is correct.
+  auto knownRoot = fsMap->tryGetKnownRootForUrl("file:///mything/bundle"_url);
+  KJ_EXPECT(knownRoot == FsMap::KnownRoot::BUNDLE);
+
+  knownRoot = fsMap->tryGetKnownRootForUrl("file:///mything/temp"_url);
+  KJ_EXPECT(knownRoot == FsMap::KnownRoot::TEMP);
+
+  knownRoot = fsMap->tryGetKnownRootForPath(kj::Path({"mything", "bundle"}));
+  KJ_EXPECT(knownRoot == FsMap::KnownRoot::BUNDLE);
+
+  knownRoot = fsMap->tryGetKnownRootForPath(kj::Path({"mything", "temp"}));
+  KJ_EXPECT(knownRoot == FsMap::KnownRoot::TEMP);
+
+  KJ_EXPECT(fsMap->tryGetKnownRootForPath(kj::Path({"does", "not", "exist"})) == kj::none);
+  KJ_EXPECT(fsMap->tryGetKnownRootForUrl("http://not/supported"_url) == kj::none);
+}
+
+KJ_TEST("TempDirStoreScope") {
+  // We can create multiple temp storages on the heap...
+  auto tempStoreOnHeap = TempDirStoreScope::create(UnixEpochClock::instance);
+  auto tempStoreOnHeap2 = TempDirStoreScope::create(UnixEpochClock::instance);
+
+  KJ_EXPECT(!TempDirStoreScope::hasCurrent());
+
+  {
+    // But we can only have one on the stack at a time per thread.
+    TempDirStoreScope tempDirStoreScope;
+    KJ_EXPECT(TempDirStoreScope::hasCurrent());
+    KJ_ASSERT(&TempDirStoreScope::current() == &tempDirStoreScope);
+    KJ_ASSERT(&TempDirStoreScope::current() != tempStoreOnHeap.get());
+    KJ_ASSERT(&TempDirStoreScope::current() != tempStoreOnHeap2.get());
+
+    auto temp = tempDirStoreScope.getDirectory().createTemporary();
+    temp->write(0, kj::arr<kj::byte>(0, 1, 2, 3, 4));
+    KJ_EXPECT(temp->stat().lastModified == kj::UNIX_EPOCH);
+    KJ_EXPECT(temp->stat().size == 5);
+  }
+}
+
+KJ_TEST("WorkerFileSystem") {
+  auto fsMap = createTestFsMap();
+
+  // In a real worker, the bundle delegate is created from the
+  // worker configuration. Here we just create a dummy one using
+  // the temp directory delegate.
+  TempDirStoreScope tempDirStoreScope;
+  auto bundle = getTempDirectoryDelegate();
+  KJ_ASSERT_NONNULL(bundle->tryOpenFile(kj::Path("foo"), kj::WriteMode::CREATE));
+
+  auto workerFs = getWorkerFileSystem(*fsMap, kj::mv(bundle));
+
+  auto& root = workerFs->getRoot();
+  auto metadata = root.stat();
+  KJ_EXPECT(metadata.type == kj::FsNode::Type::DIRECTORY);
+  KJ_EXPECT(metadata.size == 0);
+
+  auto names = root.listNames();
+  KJ_EXPECT(names.size() == 1);
+  KJ_EXPECT(names[0] == "mything");
+
+  auto entries = root.listEntries();
+  KJ_EXPECT(entries.size() == 1);
+  KJ_EXPECT(entries[0].type == kj::FsNode::Type::DIRECTORY);
+  KJ_EXPECT(entries[0].name == "mything");
+
+  KJ_EXPECT(root.exists(kj::Path({"mything"})));
+  KJ_EXPECT(root.exists(kj::Path({"mything", "bundle"})));
+  KJ_EXPECT(root.exists(kj::Path({"mything", "temp"})));
+  KJ_EXPECT(root.exists(kj::Path({"mything", "bundle", "foo"})));
+
+  metadata = root.lstat(kj::Path({"mything"}));
+  KJ_EXPECT(metadata.type == kj::FsNode::Type::DIRECTORY);
+  KJ_EXPECT(metadata.size == 0);
+
+  metadata = root.lstat(kj::Path({"mything", "bundle"}));
+  KJ_EXPECT(metadata.type == kj::FsNode::Type::DIRECTORY);
+  KJ_EXPECT(metadata.size == 0);
+
+  metadata = root.lstat(kj::Path({"mything", "bundle", "foo"}));
+  KJ_EXPECT(metadata.type == kj::FsNode::Type::FILE);
+  KJ_EXPECT(metadata.size == 0);
+
+  metadata = root.lstat(kj::Path({"mything", "temp"}));
+  KJ_EXPECT(metadata.type == kj::FsNode::Type::DIRECTORY);
+  KJ_EXPECT(metadata.size == 0);
+
+  auto subdir = root.openSubdir(kj::Path({"mything", "bundle"}));
+  KJ_EXPECT(subdir->stat().type == kj::FsNode::Type::DIRECTORY);
+}
+
+}  // namespace
+}  // namespace workerd