feat: loaderContext.loadModule

[jerrykingxyz]

Summary

This PR implements loaderContext.loadModule feature. It references the implementation of loaderContext.importModule.

How webpack does it

Webpack LoaderContex.importModule implementation is relatively simple

1. start LoaderDependency ( code )
2. set recursive: false to prevent submodule creation ( code )
3. return module information after the module is built ( code )

Implementation details

Rspack's implementation of loadModule is similar to that of importModule, both using an implementation that is completely independent of make moduleGraph.

Overwrite Task

The Overwrite Task is mainly used to overwrite the task defined by Make while retaining its functionality

1. Support the use of custom context.
2. Intercept the Make Module build process, synchronize it to the Module Tracker, and prevent the recursion creation of submodules.

impl Task<ExecutorTaskContext> for OverwriteTask {
  async fn main_run(
    self: Box<Self>,
    context: &mut ExecutorTaskContext,
  ) -> TaskResult<ExecutorTaskContext> {
     // factorize result task
    if let Some(factorize_result_task) = origin_task.as_any().downcast_ref::<FactorizeResultTask>()
    {      
      tracker.on_factorize_failed(...);
    }
    ...
    // build finish
    if let Some(build_result_task) = origin_task.as_any().downcast_ref::<BuildResultTask>() {
        // ignore create sub module
        let _ = origin_task.main_run(origin_context).await?;
        return Ok(tracker.on_build_result(origin_context, &module_identifier));
    }
  }
}

Module Tracker

Module Tracker records and tracks the construction of the Module through OverwriteTask. Compared with the implementation of ImportModule, this implementation is simpler because the modules are all isolated nodes.

pub struct ModuleTracker {
  /// Entry dependency id to target box tasks.
  entry_finish_tasks: HashMap<DependencyId, Vec<BoxTask>>,
}

ModuleTracker records all running entry_dependencies and their executeTask,

1. Execute executeTask directly when dep factorize fails
2. When the module has been added and built, executeTask is directly executed
3. When the module build is completed, execute all the executeTask corresponding to its incoming connection

Cleanup Task

Since there are only isolated nodes in the module graph, the clearing methods need to be implemented separately.

CleanModuleTask

The module cleanup task is mainly to delete the old module before loadModule is executed. Since all nodes are isolated nodes, the entry_dependency can be deleted together while deleting the unnecessary module.

pub struct CleanModuleTask {
  pub changed_files: HashSet<ArcPath>,
}

impl Task<LoadTaskContext> for CleanModuleTask {
  fn get_task_type(&self) -> TaskType {
    TaskType::Main
  }
  async fn main_run(self: Box<Self>, context: &mut LoadTaskContext) -> TaskResult<LoadTaskContext> {
    let artifact = &mut context.origin_context.artifact; 
    let mut mg = ModuleGraph::new(vec![], Some(&mut artifact.module_graph_partial));
    for (mid, module) in mg.modules() {
      if module.need_build() || module.depends_on(&self.changed_files) {
        // remove module
        for (dep_id, _) in mg.revoke_module(&mid) {
          // remove module entry dep
          mg.revoke_dependency(&dep_id, true);
          context.entries.retain(|_k, v| v !== dep_id);
        }
      }
    }
    ...
  }
}

CleanEntryTask

The clear Entry task is used to delete the Entry that is no longer used by revoke_module at the end of make.

pub struct CleanEntryTask {
  pub revoked_module: IdentifierSet
}

impl Task<LoadTaskContext> for CleanEntryTask {
  fn get_task_type(&self) -> TaskType {
    TaskType::Main
  }
  async fn main_run(self: Box<Self>, context: &mut LoadTaskContext) -> TaskResult<LoadTaskContext> {
    let revoked_module = context
      .origin_context
      .artifact
      .revoked_modules
      .iter()
      .chain(self.revoked_module.iter())
      .collect::<HashSet<_>>();
    context.entries.retain(|k, v| {
      !revoked_module.contains(&k.origin_module_identifier) || context.used_entry.contains(v)
    });
    Ok(vec![])
  }
}

Main Process of Module Loader

Suspend task_loop and add CleanModuleTask and CtrlTask. CtrlTask is responsible for dynamically adding external tasks to the task queue.

1. When there is a new loadModule task, it will first be converted into an entryTask and sent to the ctrlTask to join the task queue.
2. When entryTask is executed, the corresponding dependency and ExecuteTask will be generated and registered in the module tracker.
3. When module tracker determines that ExecuteTask can be executed, it will be thrown back into the task queue.
4. When ExecuteTask is called, the callback of loadModule is called to execute the js side code.

By passing CleanEntryTask to the task queue at the end of make, the task queue will exit after execution.

pub struct ModuleLoader {
  // data
  pub make_artifact: MakeArtifact,
  pub entries: HashMap<LoadModuleMeta, DependencyId>,

  // temporary data, used by hook_after_finish_modules
  event_sender: Option<UnboundedSender<Event>>,
  stop_receiver: Option<oneshot::Receiver<LoadTaskContext>>,
}

impl ModuleLoader {
  pub async fn hook_before_make(&mut self, compilation: &Compilation) -> Result<()> {
    let ctx = ...;
    tokio::spawn(task::unconstrained(async move {
      let _ = run_task_loop(
        &mut ctx,
        vec![
          Box::new(CleanModuleTask { changed_files }),
          Box::new(CtrlTask { event_receiver }),
        ],
      )
      .await;

      // ignore error, stop_receiver may be dropped if make stage occur error.
      let _ = stop_sender.send(ctx);
    }));

    Ok(())
  }

  pub async fn hook_after_finish_modules(&mut self, compilation: &mut Compilation) -> Result<()> {
    let sender = std::mem::take(&mut self.event_sender);
    sender
      .expect("should have sender")
      .send(Event::Stop(CleanEntryTask {
        revoked_module: compilation.make_artifact.revoked_modules.clone(),
      }))
      .expect("should success");

    let stop_receiver = std::mem::take(&mut self.stop_receiver);
    let Ok(ctx) = stop_receiver.expect("should have receiver").await else {
      panic!("receive make artifact failed");
    };
    self.make_artifact = ctx.origin_context.artifact;
    self.entries = ctx.entries;

    ...
  }

  pub fn load_module(
    &self,
    request: String,
    layer: Option<String>,
    origin_module_context: Context,
    origin_module_identifier: Identifier,
    callback: impl FnOnce(Result<&BoxModule>) + 'static,
  ) -> Result<()> {
    let sender = self
      .event_sender
      .as_ref()
      .expect("should have event sender");

    let meta = LoadModuleMeta {
      origin_module_identifier,
      request,
      layer,
    };
    sender
      .send(Event::LoadModule(EntryTask {
        meta: meta.clone(),
        origin_module_context,
        execute_task: ExecuteTask {
          meta,
          callback: Callback(Box::new(callback)),
        },
      }))
      .map_err(|_| diagnostic!("send to ctrl task failed").into())
  }
}

Checklist

• Tests updated (or not required).
• Documentation updated (or not required).

[netlify]

✅ Deploy Preview for rspack canceled.

Name	Link
🔨 Latest commit	db825bf
🔍 Latest deploy log	https://app.netlify.com/projects/rspack/deploys/684137214b401800096cb66d

[codspeed-hq]

CodSpeed Performance Report

Merging #10532 will not alter performance

_{Comparing jerry/loadModule (db825bf) with main (336502b)}

🎉 Hooray! codspeed-rust just leveled up to 2.7.2!

A heads-up, this is a breaking change and it might affect your current performance baseline a bit. But here's the exciting part - it's packed with new, cool features and promises improved result stability 🥳!
Curious about what's new? Visit our releases page to delve into all the awesome details about this new version.

Summary

✅ 12 untouched benchmarks