Ideas for preventing memory leaks when hash used in output.filename

[srmagura]

My team was using:

{
  /* ... */
  output: {
    filename: '[name].[chunkhash].js',
    clean: true
  },
  devtool: 'cheap-module-source-map'
}

This caused Webpack's memory usage to increase by ~300 MB on each rebuild. I eventually found this comment by @sokra:

Best avoid using contenthashes during development. This does have similar problems in other places, e. g. it writes a lot files into the filesystem (on disk or in memory).

I removed the output.filename option, so that the default of [name].js is used, and this fixed the rapidly increasing memory usage.

My questions:

1. Since including [fullhash] / [chunkhash] / [contenthash] in output.filename is known to cause memory issues, could Webpack print a warning if output.filename contains one of these variables and mode === 'development'? If yes, then I can attempt a PR.
2. Could Webpack be changed so that it clears cached assets from old builds automatically? I don't know enough about Webpack internals to know if this is feasible.

Ref #12947

[alexander-akait]

How many modules do you have? Search and Replace hashes can consume more memories (but not ~300MB), do you have any non official plugins (will be great to see at configuration too)?

[alexander-akait]

Feel free to ping me

[srmagura]

Hey @alexander-akait, here are the verbose logs! This was from running our full webpack config (not the simplified config I showed earlier). Hopefully this gives you an idea of the total number of modules we have. Thank you 😊

spot-webpack-verbose.zip

[alexander-akait]

Oh, you have really big application 😄 Can you check that what using [fullhash]/[contenthash] (any value except chunkhash) is leeking memory too?

I found you use worker-loader too, there is some magic there. Why don't use built-in solution (i.e. new Worker(new URL(...)))? worker-loader is deprecated and uses the old approach (which potential has memory leading in some rare cases).

Ideally I recommend to disable loaders and plugins step by step to check that no other things are leaking (I know it requires time, but we should check it out first). Also can you disable source maps and check too?

[srmagura]

@alexander-akait

Ideally I recommend to disable loaders and plugins step by step to check that no other things are leaking (I know it requires time, but we should check it out first).

I did do this (posted the minimal config earlier), but the logs I sent you are from running Webpack with our full config.

Just wondering, are you asking me to try these different things because you think there could be a bug in Webpack? If so, I will try them out and report back to you. If you are just trying to help solve my specific issue, there is no need since the issue was resolved once I removed chunkhash.

Thanks for mentioning the worker-loader thing. I found out it is being brought in by pdfjs-dist/webpack, rather than our Webpack config.

[alexander-akait]

Just wondering, are you asking me to try these different things because you think there could be a bug in Webpack? If so, I will try them out and report back to you. If you are just trying to help solve my specific issue, there is no need since the issue was resolved once I removed chunkhash.

The problem is that there are a lot of places where we can have a bug - non official loaders/plugins and including in the webpack itself. The strange circumstance is that the problem is only [chunkhash]. Feeling like some kind of regex try to parser filename, usually it makes a spike in memory consumption (but that's just my guess). Maybe you can profile and capture memory usage (here example of the another memory problem - #12771)?