Out of Memory on 33103 html and with image 671111 regular files #5552

githubbbie · 2024-09-30T20:48:20Z

Check List

I have already read Docs page & Troubleshooting page.
I have already searched existing issues and they are not help to me.
I examined error or warning messages and it's difficult to solve.
I am using the latest version of Hexo. (run hexo version to check)
My Node.js is matched the required version.

Expected behavior

Run:

hexo server

or

hexo generate

And the site is generated in record time :)

Actual behavior

Tried #2165 and no luck.

Get out of memory error as such:

Ok, trying on 33103 files and getting fail at 16 gb

#!/usr/bin/env node --max_old_space_size=16384

in node_modules/hexo-cli/bin/hexo

Error:

<--- Last few GCs --->

[137:0x7964cb02b000]    33039 ms: Scavenge (reduce) (interleaved) 3446.1 (3671.1) -> 3446.1 (3668.1) MB, pooled: 0 MB, 8.24 / 0.00 ms  (average mu = 0.274, current mu = 0.252) external memory pressure; 
[137:0x7964cb02b000]    33791 ms: Mark-Compact (reduce) 3446.3 (3668.1) -> 3444.6 (3648.6) MB, pooled: 0 MB, 96.37 / 0.00 ms  (+ 649.8 ms in 0 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 789 ms) (average mu = 0.262, 

<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
----- Native stack trace -----

Aborted (core dumped)

AND, here is the error at 32gb

<--- Last few GCs --->

[148:0x7aed4cf6f000]    34166 ms: Scavenge (reduce) (interleaved) 3444.6 (3660.1) -> 3444.6 (3657.1) MB, pooled: 0 MB, 8.24 / 0.00 ms  (average mu = 0.262, current mu = 0.248) external memory pressure; 
[148:0x7aed4cf6f000]    34920 ms: Mark-Compact (reduce) 3444.7 (3657.1) -> 3443.1 (3638.9) MB, pooled: 0 MB, 99.86 / 0.00 ms  (+ 648.2 ms in 0 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 791 ms) (average mu = 0.256, 

<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
----- Native stack trace -----

Aborted (core dumped)

How to reproduce?

hexo generate

Is the problem still there under `Safe mode`?

Yes

Your Node.js & npm version

node -v && npm -v

v22.9.0
10.8.3

Your Hexo and Plugin version

/hexo # npm ls --depth 0
[email protected] /hexo
+-- [email protected]
+-- [email protected]
+-- [email protected]
+-- [email protected]
+-- [email protected]
+-- [email protected]
+-- [email protected]
+-- [email protected]
+-- [email protected]
+-- [email protected]
`-- [email protected]

Your `package.json`

{
  "name": "hexo-site",
  "version": "0.0.0",
  "private": true,
  "scripts": {
    "build": "hexo generate",
    "clean": "hexo clean",
    "deploy": "hexo deploy",
    "server": "hexo server"
  },
  "hexo": {
    "version": "7.3.0"
  },
  "dependencies": {
    "hexo": "^7.3.0",
    "hexo-bridge": "^2.0.3",
    "hexo-generator-archive": "^2.0.0",
    "hexo-generator-category": "^2.0.0",
    "hexo-generator-index": "^4.0.0",
    "hexo-generator-tag": "^2.0.0",
    "hexo-renderer-ejs": "^2.0.0",
    "hexo-renderer-marked": "^6.3.0",
    "hexo-renderer-stylus": "^3.0.1",
    "hexo-server": "^3.0.0",
    "hexo-theme-landscape": "^1.0.0"
  }
}

Your site's `_config.yml` (Optional)

No response

Others

No response

The text was updated successfully, but these errors were encountered:

githubbbie · 2024-09-30T22:22:33Z

Wow, got further this time:

node --max-old-space-size=5168 node_modules/hexo/bin/hexo generate -c 1

Template render error: (unknown path)
  Error: unexpected end of comment
    at Object._prettifyError (/hexo/node_modules/nunjucks/src/lib.js:32:11)
    at Template.render (/hexo/node_modules/nunjucks/src/environment.js:442:21)
    at Environment.renderString (/hexo/node_modules/nunjucks/src/environment.js:313:17)
    at /hexo/node_modules/hexo/dist/extend/tag.js:206:22
    at tryCatcher (/hexo/node_modules/bluebird/js/release/util.js:16:23)
    at Promise.fromNode.Promise.fromCallback (/hexo/node_modules/bluebird/js/release/promise.js:209:30)
    at Tag.render (/hexo/node_modules/hexo/dist/extend/tag.js:205:35)
    at Object.onRenderEnd (/hexo/node_modules/hexo/dist/hexo/post.js:399:32)
    at /hexo/node_modules/hexo/dist/hexo/render.js:77:29
    at tryCatcher (/hexo/node_modules/bluebird/js/release/util.js:16:23)
    at Promise._settlePromiseFromHandler (/hexo/node_modules/bluebird/js/release/promise.js:547:31)
    at Promise._settlePromise (/hexo/node_modules/bluebird/js/release/promise.js:604:18)
    at Promise._settlePromise0 (/hexo/node_modules/bluebird/js/release/promise.js:649:10)
    at Promise._settlePromises (/hexo/node_modules/bluebird/js/release/promise.js:729:18)
    at _drainQueueStep (/hexo/node_modules/bluebird/js/release/async.js:93:12)
    at _drainQueue (/hexo/node_modules/bluebird/js/release/async.js:86:9)

Will up to 8192

node --max-old-space-size=8192 node_modules/hexo/bin/hexo generate -c 1

githubbbie · 2024-10-01T13:07:45Z

node --max-old-space-size=8192 node_modules/hexo/bin/hexo generate -c 1

Same error

Template render error: (unknown path)
Error: unexpected end of comment
at Object._prettifyError (/hexo/node_modules/nunjucks/src/lib.js:32:11)
at Template.render (/hexo/node_modules/nunjucks/src/environment.js:442:21)
at Environment.renderString (/hexo/node_modules/nunjucks/src/environment.js:313:17)
at /hexo/node_modules/hexo/dist/extend/tag.js:206:22
at tryCatcher (/hexo/node_modules/bluebird/js/release/util.js:16:23)
at Promise.fromNode.Promise.fromCallback (/hexo/node_modules/bluebird/js/release/promise.js:209:30)
at Tag.render (/hexo/node_modules/hexo/dist/extend/tag.js:205:35)
at Object.onRenderEnd (/hexo/node_modules/hexo/dist/hexo/post.js:399:32)
at /hexo/node_modules/hexo/dist/hexo/render.js:77:29
at tryCatcher (/hexo/node_modules/bluebird/js/release/util.js:16:23)
at Promise._settlePromiseFromHandler (/hexo/node_modules/bluebird/js/release/promise.js:547:31)
at Promise._settlePromise (/hexo/node_modules/bluebird/js/release/promise.js:604:18)
at Promise._settlePromise0 (/hexo/node_modules/bluebird/js/release/promise.js:649:10)
at Promise._settlePromises (/hexo/node_modules/bluebird/js/release/promise.js:729:18)
at _drainQueueStep (/hexo/node_modules/bluebird/js/release/async.js:93:12)
at _drainQueue (/hexo/node_modules/bluebird/js/release/async.js:86:9)

githubbbie · 2024-10-01T15:58:07Z

ANDDDD

time node --max-old-space-size=16384 node_modules/hexo/bin/hexo generate -c 1

FATAL Something's wrong. Maybe you can find the solution here: https://hexo.io/docs/troubleshooting.html
Template render error: (unknown path)
  Error: unexpected end of comment
    at Object._prettifyError (/hexo/node_modules/nunjucks/src/lib.js:32:11)
    at Template.render (/hexo/node_modules/nunjucks/src/environment.js:442:21)
    at Environment.renderString (/hexo/node_modules/nunjucks/src/environment.js:313:17)
    at /hexo/node_modules/hexo/dist/extend/tag.js:206:22
    at tryCatcher (/hexo/node_modules/bluebird/js/release/util.js:16:23)
    at Promise.fromNode.Promise.fromCallback (/hexo/node_modules/bluebird/js/release/promise.js:209:30)
    at Tag.render (/hexo/node_modules/hexo/dist/extend/tag.js:205:35)
    at Object.onRenderEnd (/hexo/node_modules/hexo/dist/hexo/post.js:399:32)
    at /hexo/node_modules/hexo/dist/hexo/render.js:77:29
    at tryCatcher (/hexo/node_modules/bluebird/js/release/util.js:16:23)
    at Promise._settlePromiseFromHandler (/hexo/node_modules/bluebird/js/release/promise.js:547:31)
    at Promise._settlePromise (/hexo/node_modules/bluebird/js/release/promise.js:604:18)
    at Promise._settlePromise0 (/hexo/node_modules/bluebird/js/release/promise.js:649:10)
    at Promise._settlePromises (/hexo/node_modules/bluebird/js/release/promise.js:729:18)
    at _drainQueueStep (/hexo/node_modules/bluebird/js/release/async.js:93:12)
    at _drainQueue (/hexo/node_modules/bluebird/js/release/async.js:86:9)
Command exited with non-zero status 2
real    1h 13m 13s
user    1h 13m 22s
sys     0m 23.78s

Would like to use HEXO and find a solution as I believe in the nodejs path, but seems like when others have hit this wall, they just move to HUGO and it just works.

Any help here?

githubbbie · 2024-10-01T23:14:13Z

Ok, surely 32 GB of Ram is enough? haha this must be a deeeper issue...

time node --max-old-space-size=32768 node_modules/hexo/bin/hexo generate -c 1

FATAL Something's wrong. Maybe you can find the solution here: https://hexo.io/docs/troubleshooting.html
Template render error: (unknown path)
  Error: unexpected end of comment
    at Object._prettifyError (/hexo/node_modules/nunjucks/src/lib.js:32:11)
    at Template.render (/hexo/node_modules/nunjucks/src/environment.js:442:21)
    at Environment.renderString (/hexo/node_modules/nunjucks/src/environment.js:313:17)
    at /hexo/node_modules/hexo/dist/extend/tag.js:206:22
    at tryCatcher (/hexo/node_modules/bluebird/js/release/util.js:16:23)
    at Promise.fromNode.Promise.fromCallback (/hexo/node_modules/bluebird/js/release/promise.js:209:30)
    at Tag.render (/hexo/node_modules/hexo/dist/extend/tag.js:205:35)
    at Object.onRenderEnd (/hexo/node_modules/hexo/dist/hexo/post.js:399:32)
    at /hexo/node_modules/hexo/dist/hexo/render.js:77:29
    at tryCatcher (/hexo/node_modules/bluebird/js/release/util.js:16:23)
    at Promise._settlePromiseFromHandler (/hexo/node_modules/bluebird/js/release/promise.js:547:31)
    at Promise._settlePromise (/hexo/node_modules/bluebird/js/release/promise.js:604:18)
    at Promise._settlePromise0 (/hexo/node_modules/bluebird/js/release/promise.js:649:10)
    at Promise._settlePromises (/hexo/node_modules/bluebird/js/release/promise.js:729:18)
    at _drainQueueStep (/hexo/node_modules/bluebird/js/release/async.js:93:12)
    at _drainQueue (/hexo/node_modules/bluebird/js/release/async.js:86:9)
Command exited with non-zero status 2
real    1h 8m 35s
user    1h 8m 42s
sys     0m 24.28s

dimaslanjaka · 2024-10-20T12:28:50Z

we need separate generator. Like:

hexo generate tags only generate tags page
hexo generate post only generate post
hexo generate page only generate page (non post in source/_posts)
hexo generate categories only generate categories page
hexo generate assets only process scss, js, images, non markdown files, post assets into public folder

This memory heap error because hexo handle all items inside array json database.

for now marked, markdown-it to process single markdown file, consume 200MB RAM. hexo need migration.

hexo cannot process when the posts or assets more than device memory can handle.

Rather than process all items in one process, why not get -> process -> save to public folder of each file, it will make hexo memory friendly.

githubbbie · 2024-10-21T02:13:22Z

Wow!

…

On October 20, 2024 7:39:11 AM CDT, Dimas Lanjaka ***@***.***> wrote: for now `marked`, `markdown-it` to process single markdown file, consume 200MB RAM. `hexo` need migration. -- Reply to this email directly or view it on GitHub: #5552 (comment) You are receiving this because you authored the thread. Message ID: ***@***.***>

githubbbie · 2024-10-21T02:14:06Z

Nice idea. Who are the active devs on this project?

…

On October 20, 2024 7:29:12 AM CDT, Dimas Lanjaka ***@***.***> wrote: we need separate generator. Like: - `hexo generate tags` only generate tags page - `hexo generate post` only generate post - `hexo generate page` only generate page (non post in source/_posts) - `hexo generate categories` only generate categories page - `hexo generate assets` only process scss, js, images, non markdown files, post assets into public folder This memory heap error because hexo handle all items inside array json database. -- Reply to this email directly or view it on GitHub: #5552 (comment) You are receiving this because you authored the thread. Message ID: ***@***.***>

dimaslanjaka · 2024-10-24T22:59:55Z

Look at my debugger, main out of memory occurs while rendering markdown files

markdown-it now taking more RAM usage

githubbbie · 2024-10-26T00:05:11Z

Wow! Whonare the developers on this project? Starting to feel like this project is abandoned

…

On October 24, 2024 6:00:17 PM CDT, Dimas Lanjaka ***@***.***> wrote: Look at my debugger, main out of memory caused while rendering markdown files ![image](https://github.com/user-attachments/assets/8f8df55d-25c7-4534-9620-9b896e36121a) -- Reply to this email directly or view it on GitHub: #5552 (comment) You are receiving this because you authored the thread. Message ID: ***@***.***>

dimaslanjaka · 2024-10-26T08:30:16Z

Wow! Whonare the developers on this project? Starting to feel like this project is abandoned
…
On October 24, 2024 6:00:17 PM CDT, Dimas Lanjaka @.> wrote: Look at my debugger, main out of memory caused while rendering markdown files -- Reply to this email directly or view it on GitHub: #5552 (comment) You are receiving this because you authored the thread. Message ID: @.>

i dont know who the starter dev. but, these ppl still active to review PR @uiolee @SukkaW

dimaslanjaka mentioned this issue Oct 20, 2024

EMFILE: too many open files #4485

Closed

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Out of Memory on 33103 html and with image 671111 regular files #5552

Out of Memory on 33103 html and with image 671111 regular files #5552

githubbbie commented Sep 30, 2024

githubbbie commented Sep 30, 2024

githubbbie commented Oct 1, 2024

githubbbie commented Oct 1, 2024

githubbbie commented Oct 1, 2024

dimaslanjaka commented Oct 20, 2024 •

edited

Loading

githubbbie commented Oct 21, 2024 via email

githubbbie commented Oct 21, 2024 via email

dimaslanjaka commented Oct 24, 2024 •

edited

Loading

githubbbie commented Oct 26, 2024 via email

dimaslanjaka commented Oct 26, 2024

Out of Memory on 33103 html and with image 671111 regular files #5552

Out of Memory on 33103 html and with image 671111 regular files #5552

Comments

githubbbie commented Sep 30, 2024

Check List

Expected behavior

Actual behavior

How to reproduce?

Is the problem still there under Safe mode?

Your Node.js & npm version

Your Hexo and Plugin version

Your package.json

Your site's _config.yml (Optional)

Others

githubbbie commented Sep 30, 2024

githubbbie commented Oct 1, 2024

githubbbie commented Oct 1, 2024

githubbbie commented Oct 1, 2024

dimaslanjaka commented Oct 20, 2024 • edited Loading

githubbbie commented Oct 21, 2024 via email

githubbbie commented Oct 21, 2024 via email

dimaslanjaka commented Oct 24, 2024 • edited Loading

githubbbie commented Oct 26, 2024 via email

dimaslanjaka commented Oct 26, 2024

Is the problem still there under `Safe mode`?

Your `package.json`

Your site's `_config.yml` (Optional)

dimaslanjaka commented Oct 20, 2024 •

edited

Loading

dimaslanjaka commented Oct 24, 2024 •

edited

Loading