Enormous Wildcards directory or files causes error

[jtreminio]

Expected Behavior

My wildcards directory contains some very large files - a total of 1.7GB with three files in particular 978MB, 466MB, and 474MB.

The files are nested 3-levels deep, to prevent the /API/ListModels from scanning them constantly and encountering the attached error.

Actual Behavior

Ideally a refactor of how Wildcards are handled for large files:

• detect if a file is over xx size
• only fetch the first 100 lines of any given Wildcard from the /API/ListModels API
• fetch the full file contents when editing Wildcard directly

Steps to Reproduce

Be a hoarder like myself and have a 1GB Wildcard file

Debug Logs

03:59:25.488 [Error] Failing while yielding JSON output: System.ArgumentOutOfRangeException: The length cannot be greater than the capacity. (Parameter 'valueCount')
   at System.Text.StringBuilder.AppendWithExpansion(Char& value, Int32 valueCount)
   at System.Text.StringBuilder.Append(String value)
   at SwarmUI.Utils.Utilities.ToStringFast(JToken jval, StringBuilder builder) in /workspace/src/Utils/Utilities.cs:line 440
   at SwarmUI.Utils.Utilities.ToStringFast(JToken jval, StringBuilder builder) in /workspace/src/Utils/Utilities.cs:line 417
   at SwarmUI.Utils.Utilities.ToStringFast(JToken jval, StringBuilder builder) in /workspace/src/Utils/Utilities.cs:line 431
   at SwarmUI.Utils.Utilities.ToStringFast(JToken jval, StringBuilder builder) in /workspace/src/Utils/Utilities.cs:line 417
   at SwarmUI.Utils.Utilities.JsonToByteArray(JObject jdata) in /workspace/src/Utils/Utilities.cs:line 448
   at SwarmUI.Utils.Utilities.YieldJsonOutput(HttpContext context, WebSocket socket, Int32 status, JObject obj) in /workspace/src/Utils/Utilities.cs:line 478

Other

No response

[mcmonkey4eva]

I genuinely do not know how it's physically possible to create a gigabyte of text in a single file. That doesn't even make sense.

[jtreminio]

I have downloaded several datasets that include prompt dumps from different sources, like danbooru: https://huggingface.co/datasets/hipete12/danbooru-2025-10-05

This one has 12m prompts. I prune them and extract the prompts and throw them into a text file. It helps me finetune my LLM prompts to generate interesting, new prompts from them.