panic at high scale

[goingaround]

2025-01-16 10:56:53 panic: runtime error: slice bounds out of range [:4294967296:4293918720] (recovered by wazero)
2025-01-16 10:56:53 wasm stack trace:
2025-01-16 10:56:53 .$38(i32) i32
2025-01-16 10:56:53 .$40(i32) i32
2025-01-16 10:56:53 .$39(i32) i32
2025-01-16 10:56:53
2025-01-16 10:56:53 Go runtime stack trace:
2025-01-16 10:56:53 goroutine 11038 [running]:
2025-01-16 10:56:53 runtime/debug.Stack()
2025-01-16 10:56:53 /usr/local/go/src/runtime/debug/stack.go:26 +0x5e
2025-01-16 10:56:53 github.com/tetratelabs/wazero/internal/wasmdebug.(*stackTrace).FromRecovered(0xc01244d708?, {0x8e2ec0, 0xc001622108})
2025-01-16 10:56:53 /go/pkg/mod/github.com/tetratelabs/wazero@v1.8.2/internal/wasmdebug/debug.go:142 +0x15c
2025-01-16 10:56:53 github.com/tetratelabs/wazero/internal/engine/wazevo.(*callEngine).callWithStack.func1()
2025-01-16 10:56:53 /go/pkg/mod/github.com/tetratelabs/wazero@v1.8.2/internal/engine/wazevo/call_engine.go:255 +0x3ec
2025-01-16 10:56:53 panic({0x8e2ec0?, 0xc001622108?})
2025-01-16 10:56:53 /usr/local/go/src/runtime/panic.go:785 +0x132
2025-01-16 10:56:53 github.com/wasilibs/go-re2/internal/alloc.(*mmappedMemory).Reallocate(0x0?, 0x0?)
2025-01-16 10:56:53 /go/pkg/mod/github.com/wasilibs/go-re2@v1.6.0/internal/alloc/alloc_unix.go:90 +0xee
2025-01-16 10:56:53 github.com/tetratelabs/wazero/internal/wasm.(*MemoryInstance).Grow(0xc00021a500, 0x0?)
2025-01-16 10:56:53 /go/pkg/mod/github.com/tetratelabs/wazero@v1.8.2/internal/wasm/memory.go:246 +0x255
2025-01-16 10:56:53 github.com/tetratelabs/wazero/internal/engine/wazevo.(*callEngine).callWithStack(0xc01244d708, {0x9deb30, 0xf89be0}, {0xc003f8d328, 0x1, 0x6eb128?})
2025-01-16 10:56:53 /go/pkg/mod/github.com/tetratelabs/wazero@v1.8.2/internal/engine/wazevo/call_engine.go:308 +0x465
2025-01-16 10:56:53 github.com/tetratelabs/wazero/internal/engine/wazevo.(*callEngine).CallWithStack(0x890de0?, {0x9deb30?, 0xf89be0?}, {0xc003f8d328?, 0x0?, 0x0?})
2025-01-16 10:56:53 /go/pkg/mod/github.com/tetratelabs/wazero@v1.8.2/internal/engine/wazevo/call_engine.go:191 +0x9f
2025-01-16 10:56:53 github.com/wasilibs/go-re2/internal.(*lazyFunction).callWithStack(0xc00165bc90, {0x9deb30, 0xf89be0}, {0xc003f8d328, 0x1, 0x1})
2025-01-16 10:56:53 /go/pkg/mod/github.com/wasilibs/go-re2@v1.6.0/internal/re2_wazero.go:605 +0x176
2025-01-16 10:56:53 github.com/wasilibs/go-re2/internal.(*lazyFunction).Call1(0xc00165bc90, {0x9deb30, 0xf89be0}, 0xfc50c)
2025-01-16 10:56:53 /go/pkg/mod/github.com/wasilibs/go-re2@v1.6.0/internal/re2_wazero.go:564 +0x55
2025-01-16 10:56:53 github.com/wasilibs/go-re2/internal.malloc(0x0?, 0x0?)
2025-01-16 10:56:53 /go/pkg/mod/github.com/wasilibs/go-re2@v1.6.0/internal/re2_wazero.go:463 +0x2b
2025-01-16 10:56:53 github.com/wasilibs/go-re2/internal.(*libre2ABI).reserve(...)
2025-01-16 10:56:53 /go/pkg/mod/github.com/wasilibs/go-re2@v1.6.0/internal/re2_wazero.go:484
2025-01-16 10:56:53 github.com/wasilibs/go-re2/internal.(*libre2ABI).startOperation(...)
2025-01-16 10:56:53 /go/pkg/mod/github.com/wasilibs/go-re2@v1.6.0/internal/re2_wazero.go:235
2025-01-16 10:56:53 github.com/wasilibs/go-re2/internal.(*Regexp).FindAllStringSubmatch(0xc0000a19e0, {0xc00af2e000, 0xfc4f4}, 0xffffffffffffffff)
2025-01-16 10:56:53 /go/pkg/mod/github.com/wasilibs/go-re2@v1.6.0/internal/re2.go:451 +0x85
2025-01-16 10:56:53 entro.security/detector/service/exposure.(*scanner).Scan(0xc0011a3560, {0x9debf0, 0xc013ee0550}, {0xc00af2e000?, 0x0?})
2025-01-16 10:56:53 /app/detector/service/exposure/scanner.go:33 +0x74
2025-01-16 10:56:53 entro.security/detector.(*executorGroup).Execute.(*executorGroup).executeFn.func2()
2025-01-16 10:56:53 /app/detector/executor_group.go:49 +0x44
2025-01-16 10:56:53 golang.org/x/sync/errgroup.(*Group).Go.func1()
2025-01-16 10:56:53 /go/pkg/mod/golang.org/x/sync@v0.10.0/errgroup/errgroup.go:78 +0x50
2025-01-16 10:56:53 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 2017
2025-01-16 10:56:53 /go/pkg/mod/golang.org/x/sync@v0.10.0/errgroup/errgroup.go:75 +0x96
2025-01-16 10:56:53
2025-01-16 10:56:53
2025-01-16 10:56:53 goroutine 11038 [running]:
2025-01-16 10:56:53 github.com/wasilibs/go-re2/internal.malloc(0x0?, 0x0?)
2025-01-16 10:56:53 /go/pkg/mod/github.com/wasilibs/go-re2@v1.6.0/internal/re2_wazero.go:464 +0x47
2025-01-16 10:56:53 github.com/wasilibs/go-re2/internal.(*libre2ABI).reserve(...)
2025-01-16 10:56:53 /go/pkg/mod/github.com/wasilibs/go-re2@v1.6.0/internal/re2_wazero.go:484
2025-01-16 10:56:53 github.com/wasilibs/go-re2/internal.(*libre2ABI).startOperation(...)
2025-01-16 10:56:53 /go/pkg/mod/github.com/wasilibs/go-re2@v1.6.0/internal/re2_wazero.go:235
2025-01-16 10:56:53 github.com/wasilibs/go-re2/internal.(*Regexp).FindAllStringSubmatch(0xc0000a19e0, {0xc00af2e000, 0xfc4f4}, 0xffffffffffffffff)
2025-01-16 10:56:53 /go/pkg/mod/github.com/wasilibs/go-re2@v1.6.0/internal/re2.go:451 +0x85
2025-01-16 10:56:53 entro.security/detector/service/exposure.(*scanner).Scan(0xc0011a3560, {0x9debf0, 0xc013ee0550}, {0xc00af2e000?, 0x0?})
2025-01-16 10:56:53 /app/detector/service/exposure/scanner.go:33 +0x74
2025-01-16 10:56:53 entro.security/detector.(*executorGroup).Execute.(*executorGroup).executeFn.func2()
2025-01-16 10:56:53 /app/detector/executor_group.go:49 +0x44
2025-01-16 10:56:53 golang.org/x/sync/errgroup.(*Group).Go.func1()
2025-01-16 10:56:53 /go/pkg/mod/golang.org/x/sync@v0.10.0/errgroup/errgroup.go:78 +0x50
2025-01-16 10:56:53 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 2017
2025-01-16 10:56:53 /go/pkg/mod/golang.org/x/sync@v0.10.0/errgroup/errgroup.go:75 +0x96

[anuraaga]

Hi @goingaround - it looks like it is trying to allocate beyond 4GB memory. Can you confirm how many regex you compile and what ops per second you are trying? Also a rough estimate of how many items FindAllStringSubmatch would generally to return with the expressions?

[goingaround]

@anuraaga hi!
i was trying to have a stress test with 1000 parallel requests with string value of 1mb each, every request is initiating 25 goroutines and taking a value through 185 regexes. each performs FindAllStringSubmatch. so the potential for concurrent use is 25,000 in the same time (if all the requests are handled on same sec - which is less likely)
is there a way to extend this memory?

[anuraaga]

Thanks for the info @goingaround - that is definitely a strong test. If you're using the latest version, can you also check with v1.7.0?

By default, re2 is used via WebAssembly which has a limit of 4GB memory which can't be raised at least by normal means. Somehow this felt like a large limit but with large machines now I guess it can be limiting, I will think about whether this is a reasonable approach to raise the limit and should first probably document this limit. In the meantime, if you use the cgo backend (requires installing the native re2 library), it should have no limit - and while the Wasm module performs fine for most workloads and better than stdlib, for an intense one like that the smaller improvement in performance with cgo may be significant for you so could be an option.

[goingaround]

Thank you @anuraaga
I will try also that version and let u know.

About cgo, maybe u got a dockerfile with alpine for example? I tried but the image did not built, could also be bc i build it on my Mac.

Another thing I noticed is that if I limit concurrency for up to 10 (my number of CPUs in this case) max Regex patterns processing in parallel, I saw a pretty big improvement on the time elapsed of my test and I was actually able to finish running this same test with no panic.
Important to mention that I'm not compiling the same patterns again so the same 200 patterns are shared in this test.

Do u have any experience with concurrently using the pkg? Is this expected?
Are there better ways to scale up?

[anuraaga]

About cgo, maybe u got a dockerfile with alpine for example

To be honest, I've only tried with ubuntu, which we use in CI. I think it's important to also work with alpine and will see what the steps for it are.

Another thing I noticed is that if I limit concurrency for up to 10 (my number of CPUs in this case) max Regex patterns processing in parallel, I saw a pretty big improvement on the time elapsed of my test and I was actually able to finish running this same test with no panic.

Thanks for this info - actually by the way Wasm works, it cannot be preempted meaning it by nature only runs up to # CPUs concurrently, which generally is fine because it is a CPU-bound task with no I/O. However only when growing available memory space it may be preempted - this itself should be globally locked (it doesn't happen often in a process) but I will double check if there could be some problem with this situation. While concurrency being capped at # CPUs is expected, it isn't expected for the user to have to care about it.

Btw just for reference this is the concurrency test we run, expecting it to be a stress test for GitHub runners that have 2 CPUs. The input size is far less than a megabyte though and I think isn't able to trigger possible concurrency issues in memory.Grow so will look into it.

https://github.com/wasilibs/go-re2/blob/main/concurrency_test.go

[anuraaga]

To be honest, I've only tried with ubuntu, which we use in CI. I think it's important to also work with alpine and will see what the steps for it are.

I was able to build cgo with Alpine and added to the README in #189

Note, it should be possible to build a static binary even with cgo on Alpine but that will require some more investigation. Such issues are why I do still prefer the Wasm backend but the memory issue here is definitely something that needs more consideration.

I am working on a fix so that it doesn't panic as above, though it will hit the memory limit all the same - this may still perform better but possibly not. If you're able to provide your repro, I can try to run it to check its behavior, otherwise I will need to see if I can come up with and run such a test. One thing that comes to mind is that the malloc in the Wasm backend is not optimized for such high concurrency and may be causing memory fragmentation issues. If I can reproduce the memory growth behavior I can see if an alternative malloc improves the situation.

[anuraaga]

Hi @goingaround - sorry I completely missed from your original post that you were using go-re2 1.6.0 there. The panic you saw was already fixed so could you try again with 1.8.0? I suggested 1.7.0 earlier thinking it was an earlier version than what you were using, but first it would be great to understand your behavior on the latest version. Notably, I don't think 1.6.0 had the global lock on memory allocation either. Thanks.