[core:math] Optional validation layer

[jakubtomsu]

The idea is to have a way to ensure correctness all throughout math related code by checking function input domains, as well as NaNs and Infs.

The idea is to eventually add similar validation code to core:math/linalg as well.

This is just a draft. The implementation is probably not ideal, I just went line by line through the entire math lib to see where things could go wrong. There are some things I'm not sure I like.

The entire validation layer is enabled by default in debug builds, otherwise it's disabled by default. It can be force disabled with -define:MATH_ASSERT_ENABLED=false or just -disable-assert commandline parameters. It's also possible to enable function domain checking, but disable NaN and Inf checks with -define:MATH_FINITE_VALIDATION_ENABLED=false.

NOTE: the assertion procedure is contextless and calls runtime.default_assertion_contextless_failure_proc just to keep all math code contextless as well.

Why

I have been using a similar math validation layer in "userspace" in my game/engine for a while, and it's unbelievable how many bugs go COMPLETELY unnoticed for months. In game/graphics projects there is so much math code, it's easy to forget to check for some edge case conditions and then problems start to propagate.
And most of the time it's not immediately noticeable, e.g. when a particle disappears because the collision code resulted in NaNs, or some unimportant entity renders the mesh with NaN quaternion, etc etc.

Of course there is still plenty of other issues a validation layer like this cannot catch, but IMO it's still very practical.

Performance

I've done a little bit of simple benchmarking, and it looks like propagating the source code location when validation is disabled has an extremely small impact. I think it's because the math functions generally have very few parameters, so everything fits into registers. So it usually ends up generating one additional mov instruction per function.

When the validation layer is fully enabled, math code gets about ~7% slower with -o:speed and ~15% slower with -o:minimal. That's a pretty big slowdown, but still might be worth it for debugging and "hardening" purposes.

[GloriousPtr]

I second this, we do something similar at my workplace as well.

[leecommamichael]

When I imagine where this ends, where all math libraries are wrapped this way, I like that there'd be one unified way to hook into classifying floats. I don't like the behavior of asserting when these things are detected. What I'd like a lot more, is my own proc I can provide, which I would use to log. Something like:

[DEBUG] --- [core:math] detected Inf: C:/.../asset_bundle.odin(22:2)
[DEBUG] --- [core:math] detected Inf: C:/.../physics.odin(225:51)

I'd rather this behavior because, in the event that I have a known-bug, I can just put a breakpoint in my proc and peek at the call-stack in the debugger. Otherwise I'll be a little frustrated if I'm working on something unrelated and this brings my program down. Much nicer to see it pop in a log (or do nothing) and be able to prioritize a deeper look later.

I wonder if I can actually implement that with the default assert proc in runtime, or if filtering for these math validation errors would be a pain. I haven't given it much thought.