2021-06-23: more ANF

[nikodemus]

ANF, aka A-normal Form - to simplify things a bit - is fancy way of saying "all arguments to a function call are transformed to variable references before the call".

Ie.

func(arg1(), arg2());

becomes

tmp1 = arg1();
tmp2 = arg2();
func(tmp1, tmp2);

it's useful for a whole host of reasons, which you can learn about in the linked Wikipedia article.

Foolang's current compiler doesn't really use ANF as a representation, as it's very simple and operates directly on the AST - but it generates C code in ANF: all message arguments are first saved to current frame, and the foo_send gets its arguments pulled out of the frame.

This provides two things:

1. It makes the intermediate values trivially visible to the GC.
2. It ensures evaluation order, since C doesn't guarantee evaluation order of function call arguments.

Or that was the intention, but the bit of code that translated record constructors into C didn't do that, leading to #656.

After an appropriate amount of WTF I tracked down the problem to CTranspiler#visitRecord: which merrily generated C code doing nested sends without saving arguments to frame or ensuring evaluation order.

Happily it seems that was the only place where I was emitting a foo_send out of band like that, so the fix was easy. (#659)

Small wonder I ran into this as an evaluation order problem and not as a use-after-free though, because those arguments were also invisible to GC during a reasonably sized window!

This is perhaps a sign that I should really have an IR for the compiler / transpiler besides AST. I've known that, but I wanted to get the self-hosting done with minimal effort and thought emitting C directly from the AST would be it. Now I'm not so sure...

[nikodemus]

Of course clang decided to make func(arg1(), arg2()) call arg2 first, because (1) this is legal in C and you don't want people to rely on undefined behaviour, (2) it is presumably marginally more convenient for codegen.

I'm kind of happy it did so, though, because that led me to finding my bug.

....but I wish there was -Wfunction-call-argument-order