Automate the loss curve correctness tests

[tengyifei]

In #90, #181, #180, we'll do a one point in time test to ensure that the model loss curve tracks a known reference.

It's also useful to have an automated job to check these over time. For large models like Llama 3.1 405B, we can't afford to test it for every PR, so we'll probably need some sort of post-submit testing.